[jira] [Commented] (FLINK-32307) Incorrect docs for default behavior of "scan.startup.mode" option
[
https://issues.apache.org/jira/browse/FLINK-32307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731487#comment-17731487
]
Gunnar Morling commented on FLINK-32307:
Happy to send a PR, unless someone feels I'm off here.
> Incorrect docs for default behavior of "scan.startup.mode" option
> -
>
> Key: FLINK-32307
> URL: https://issues.apache.org/jira/browse/FLINK-32307
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Kafka
>Reporter: Gunnar Morling
>Priority: Major
>
> The [Kafka connector
> docs|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#start-reading-position]
> say this in regards to the {{scan.startup.mode}} option:
> {quote}
> The default option value is group-offsets which indicates to consume from
> last committed offsets in ZK / Kafka brokers.
> {quote}
> Whereas what I actually observe is that the "earliest-offset" mode is used
> when not specifying a value for this option. This matches the implementation
> in
> [{{KafkaSourceBuilder}}|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L106]
> from a quick glimpse.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32307) Incorrect docs for default behavior of "scan.startup.mode" option
Gunnar Morling created FLINK-32307:
--
Summary: Incorrect docs for default behavior of
"scan.startup.mode" option
Key: FLINK-32307
URL: https://issues.apache.org/jira/browse/FLINK-32307
Project: Flink
Issue Type: Technical Debt
Components: Connectors / Kafka
Reporter: Gunnar Morling
The [Kafka connector
docs|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#start-reading-position]
say this in regards to the {{scan.startup.mode}} option:
{quote}
The default option value is group-offsets which indicates to consume from last
committed offsets in ZK / Kafka brokers.
{quote}
Whereas what I actually observe is that the "earliest-offset" mode is used when
not specifying a value for this option. This matches the implementation in
[{{KafkaSourceBuilder}}|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L106]
from a quick glimpse.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30636) Typo fix: "to to" -> "to"
[ https://issues.apache.org/jira/browse/FLINK-30636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17672294#comment-17672294 ] Gunnar Morling commented on FLINK-30636: Will send a PR for this shortly. > Typo fix: "to to" -> "to" > - > > Key: FLINK-30636 > URL: https://issues.apache.org/jira/browse/FLINK-30636 > Project: Flink > Issue Type: Technical Debt >Reporter: Gunnar Morling >Priority: Minor > > There's a surprising number of occurrences of "to to" in JavaDoc and the > like, where it actually should just be "to". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30636) Typo fix: "to to" -> "to"
Gunnar Morling created FLINK-30636: -- Summary: Typo fix: "to to" -> "to" Key: FLINK-30636 URL: https://issues.apache.org/jira/browse/FLINK-30636 Project: Flink Issue Type: Technical Debt Reporter: Gunnar Morling There's a surprising number of occurrences of "to to" in JavaDoc and the like, where it actually should just be "to". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30440) Update operations playground for 1.16
[ https://issues.apache.org/jira/browse/FLINK-30440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653636#comment-17653636 ] Gunnar Morling commented on FLINK-30440: Ok, got it. I'm planning to send the PRs for the two other playgrounds in the course of this week. > Update operations playground for 1.16 > - > > Key: FLINK-30440 > URL: https://issues.apache.org/jira/browse/FLINK-30440 > Project: Flink > Issue Type: Sub-task >Reporter: David Anderson >Assignee: Gunnar Morling >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30440) Update operations playground for 1.16
[ https://issues.apache.org/jira/browse/FLINK-30440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653614#comment-17653614 ] Gunnar Morling commented on FLINK-30440: [~danderson], thanks for merging that PR! What is the procedure in regards to Jira? It seems you'd have to set the Fix Version and transition this issue to Resolved, as I'm lacking the rights to do so myself. Thanks again. > Update operations playground for 1.16 > - > > Key: FLINK-30440 > URL: https://issues.apache.org/jira/browse/FLINK-30440 > Project: Flink > Issue Type: Sub-task >Reporter: David Anderson >Assignee: Gunnar Morling >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30443) Expand list of sensitive keys
[
https://issues.apache.org/jira/browse/FLINK-30443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651289#comment-17651289
]
Gunnar Morling commented on FLINK-30443:
Hey [~jiabao.sun] and [~zhuzh], thanks for your feedback! I certainly can make
this configurable, but do you think it's worth it? I'm not so much concerned
about the engineering time for making it happen, this shouldn't take long. But
it adds to the configuration surface and complexity of Flink; generally, the
less bells and whistles, the better, IMO. It makes things easier for users. How
about we add those keys verbatim for now (I've just sent a PR for that), and if
there's another request for extending them down the road, we'll make it
configurable then?
> Expand list of sensitive keys
> -
>
> Key: FLINK-30443
> URL: https://issues.apache.org/jira/browse/FLINK-30443
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Reporter: Gunnar Morling
>Priority: Major
> Labels: pull-request-available
>
> In {{{}GlobalConfiguration{}}}, there is [a list of known configuration
> keys|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L47-L48]
> whose values will be masked in log output. In our Flink deployment there's a
> few more keys which we would like to mask, specifically, the following ones:
> * "auth-params"
> * "service-key"
> * "token"
> * "basic-auth"
> * "jaas.config"
> While those are somewhat use-case specific, I feel they are generic enough
> for being added to that list, and there already is precedence in form of
> "fs.azure.account.key". In that light, I don't think it's worth making this
> somehow pluggable, but I'm curious what other folks here think. Thanks!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30478) Don't depend on IPAddressUtil
Gunnar Morling created FLINK-30478:
--
Summary: Don't depend on IPAddressUtil
Key: FLINK-30478
URL: https://issues.apache.org/jira/browse/FLINK-30478
Project: Flink
Issue Type: Sub-task
Components: API / Core
Reporter: Gunnar Morling
The class \{{org.apache.flink.util.NetUtils}} uses the JDK-internal class
\{{sun.net.util.IPAddressUtil}}. On current JDKs (16+), this causes issues as
access to this class is prevented by default and would require an additional
\{{--add-opens}} clause. That's undesirable in particular in cases where we
don't control the JVM start-up arguments, e.g. when using Flink embedded into a
custom Java application.
I suggest to replace this logic using the
[IPAddress|https://github.com/seancfoley/IPAddress/] library (Apache License
v2), which implements everything we need without relying on internal classes. I
have a patch for that ready and will submit it for discussion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30455) Avoid "cleaning" of java.lang.String in ClosureCleaner
[
https://issues.apache.org/jira/browse/FLINK-30455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649387#comment-17649387
]
Gunnar Morling commented on FLINK-30455:
Sent PR [https://github.com/apache/flink/pull/21532,] seems simple enough.
> Avoid "cleaning" of java.lang.String in ClosureCleaner
> --
>
> Key: FLINK-30455
> URL: https://issues.apache.org/jira/browse/FLINK-30455
> Project: Flink
> Issue Type: Sub-task
>Reporter: Gunnar Morling
>Assignee: Gunnar Morling
>Priority: Major
> Labels: pull-request-available
>
> When running a simple "hello world" example on JDK 17, I noticed the closure
> cleaner tries to reflectively access the {{java.lang.String}} class, which
> fails due to the strong encapsulation enabled by default in JDK 17 and
> beyond. I don't think the closure cleaner needs to act on {{String}} at all,
> as it doesn't contain any inner classes. Unless there are objections, I'll
> provide a fix along those lines. With this change in place, I can run that
> example on JDK 17.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30454) Inconsistent class hierarchy in TaskIOMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-30454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649383#comment-17649383
]
Gunnar Morling edited comment on FLINK-30454 at 12/19/22 3:51 PM:
--
Ok, cool. Created PR
[https://github.com/apache/flink/pull/21531/.|https://github.com/apache/flink/pull/21531/]
was (Author: gunnar.morling):
Ok, cool. Logged
[https://github.com/apache/flink/pull/21531/.|https://github.com/apache/flink/pull/21531/]
> Inconsistent class hierarchy in TaskIOMetricGroup
> -
>
> Key: FLINK-30454
> URL: https://issues.apache.org/jira/browse/FLINK-30454
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Gunnar Morling
>Priority: Major
> Labels: pull-request-available
>
> I noticed an interesting issue when trying to compile the flink-runtime
> module with Eclipse (same for VSCode): the _private_ inner class
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has
> yet another _public_ inner class, {{{}SizeSupplier{}}}. The public method
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
> has a parameter of that type.
> The invocation of this method in
> {{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
> TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
> TaskMailbox)}} can be compiled with the javac compiler of the JDK, but fails
> to compile with ecj:
> {code:java}
> The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
> target context is not visible here.
> {code}
> I tend to believe that the behavior of Eclipse's compiler is the correct one.
> After all, you couldn't explicitly reference the {{SizeSupplier}} type either.
> One possible fix would be to promote {{SizeSupplier}} to the same level as
> {{{}SizeGauge{}}}. This would be source-compatible but not binary-compatible,
> though. I.e. code compiled against the earlier signature of
> {{registerMailboxSizeSupplier()}} would be broken with a
> {{{}NoSuchMethodError{}}}. Depending on whether
> {{registerMailboxSizeSupplier()}} are expected in client code or not, this
> may or may not be acceptable.
> Another fix would be to make {{SizeGauge}} public. I think that's the change
> I'd do. Curious what other folks here think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30454) Inconsistent class hierarchy in TaskIOMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-30454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649383#comment-17649383
]
Gunnar Morling commented on FLINK-30454:
Ok, cool. Logged
[https://github.com/apache/flink/pull/21531/.|https://github.com/apache/flink/pull/21531/]
> Inconsistent class hierarchy in TaskIOMetricGroup
> -
>
> Key: FLINK-30454
> URL: https://issues.apache.org/jira/browse/FLINK-30454
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Gunnar Morling
>Priority: Major
> Labels: pull-request-available
>
> I noticed an interesting issue when trying to compile the flink-runtime
> module with Eclipse (same for VSCode): the _private_ inner class
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has
> yet another _public_ inner class, {{{}SizeSupplier{}}}. The public method
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
> has a parameter of that type.
> The invocation of this method in
> {{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
> TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
> TaskMailbox)}} can be compiled with the javac compiler of the JDK, but fails
> to compile with ecj:
> {code:java}
> The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
> target context is not visible here.
> {code}
> I tend to believe that the behavior of Eclipse's compiler is the correct one.
> After all, you couldn't explicitly reference the {{SizeSupplier}} type either.
> One possible fix would be to promote {{SizeSupplier}} to the same level as
> {{{}SizeGauge{}}}. This would be source-compatible but not binary-compatible,
> though. I.e. code compiled against the earlier signature of
> {{registerMailboxSizeSupplier()}} would be broken with a
> {{{}NoSuchMethodError{}}}. Depending on whether
> {{registerMailboxSizeSupplier()}} are expected in client code or not, this
> may or may not be acceptable.
> Another fix would be to make {{SizeGauge}} public. I think that's the change
> I'd do. Curious what other folks here think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30455) Avoid "cleaning" of java.lang.String in ClosureCleaner
[
https://issues.apache.org/jira/browse/FLINK-30455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649371#comment-17649371
]
Gunnar Morling commented on FLINK-30455:
[~rmetzger] et al., if you think this makes sense, could you assign this issue
to me? Thanks!
> Avoid "cleaning" of java.lang.String in ClosureCleaner
> --
>
> Key: FLINK-30455
> URL: https://issues.apache.org/jira/browse/FLINK-30455
> Project: Flink
> Issue Type: Sub-task
>Reporter: Gunnar Morling
>Priority: Major
>
> When running a simple "hello world" example on JDK 17, I noticed the closure
> cleaner tries to reflectively access the {{java.lang.String}} class, which
> fails due to the strong encapsulation enabled by default in JDK 17 and
> beyond. I don't think the closure cleaner needs to act on {{String}} at all,
> as it doesn't contain any inner classes. Unless there are objections, I'll
> provide a fix along those lines. With this change in place, I can run that
> example on JDK 17.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30455) Avoid "cleaning" of java.lang.String in ClosureCleaner
Gunnar Morling created FLINK-30455:
--
Summary: Avoid "cleaning" of java.lang.String in ClosureCleaner
Key: FLINK-30455
URL: https://issues.apache.org/jira/browse/FLINK-30455
Project: Flink
Issue Type: Sub-task
Reporter: Gunnar Morling
When running a simple "hello world" example on JDK 17, I noticed the closure
cleaner tries to reflectively access the {{java.lang.String}} class, which
fails due to the strong encapsulation enabled by default in JDK 17 and beyond.
I don't think the closure cleaner needs to act on {{String}} at all, as it
doesn't contain any inner classes. Unless there are objections, I'll provide a
fix along those lines. With this change in place, I can run that example on JDK
17.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30454) Inconsistent class hierarchy in TaskIOMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-30454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunnar Morling updated FLINK-30454:
---
Description:
I noticed an interesting issue when trying to compile the flink-runtime module
with Eclipse (same for VSCode): the _private_ inner class
{{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has yet
another _public_ inner class, {{{}SizeSupplier{}}}. The public method
{{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
has a parameter of that type.
The invocation of this method in
{{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
TaskMailbox)}} can be compiled with the javac compiler of the JDK, but fails to
compile with ecj:
{code:java}
The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
target context is not visible here.
{code}
I tend to believe that the behavior of Eclipse's compiler is the correct one.
After all, you couldn't explicitly reference the {{SizeSupplier}} type either.
One possible fix would be to promote {{SizeSupplier}} to the same level as
{{{}SizeGauge{}}}. This would be source-compatible but not binary-compatible,
though. I.e. code compiled against the earlier signature of
{{registerMailboxSizeSupplier()}} would be broken with a
{{{}NoSuchMethodError{}}}. Depending on whether
{{registerMailboxSizeSupplier()}} are expected in client code or not, this may
or may not be acceptable.
Another fix would be to make {{SizeGauge}} public. I think that's the change
I'd do. Curious what other folks here think.
was:
I noticed an interesting issue when trying to compile the flink-runtime module
with Eclipse (same for VSCode): the _private_ inner class
{{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has yet
another _public_ inner class, {{SizeSupplier}}. The public method
{{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
has a parameter of that type. The invocation of this method in
{{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
TaskMailbox)}} can be compiled with the javac compiler of the JDK but fails to
compile with ecj:
{code}
The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
target context is not visible here.
{code}
I tend to believe that the behavior of Eclipse's compiler is the correct one.
After all, you couldn't explicitly reference the {{SizeSupplier}} type either.
One possible fix would be to promote {{SizeSupplier}} to the same level as
{{SizeGauge}}. This would be source-compatible but not binary-compatible,
though. I.e. code compiled against the earlier signature of
{{registerMailboxSizeSupplier()}} would be broken with a {{NoSuchMethodError}}.
Depending on whether {{registerMailboxSizeSupplier()}} are expected in client
code or not, this may or may not be acceptable. Another fix would be to make
{{SizeGauge}} public. I think that's the change I'd do. Curious what other
folks here think.
> Inconsistent class hierarchy in TaskIOMetricGroup
> -
>
> Key: FLINK-30454
> URL: https://issues.apache.org/jira/browse/FLINK-30454
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Gunnar Morling
>Priority: Major
>
> I noticed an interesting issue when trying to compile the flink-runtime
> module with Eclipse (same for VSCode): the _private_ inner class
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has
> yet another _public_ inner class, {{{}SizeSupplier{}}}. The public method
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
> has a parameter of that type.
> The invocation of this method in
> {{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
> TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
> TaskMailbox)}} can be compiled with the javac compiler of the JDK, but fails
> to compile with ecj:
> {code:java}
> The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
> target context is not visible here.
> {code}
> I tend to believe that the behavior of Eclipse's compiler is the correct one.
> After all, you couldn't explicitly reference the {{SizeSupplier}} type either.
> One possible fix would be to promote {{SizeSupplier}} to the same level as
> {{{}SizeGauge{}}}. This would be source-compatible but not binary-compatible,
> though. I.e. code compiled against the earlier signature of
> {{registerMailboxSizeSupplier()}} would be
[jira] [Comment Edited] (FLINK-30454) Inconsistent class hierarchy in TaskIOMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-30454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649345#comment-17649345
]
Gunnar Morling edited comment on FLINK-30454 at 12/19/22 2:16 PM:
--
[~rmetzger], curious about your thoughts on that compatibility question.
was (Author: gunnar.morling):
[~rmetzger] , curious about your thoughts on that compatibility question.
> Inconsistent class hierarchy in TaskIOMetricGroup
> -
>
> Key: FLINK-30454
> URL: https://issues.apache.org/jira/browse/FLINK-30454
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Gunnar Morling
>Priority: Major
>
> I noticed an interesting issue when trying to compile the flink-runtime
> module with Eclipse (same for VSCode): the _private_ inner class
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has
> yet another _public_ inner class, {{SizeSupplier}}. The public method
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
> has a parameter of that type. The invocation of this method in
> {{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
> TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
> TaskMailbox)}} can be compiled with the javac compiler of the JDK but fails
> to compile with ecj:
> {code}
> The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
> target context is not visible here.
> {code}
> I tend to believe that the behavior of Eclipse's compiler is the correct one.
> After all, you couldn't explicitly reference the {{SizeSupplier}} type
> either. One possible fix would be to promote {{SizeSupplier}} to the same
> level as {{SizeGauge}}. This would be source-compatible but not
> binary-compatible, though. I.e. code compiled against the earlier signature
> of {{registerMailboxSizeSupplier()}} would be broken with a
> {{NoSuchMethodError}}. Depending on whether {{registerMailboxSizeSupplier()}}
> are expected in client code or not, this may or may not be acceptable.
> Another fix would be to make {{SizeGauge}} public. I think that's the change
> I'd do. Curious what other folks here think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30454) Inconsistent class hierarchy in TaskIOMetricGroup
Gunnar Morling created FLINK-30454:
--
Summary: Inconsistent class hierarchy in TaskIOMetricGroup
Key: FLINK-30454
URL: https://issues.apache.org/jira/browse/FLINK-30454
Project: Flink
Issue Type: Improvement
Components: Runtime / Metrics
Reporter: Gunnar Morling
I noticed an interesting issue when trying to compile the flink-runtime module
with Eclipse (same for VSCode): the _private_ inner class
{{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has yet
another _public_ inner class, {{SizeSupplier}}. The public method
{{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
has a parameter of that type. The invocation of this method in
{{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
TaskMailbox)}} can be compiled with the javac compiler of the JDK but fails to
compile with ecj:
{code}
The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
target context is not visible here.
{code}
I tend to believe that the behavior of Eclipse's compiler is the correct one.
After all, you couldn't explicitly reference the {{SizeSupplier}} type either.
One possible fix would be to promote {{SizeSupplier}} to the same level as
{{SizeGauge}}. This would be source-compatible but not binary-compatible,
though. I.e. code compiled against the earlier signature of
{{registerMailboxSizeSupplier()}} would be broken with a {{NoSuchMethodError}}.
Depending on whether {{registerMailboxSizeSupplier()}} are expected in client
code or not, this may or may not be acceptable. Another fix would be to make
{{SizeGauge}} public. I think that's the change I'd do. Curious what other
folks here think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30454) Inconsistent class hierarchy in TaskIOMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-30454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649345#comment-17649345
]
Gunnar Morling commented on FLINK-30454:
[~rmetzger] , curious about your thoughts on that compatibility question.
> Inconsistent class hierarchy in TaskIOMetricGroup
> -
>
> Key: FLINK-30454
> URL: https://issues.apache.org/jira/browse/FLINK-30454
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Gunnar Morling
>Priority: Major
>
> I noticed an interesting issue when trying to compile the flink-runtime
> module with Eclipse (same for VSCode): the _private_ inner class
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.SizeGauge}} has
> yet another _public_ inner class, {{SizeSupplier}}. The public method
> {{org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup.registerMailboxSizeSupplier(SizeSupplier)}}
> has a parameter of that type. The invocation of this method in
> {{org.apache.flink.streaming.runtime.tasks.StreamTask.StreamTask(Environment,
> TimerService, UncaughtExceptionHandler, StreamTaskActionExecutor,
> TaskMailbox)}} can be compiled with the javac compiler of the JDK but fails
> to compile with ecj:
> {code}
> The type TaskIOMetricGroup.SizeGauge from the descriptor computed for the
> target context is not visible here.
> {code}
> I tend to believe that the behavior of Eclipse's compiler is the correct one.
> After all, you couldn't explicitly reference the {{SizeSupplier}} type
> either. One possible fix would be to promote {{SizeSupplier}} to the same
> level as {{SizeGauge}}. This would be source-compatible but not
> binary-compatible, though. I.e. code compiled against the earlier signature
> of {{registerMailboxSizeSupplier()}} would be broken with a
> {{NoSuchMethodError}}. Depending on whether {{registerMailboxSizeSupplier()}}
> are expected in client code or not, this may or may not be acceptable.
> Another fix would be to make {{SizeGauge}} public. I think that's the change
> I'd do. Curious what other folks here think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20092) [Java 11] Multi-thread Flink compilation not working
[
https://issues.apache.org/jira/browse/FLINK-20092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649220#comment-17649220
]
Gunnar Morling commented on FLINK-20092:
I can confirm that parallel builds work with the two PRs
[#21344|https://github.com/apache/flink/pull/21344/] and
[21477|https://github.com/apache/flink/pull/21477/] applied.
> [Java 11] Multi-thread Flink compilation not working
>
>
> Key: FLINK-20092
> URL: https://issues.apache.org/jira/browse/FLINK-20092
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System
>Reporter: Maciej Bryński
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> I'd like to use maven -T option when compiling flink.
> {code:java}
> mvn -T 2C clean install -D"scala-2.12" -DskipTests{code}
> Unfortunately my build is stuck on:
> {code:java}
> [INFO] --- maven-shade-plugin:3.2.1:shade (shade-flink) @
> flink-fs-hadoop-shaded ---
> [INFO] Including org.apache.hadoop:hadoop-common:jar:3.1.0 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-annotations:jar:3.1.0 in the shaded
> jar.
> [INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
> [INFO] Including commons-io:commons-io:jar:2.7 in the shaded jar.
> [INFO] Including commons-collections:commons-collections:jar:3.2.2 in the
> shaded jar.
> [INFO] Including commons-logging:commons-logging:jar:1.1.3 in the shaded jar.
> [INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
> [INFO] Including commons-beanutils:commons-beanutils:jar:1.9.3 in the shaded
> jar.
> [INFO] Including org.apache.commons:commons-configuration2:jar:2.1.1 in the
> shaded jar.
> [INFO] Including org.apache.commons:commons-lang3:jar:3.3.2 in the shaded jar.
> [INFO] Including com.google.re2j:re2j:jar:1.1 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-auth:jar:3.1.0 in the shaded jar.
> [INFO] Including org.apache.htrace:htrace-core4:jar:4.1.0-incubating in the
> shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.10.1 in
> the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.10.1 in
> the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.10.1 in the
> shaded jar.
> [INFO] Including org.codehaus.woodstox:stax2-api:jar:3.1.4 in the shaded jar.
> [INFO] Including com.fasterxml.woodstox:woodstox-core:jar:5.0.3 in the shaded
> jar.
> [INFO] Including org.apache.flink:force-shading:jar:1.12-SNAPSHOT in the
> shaded jar.
> [INFO] No artifact matching filter io.netty:netty
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [INFO] Replacing original artifact with shaded artifact.
> [INFO] Replacing
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/flink-fs-hadoop-shaded-1.12-SNAPSHOT.jar
> with
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/flink-fs-hadoop-shaded-1.12-SNAPSHOT-shaded.jar
> [INFO] Dependency-reduced POM written at:
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/dependency-reduced-pom.xml
> {code}
> Can we make flink compilation working with multiple maven threads ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-15736) Support Java 17 (LTS)
[ https://issues.apache.org/jira/browse/FLINK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648938#comment-17648938 ] Gunnar Morling commented on FLINK-15736: [~martijnvisser], thank you so much for the quick reply! I'll start by taking a look into FLINK-25003 then. > Support Java 17 (LTS) > - > > Key: FLINK-15736 > URL: https://issues.apache.org/jira/browse/FLINK-15736 > Project: Flink > Issue Type: New Feature > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > Long-term issue for preparing Flink for Java 17. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25003) RestClientTest#testConnectionTimeout fails on Java 17
[ https://issues.apache.org/jira/browse/FLINK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648939#comment-17648939 ] Gunnar Morling commented on FLINK-25003: I'll take a look, can any of the committers assign this one to me? // CC [~rmetzger] > RestClientTest#testConnectionTimeout fails on Java 17 > - > > Key: FLINK-25003 > URL: https://issues.apache.org/jira/browse/FLINK-25003 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST, Tests >Reporter: Chesnay Schepler >Priority: Major > > The test fails because the exception type has changed; with the returned > exception no longer being a ConnectException but just a plain > SocketException, presumably because the JDK fails earlier since it can't find > a route. > We could change the assertion, or change the test somehow to actually work > against a server which doesn't allow the establishment of a connection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30443) Expand list of sensitive keys
Gunnar Morling created FLINK-30443:
--
Summary: Expand list of sensitive keys
Key: FLINK-30443
URL: https://issues.apache.org/jira/browse/FLINK-30443
Project: Flink
Issue Type: Improvement
Components: API / Core
Reporter: Gunnar Morling
In {{{}GlobalConfiguration{}}}, there is [a list of known configuration
keys|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L47-L48]
whose values will be masked in log output. In our Flink deployment there's a
few more keys which we would like to mask, specifically, the following ones:
* "auth-params"
* "service-key"
* "token"
* "basic-auth"
* "jaas.config"
While those are somewhat use-case specific, I feel they are generic enough for
being added to that list, and there already is precedence in form of
"fs.azure.account.key". In that light, I don't think it's worth making this
somehow pluggable, but I'm curious what other folks here think. Thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30439) Update playgrounds for 1.16
[ https://issues.apache.org/jira/browse/FLINK-30439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648703#comment-17648703 ] Gunnar Morling commented on FLINK-30439: Exciting! Thanks, [~rmetzger]. > Update playgrounds for 1.16 > --- > > Key: FLINK-30439 > URL: https://issues.apache.org/jira/browse/FLINK-30439 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training >Reporter: David Anderson >Assignee: Gunnar Morling >Priority: Major > Labels: starter > Fix For: 1.16.0 > > > All of the playgrounds should be updated for Flink 1.16. This should include > reworking the code as necessary to avoid using anything that has been > deprecated. > See [https://github.com/apache/flink-playgrounds] for more info. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-15736) Support Java 17 (LTS)
[ https://issues.apache.org/jira/browse/FLINK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648701#comment-17648701 ] Gunnar Morling commented on FLINK-15736: Hey [~chesnay], this issue came up for our team recently, and I would love to help with making Flink run smoothly on Java 17 and beyond. What would be the recommended way for doing so from your PoV, is there e.g. any of the open sub-tasks I could look into first? > Support Java 17 (LTS) > - > > Key: FLINK-15736 > URL: https://issues.apache.org/jira/browse/FLINK-15736 > Project: Flink > Issue Type: New Feature > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > Long-term issue for preparing Flink for Java 17. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30439) Update playgrounds for 1.16
[ https://issues.apache.org/jira/browse/FLINK-30439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648696#comment-17648696 ] Gunnar Morling edited comment on FLINK-30439 at 12/16/22 3:42 PM: -- Got pointed to this one by [~danderson] and would love to give it a try. Seems I lack the right role to have issues assigned, so I hope this comment is enough for now. IIUC, an existing committer can assign it to me? was (Author: gunnar.morling): Got pointed to this one by [~danderson] and would love to give it a try. Seems I lack the right role to have issues assigned, so I hope this comment is enough for now. > Update playgrounds for 1.16 > --- > > Key: FLINK-30439 > URL: https://issues.apache.org/jira/browse/FLINK-30439 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training >Reporter: David Anderson >Priority: Major > Labels: starter > Fix For: 1.16.0 > > > All of the playgrounds should be updated for Flink 1.16. This should include > reworking the code as necessary to avoid using anything that has been > deprecated. > See [https://github.com/apache/flink-playgrounds] for more info. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30439) Update playgrounds for 1.16
[ https://issues.apache.org/jira/browse/FLINK-30439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648696#comment-17648696 ] Gunnar Morling commented on FLINK-30439: Got pointed to this one by [~danderson] and would love to give it a try. Seems I lack the right role to have issues assigned, so I hope this comment is enough for now. > Update playgrounds for 1.16 > --- > > Key: FLINK-30439 > URL: https://issues.apache.org/jira/browse/FLINK-30439 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training >Reporter: David Anderson >Priority: Major > Labels: starter > Fix For: 1.16.0 > > > All of the playgrounds should be updated for Flink 1.16. This should include > reworking the code as necessary to avoid using anything that has been > deprecated. > See [https://github.com/apache/flink-playgrounds] for more info. -- This message was sent by Atlassian Jira (v8.20.10#820010)
