[GitHub] [flink] flinkbot edited a comment on pull request #6787: [FLINK-8577][table] Implement proctime DataStream to Table upsert conversion
flinkbot edited a comment on pull request #6787: URL: https://github.com/apache/flink/pull/6787#issuecomment-679502953 ## CI report: * 71aa0e61536786901c6770adadd6f87b146a01db Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/181219241) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] klion26 commented on a change in pull request #13172: [FLINK-18854][docs-zh] Translate the 'API Migration Guides' page of 'Application Development' into Chinese
klion26 commented on a change in pull request #13172:
URL: https://github.com/apache/flink/pull/13172#discussion_r476181673
##
File path: docs/dev/migration.zh.md
##
@@ -25,18 +25,18 @@ under the License.
* This will be replaced by the TOC
{:toc}
-See the [older migration
-guide](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/migration.html)
-for information about migrating from older versions than Flink 1.3.
+有关从 Flink 1.3
之前版本迁移的信息，请参阅[旧版本迁移指南](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/migration.html)。
Review comment:
看上去这个链接也是有中文版的，这里我们换成中文版的话会更好一些吗？
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
flinkbot edited a comment on pull request #13235: URL: https://github.com/apache/flink/pull/13235#issuecomment-679625638 ## CI report: * 96c1c36cfd2ee8926f97dd48fa88d6fe3e0bdaba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5839) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liupc commented on pull request #13141: [FLINK-18852] Fix StreamScan doesn't inherit parallelism from input in legacy planner
liupc commented on pull request #13141: URL: https://github.com/apache/flink/pull/13141#issuecomment-679635208 gentle ping~ @wuchong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
flinkbot commented on pull request #13235: URL: https://github.com/apache/flink/pull/13235#issuecomment-679625638 ## CI report: * 96c1c36cfd2ee8926f97dd48fa88d6fe3e0bdaba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18974) Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183734#comment-17183734 ] Roc Marshal edited comment on FLINK-18974 at 8/25/20, 4:49 AM: --- Thanks for offering! [~ZhuShang] I'm willing to review it and I'll give response asap on the [Git Pull Request #13225|https://github.com/apache/flink/pull/13225]. was (Author: rocmarshal): [~ZhuShang] OK, I'm willing to review it and I'll give response asap on the [Git Pull Request #13225| https://github.com/apache/flink/pull/13225]. > Translate the 'User-Defined Functions' page of "Application Development's > DataStream API" into Chinese > -- > > Key: FLINK-18974 > URL: https://issues.apache.org/jira/browse/FLINK-18974 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: Pua >Priority: Major > Labels: Translation, pull-request-available, translation-zh > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/user_defined_functions.html > The markdown file is located in flink/docs/dev/user_defined_functions.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18974) Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183734#comment-17183734 ] Roc Marshal commented on FLINK-18974: - [~ZhuShang] OK, I'm willing to review it and I'll give response asap on the [Git Pull Request #13225| https://github.com/apache/flink/pull/13225]. > Translate the 'User-Defined Functions' page of "Application Development's > DataStream API" into Chinese > -- > > Key: FLINK-18974 > URL: https://issues.apache.org/jira/browse/FLINK-18974 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: Pua >Priority: Major > Labels: Translation, pull-request-available, translation-zh > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/user_defined_functions.html > The markdown file is located in flink/docs/dev/user_defined_functions.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
flinkbot commented on pull request #13235: URL: https://github.com/apache/flink/pull/13235#issuecomment-679610701 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 96c1c36cfd2ee8926f97dd48fa88d6fe3e0bdaba (Tue Aug 25 04:40:09 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19036) Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-19036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19036: --- Labels: pull-request-available translation-zh (was: translation-zh) > Translate page 'Application Profiling & Debugging' of 'Debugging & > Monitoring' into Chinese > --- > > Key: FLINK-19036 > URL: https://issues.apache.org/jira/browse/FLINK-19036 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: Roc Marshal >Priority: Major > Labels: pull-request-available, translation-zh > > The markdown file location: flink/docs/monitoring/application_profiling.zh.md > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] RocMarshal commented on pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
RocMarshal commented on pull request #13235: URL: https://github.com/apache/flink/pull/13235#issuecomment-679609380 Hi @xccui Could you help me to review this PR if you have free time? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal opened a new pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
RocMarshal opened a new pull request #13235: URL: https://github.com/apache/flink/pull/13235 ## What is the purpose of the change *Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese* ## Brief change log *Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese* - The markdown file location: flink/docs/monitoring/application_profiling.zh.md - The page url is https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html ## Verifying this change *A pure translation work in documentation module.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18909) SequenceGenerator doesn't work as expected
[
https://issues.apache.org/jira/browse/FLINK-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183720#comment-17183720
]
Jingsong Lee edited comment on FLINK-18909 at 8/25/20, 4:20 AM:
Thanks [~aljoscha] for reminding. Hi [~Liquidcoder] , SequenceGenerator can not
accept too huge range.
The {{SequenceGenerator}} and {{StatefulSequenceSource}} have the same
generator strategy. At first, they store all numbers to {{Deque
valuesToEmit}} . So if the range is to huge, there will be a OOM here... We can
have a better exception message.
I believe we can optimize it to not store all numbers, after the optimization,
it can accept Long.MAX_VALUE.
was (Author: lzljs3620320):
SequenceGenerator can not accept too huge range.
The {{SequenceGenerator}} and {{StatefulSequenceSource}} have the same
generator strategy. At first, they store all numbers to {{Deque
valuesToEmit}} . So if the range is to huge, there will be a OOM here... We can
have a better exception message.
I believe we can optimize it to not store all numbers, after the optimization,
it can accept Long.MAX_VALUE.
> SequenceGenerator doesn't work as expected
> --
>
> Key: FLINK-18909
> URL: https://issues.apache.org/jira/browse/FLINK-18909
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.1
>Reporter: Alan
>Priority: Minor
>
> When using a
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator
> with 0 and Long.MaxValue the following exception is rasied:
> <>
> new SequenceGenerator(0, Long.MAX_VALUE);
>
> Caused by: java.lang.IllegalArgumentExceptionCaused by:
> java.lang.IllegalArgumentException at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123) at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.safeDivide(SequenceGenerator.java:110)
> at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.open(SequenceGenerator.java:83)
> at
> org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.initializeState(DataGeneratorSource.java:64)
>
> This is because the Long overflows. in the check:
> long totalNoOfElements = Math.abs(end - start + 1);
> Which is incorrect. - this code should use BigInteger.
>
> Using
> new SequenceGenerator(0, Long.MAX_VALUE-1);
> Still fails on the following check:
> Caused by: java.lang.IllegalArgumentExceptionCaused by:
> java.lang.IllegalArgumentException at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123) at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.safeDivide(SequenceGenerator.java:111)
> at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.open(SequenceGenerator.java:83)
> at
> org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.initializeState(DataGeneratorSource.java:64)
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18909) SequenceGenerator doesn't work as expected
[
https://issues.apache.org/jira/browse/FLINK-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183720#comment-17183720
]
Jingsong Lee commented on FLINK-18909:
--
SequenceGenerator can not accept too huge range.
The {{SequenceGenerator}} and {{StatefulSequenceSource}} have the same
generator strategy. At first, they store all numbers to {{Deque
valuesToEmit}} . So if the range is to huge, there will be a OOM here... We can
have a better exception message.
I believe we can optimize it to not store all numbers, after the optimization,
it can accept Long.MAX_VALUE.
> SequenceGenerator doesn't work as expected
> --
>
> Key: FLINK-18909
> URL: https://issues.apache.org/jira/browse/FLINK-18909
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.1
>Reporter: Alan
>Priority: Minor
>
> When using a
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator
> with 0 and Long.MaxValue the following exception is rasied:
> <>
> new SequenceGenerator(0, Long.MAX_VALUE);
>
> Caused by: java.lang.IllegalArgumentExceptionCaused by:
> java.lang.IllegalArgumentException at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123) at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.safeDivide(SequenceGenerator.java:110)
> at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.open(SequenceGenerator.java:83)
> at
> org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.initializeState(DataGeneratorSource.java:64)
>
> This is because the Long overflows. in the check:
> long totalNoOfElements = Math.abs(end - start + 1);
> Which is incorrect. - this code should use BigInteger.
>
> Using
> new SequenceGenerator(0, Long.MAX_VALUE-1);
> Still fails on the following check:
> Caused by: java.lang.IllegalArgumentExceptionCaused by:
> java.lang.IllegalArgumentException at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123) at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.safeDivide(SequenceGenerator.java:111)
> at
> org.apache.flink.streaming.api.functions.source.datagen.SequenceGenerator.open(SequenceGenerator.java:83)
> at
> org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.initializeState(DataGeneratorSource.java:64)
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13227: [FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph
flinkbot edited a comment on pull request #13227: URL: https://github.com/apache/flink/pull/13227#issuecomment-679046587 ## CI report: * 213695b1635186819bcde49ed3c94f0238943ee2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5838) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17397) FileSystemTableFactory supports LookupableTableSource
[
https://issues.apache.org/jira/browse/FLINK-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183715#comment-17183715
]
Jingsong Lee commented on FLINK-17397:
--
Conceptually, support {{ScanTableSource}} for processing-time temporal join
looks better.
But, in terms of requirements, what do users want, or what scenarios do we want
to support?
Join a static bounded table, which has no lookup capability.
* If we really map it to "processing-time temporal join". First, we should
deal with batch source and streaming source in a Job. Second, the temporal join
operator needs wait batch source for receiving all static records.
Join a slowly changing bounded table, which has no lookup capability.
* If we really map it to "processing-time temporal join", actually, it is hard
to get change log from a filesystem bounded table, so how to support it?
Optimize it to lookup table source in plan? Send a flag to downstream operator
to tell it that it is time to reload, you should delete all old records?
It seems that we still lack a lot of design and implementation.
> FileSystemTableFactory supports LookupableTableSource
> --
>
> Key: FLINK-17397
> URL: https://issues.apache.org/jira/browse/FLINK-17397
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem
>Reporter: Jingsong Lee
>Priority: Major
> Labels: starter
> Fix For: 1.12.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #6787: [FLINK-8577][table] Implement proctime DataStream to Table upsert conversion
flinkbot edited a comment on pull request #6787: URL: https://github.com/apache/flink/pull/6787#issuecomment-679502953 ## CI report: * 71aa0e61536786901c6770adadd6f87b146a01db Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/181219241) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18974) Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183705#comment-17183705 ] Pua commented on FLINK-18974: - [~RocMarshal] could you or someone else hava a review? > Translate the 'User-Defined Functions' page of "Application Development's > DataStream API" into Chinese > -- > > Key: FLINK-18974 > URL: https://issues.apache.org/jira/browse/FLINK-18974 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: Pua >Priority: Major > Labels: Translation, pull-request-available, translation-zh > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/user_defined_functions.html > The markdown file is located in flink/docs/dev/user_defined_functions.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #6787: [FLINK-8577][table] Implement proctime DataStream to Table upsert conversion
flinkbot commented on pull request #6787: URL: https://github.com/apache/flink/pull/6787#issuecomment-679502953 ## CI report: * 71aa0e61536786901c6770adadd6f87b146a01db UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tobegit3hub commented on pull request #6787: [FLINK-8577][table] Implement proctime DataStream to Table upsert conversion
tobegit3hub commented on pull request #6787: URL: https://github.com/apache/flink/pull/6787#issuecomment-679489828 Any update for this PR? We are really looking for these APIs to convert append stream or upsert stream back to Table objects. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #13034: [FLINK-9992][tests] Fix FsStorageLocationReferenceTest#testEncodeAndDecode by adding retries to generate a valid path
dianfu commented on pull request #13034: URL: https://github.com/apache/flink/pull/13034#issuecomment-679479779 @kezhuw Thanks for your comments. - Regarding to "retrying 5 times", I think that's enough considering the frequency of this issue(it occurs two times in the last two weeks). - Regarding to use **generateRandomAlphanumericString** instead of **getRandomString**, I'm fine with this. However, it depends on the test purpose of this test case. For example, the generated string of **generateRandomAlphanumericString** only contains characters and digit, I'm not sure if this is what we want for this test case FsStorageLocationReferenceTest. What's your thought? @StephanEwen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18851) Add checkpoint type to checkpoint history entries in Web UI
[ https://issues.apache.org/jira/browse/FLINK-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183692#comment-17183692 ] goutham commented on FLINK-18851: - I can pick this up. pls assign this issue to me. > Add checkpoint type to checkpoint history entries in Web UI > --- > > Key: FLINK-18851 > URL: https://issues.apache.org/jira/browse/FLINK-18851 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Runtime / Web Frontend >Affects Versions: 1.12.0 >Reporter: Arvid Heise >Priority: Major > Labels: starter > Attachments: Checkpoint details.png > > > It would be helpful to users to better understand checkpointing times, if the > type of the checkpoint is displayed in the checkpoint history. > Possible types are savepoint, aligned checkpoint, unaligned checkpoint. > A possible place can be seen in the screenshot > !Checkpoint details.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
flinkbot edited a comment on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-667904442 ## CI report: * efe2b4b092cbce31dee74b4261ca7a20904b2000 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5766) * 0aa2e599db47cf035f6d7cf0389e7caa273b5880 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5837) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13227: [FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph
flinkbot edited a comment on pull request #13227: URL: https://github.com/apache/flink/pull/13227#issuecomment-679046587 ## CI report: * 17b779c8ed6e12edd9f6da41860a1b52c8a8390c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5817) * 213695b1635186819bcde49ed3c94f0238943ee2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5838) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13227: [FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph
flinkbot edited a comment on pull request #13227: URL: https://github.com/apache/flink/pull/13227#issuecomment-679046587 ## CI report: * 17b779c8ed6e12edd9f6da41860a1b52c8a8390c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5817) * 213695b1635186819bcde49ed3c94f0238943ee2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
flinkbot edited a comment on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-667904442 ## CI report: * efe2b4b092cbce31dee74b4261ca7a20904b2000 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5766) * 0aa2e599db47cf035f6d7cf0389e7caa273b5880 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18742) Some configuration args do not take effect at client
[
https://issues.apache.org/jira/browse/FLINK-18742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183685#comment-17183685
]
Matt Wang edited comment on FLINK-18742 at 8/25/20, 2:30 AM:
-
[~rmetzger] The `CoreOptions.CHECK_LEAKED_CLASSLOADER` is introduced by this
PR( [https://github.com/apache/flink/pull/13027).
|https://github.com/apache/flink/pull/13027]
It can work after removing this line
`[https://github.com/apache/flink/commit/d9af11d372cc76efefed50f544f67c45223b8bb5#diff-da45a35dae282e565a61be1441256883L148]`
was (Author: wangm92):
[~rmetzger] The `CoreOptions.CHECK_LEAKED_CLASSLOADER` is introduced by this
PR( [https://github.com/apache/flink/pull/13027).
|https://github.com/apache/flink/pull/13027]
It can work after removing this line
`[https://github.com/apache/flink/commit/d9af11d372cc76efefed50f544f67c45223b8bb5#diff-da45a35dae282e565a61be1441256883L148]`
[|https://github.com/apache/flink/pull/13027]
> Some configuration args do not take effect at client
>
>
> Key: FLINK-18742
> URL: https://issues.apache.org/jira/browse/FLINK-18742
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Affects Versions: 1.11.1
>Reporter: Matt Wang
>Assignee: Matt Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Some configuration args from command line will not work at client, for
> example, the job sets the {color:#505f79}_classloader.resolve-order_{color}
> to _{color:#505f79}parent-first,{color}_ it can work at TaskManager, but
> Client doesn't.
> The *FlinkUserCodeClassLoaders* will be created before calling the method of
> _{color:#505f79}getEffectiveConfiguration(){color}_ at
> {color:#505f79}org.apache.flink.client.cli.CliFrontend{color}, so the
> _{color:#505f79}Configuration{color}_ used by
> _{color:#505f79}PackagedProgram{color}_ does not include Configuration args.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19037) Introduce proper IO executor in Dispatcher
[
https://issues.apache.org/jira/browse/FLINK-19037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183687#comment-17183687
]
Yang Wang commented on FLINK-19037:
---
Maybe FLINK-17024 is also related. +1 to forward the
{{ClusterEntrypoint#ioExecutor}} to other components. After then, we could also
use the shared executor for Kubernetes client in {{KubernetesResourceManager}}.
> Introduce proper IO executor in Dispatcher
> --
>
> Key: FLINK-19037
> URL: https://issues.apache.org/jira/browse/FLINK-19037
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
>
> Currently, IO operations in the {{Dispatcher}} are scheduled on the
> {{rpcService.getExecutor()}}.
> We should introduce a separate executor for IO operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18742) Some configuration args do not take effect at client
[
https://issues.apache.org/jira/browse/FLINK-18742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183685#comment-17183685
]
Matt Wang commented on FLINK-18742:
---
[~rmetzger] The `CoreOptions.CHECK_LEAKED_CLASSLOADER` is introduced by this
PR( [https://github.com/apache/flink/pull/13027).
|https://github.com/apache/flink/pull/13027]
It can work after removing this line
`[https://github.com/apache/flink/commit/d9af11d372cc76efefed50f544f67c45223b8bb5#diff-da45a35dae282e565a61be1441256883L148]`
[|https://github.com/apache/flink/pull/13027]
> Some configuration args do not take effect at client
>
>
> Key: FLINK-18742
> URL: https://issues.apache.org/jira/browse/FLINK-18742
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Affects Versions: 1.11.1
>Reporter: Matt Wang
>Assignee: Matt Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Some configuration args from command line will not work at client, for
> example, the job sets the {color:#505f79}_classloader.resolve-order_{color}
> to _{color:#505f79}parent-first,{color}_ it can work at TaskManager, but
> Client doesn't.
> The *FlinkUserCodeClassLoaders* will be created before calling the method of
> _{color:#505f79}getEffectiveConfiguration(){color}_ at
> {color:#505f79}org.apache.flink.client.cli.CliFrontend{color}, so the
> _{color:#505f79}Configuration{color}_ used by
> _{color:#505f79}PackagedProgram{color}_ does not include Configuration args.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17397) FileSystemTableFactory supports LookupableTableSource
[
https://issues.apache.org/jira/browse/FLINK-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183684#comment-17183684
]
Jark Wu commented on FLINK-17397:
-
The fundamental need from the user and me is support to join (or called
"processing-time temporal join") a bounded side input. So I agree with you,
support {{ScanTableSource}} for processing-time temporal join can also resolve
the problem.
> FileSystemTableFactory supports LookupableTableSource
> --
>
> Key: FLINK-17397
> URL: https://issues.apache.org/jira/browse/FLINK-17397
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem
>Reporter: Jingsong Lee
>Priority: Major
> Labels: starter
> Fix For: 1.12.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17397) FileSystemTableFactory supports LookupableTableSource
[
https://issues.apache.org/jira/browse/FLINK-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183684#comment-17183684
]
Jark Wu edited comment on FLINK-17397 at 8/25/20, 2:28 AM:
---
The fundamental need from the user and me is support to join (or
"processing-time temporal join") a bounded side input. So I agree with you,
support {{ScanTableSource}} for processing-time temporal join can also resolve
the problem.
was (Author: jark):
The fundamental need from the user and me is support to join (or called
"processing-time temporal join") a bounded side input. So I agree with you,
support {{ScanTableSource}} for processing-time temporal join can also resolve
the problem.
> FileSystemTableFactory supports LookupableTableSource
> --
>
> Key: FLINK-17397
> URL: https://issues.apache.org/jira/browse/FLINK-17397
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem
>Reporter: Jingsong Lee
>Priority: Major
> Labels: starter
> Fix For: 1.12.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18959) Fail to archiveExecutionGraph because job is not finished when dispatcher close
[
https://issues.apache.org/jira/browse/FLINK-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183683#comment-17183683
]
Liu commented on FLINK-18959:
-
Thanks for the reply, [~trohrmann] and [~aljoscha]. After reverting the code in
MiniDispatcher, all tests run ok. But I also notice that something else should
be changed. In MiniDispatcher's method jobReachedGloballyTerminalState, the
cluster shuts down only if executionMode is DETACHED. Upon cancellation, the
dispatcher will not shut down if executionMode is NORMAL. So we should shut
down cluster no matter executionMode is DETACHED or NORMAL.
> Fail to archiveExecutionGraph because job is not finished when dispatcher
> close
> ---
>
> Key: FLINK-18959
> URL: https://issues.apache.org/jira/browse/FLINK-18959
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0, 1.12.0, 1.11.1
>Reporter: Liu
>Assignee: Liu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2, 1.10.3
>
> Attachments: flink-debug-log
>
>
> When job is cancelled, we expect to see it in flink's history server. But I
> can not see my job after it is cancelled.
> After digging into the problem, I find that the function
> archiveExecutionGraph is not executed. Below is the brief log:
> {panel:title=log}
> 2020-08-14 15:10:06,406 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher- 15] - Job EtlAndWindow
> (6f784d4cc5bae88a332d254b21660372) switched from state RUNNING to CANCELLING.
> 2020-08-14 15:10:06,415 DEBUG
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Shutting down per-job cluster
> because the job was canceled.
> 2020-08-14 15:10:06,629 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Stopping dispatcher
> akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
> 2020-08-14 15:10:06,629 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Stopping all currently running jobs
> of dispatcher akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
> 2020-08-14 15:10:06,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [flink-akka.actor.default-dispatcher-29] - Stopping the JobMaster for job
> EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
> 2020-08-14 15:10:06,632 DEBUG org.apache.flink.runtime.jobmaster.JobMaster
> [flink-akka.actor.default-dispatcher-29] - Disconnect TaskExecutor
> container_e144_1590060720089_2161_01_06 because: Stopping JobMaster for
> job EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
> 2020-08-14 15:10:06,646 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher-29] - Job EtlAndWindow
> (6f784d4cc5bae88a332d254b21660372) switched from state CANCELLING to CANCELED.
> 2020-08-14 15:10:06,664 DEBUG
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-4] - There is a newer JobManagerRunner
> for the job 6f784d4cc5bae88a332d254b21660372.
> {panel}
> From the log, we can see that job is not finished when dispatcher closes. The
> process is as following:
> * Receive cancel command and send it to all tasks async.
> * In MiniDispatcher, begin to shutting down per-job cluster.
> * Stopping dispatcher and remove job.
> * Job is cancelled and callback is executed in method startJobManagerRunner.
> * Because job is removed before, so currentJobManagerRunner is null which
> not equals to the original jobManagerRunner. In this case,
> archivedExecutionGraph will not be uploaded.
> In normal cases, I find that job is cancelled first and then dispatcher is
> stopped so that archivedExecutionGraph will succeed. But I think that the
> order is not constrained and it is hard to know which comes first.
> Above is what I suspected. If so, then we should fix it.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17330) Avoid scheduling deadlocks caused by cyclic input dependencies between regions
[
https://issues.apache.org/jira/browse/FLINK-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu closed FLINK-17330.
---
Resolution: Fixed
Done via
c2bb9808178f9b28742a0738280552df209f
c0f382f5f0072441ef8933f6993f1c34168004d6
> Avoid scheduling deadlocks caused by cyclic input dependencies between regions
> --
>
> Key: FLINK-17330
> URL: https://issues.apache.org/jira/browse/FLINK-17330
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Imagine a job like this:
> A -- (pipelined FORWARD) --> B -- (blocking ALL-to-ALL) --> D
> A -- (pipelined FORWARD) --> C -- (pipelined FORWARD) --> D
> parallelism=2 for all vertices.
> We will have 2 execution pipelined regions:
> R1 = {A1, B1, C1, D1}
> R2 = {A2, B2, C2, D2}
> R1 has a cross-region input edge (B2->D1).
> R2 has a cross-region input edge (B1->D2).
> Scheduling deadlock will happen since we schedule a region only when all its
> inputs are consumable (i.e. blocking partitions to be finished). This is
> because R1 can be scheduled only if R2 finishes, while R2 can be scheduled
> only if R1 finishes.
> To avoid this, one solution is to force a logical pipelined region with
> intra-region ALL-to-ALL blocking edges to form one only execution pipelined
> region, so that there would not be cyclic input dependency between regions.
> Besides that, we should also pay attention to avoid cyclic cross-region
> POINTWISE blocking edges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk merged pull request #13205: [FLINK-17330[runtime] Merge cyclic dependent pipelined regions into one region
zhuzhurk merged pull request #13205: URL: https://github.com/apache/flink/pull/13205 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] danny0405 commented on a change in pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
danny0405 commented on a change in pull request #13050:
URL: https://github.com/apache/flink/pull/13050#discussion_r476063925
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##
@@ -70,30 +75,43 @@ public static Expander create(FlinkPlannerImpl planner) {
/** Expands identifiers in a given SQL string, returning a {@link
Expanded}. */
public Expanded expanded(String ori) {
final Map identifiers = new
HashMap<>();
+ final Map funcNameToId = new HashMap<>();
Review comment:
Because the `BridgingSqlFunction` lost the parser position information.
We may need a refactor to `BridgingSqlFunction` but i don't want to in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-web] zhuzhurk merged pull request #366: Add Apache Flink release 1.10.2
zhuzhurk merged pull request #366: URL: https://github.com/apache/flink-web/pull/366 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zhuzhurk commented on pull request #366: Add Apache Flink release 1.10.2
zhuzhurk commented on pull request #366: URL: https://github.com/apache/flink-web/pull/366#issuecomment-679459626 Thanks for reviewing @uce @Myasuka We can announce the release today. I have updated the date to 2020-08-25 and will merge the change soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] danny0405 commented on a change in pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
danny0405 commented on a change in pull request #13050:
URL: https://github.com/apache/flink/pull/13050#discussion_r476063925
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##
@@ -70,30 +75,43 @@ public static Expander create(FlinkPlannerImpl planner) {
/** Expands identifiers in a given SQL string, returning a {@link
Expanded}. */
public Expanded expanded(String ori) {
final Map identifiers = new
HashMap<>();
+ final Map funcNameToId = new HashMap<>();
Review comment:
Because the `BridgingSqlFunction` lost the parser position information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19012) E2E test fails with "Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument."
[
https://issues.apache.org/jira/browse/FLINK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183673#comment-17183673
]
Dian Fu commented on FLINK-19012:
-
Upgrade to "Critical" as this issue seems not occur accidentally and it has
occurred several times these days.
> E2E test fails with "Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument."
> ---
>
> Key: FLINK-19012
> URL: https://issues.apache.org/jira/browse/FLINK-19012
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> Note: This error occurred in a custom branch with unreviewed changes. I don't
> believe my changes affect this error, but I would keep this in mind when
> investigating the error:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8307=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=0d2e35fc-a330-5cf2-a012-7267e2667b1d
>
> {code}
> 2020-08-20T20:55:30.2400645Z 2020-08-20 20:55:22,373 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Registering
> task at network: Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) [DEPLOYING].
> 2020-08-20T20:55:30.2402392Z 2020-08-20 20:55:22,401 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask [] - No state
> backend has been configured, using default (Memory / JobManager)
> MemoryStateBackend (data in heap memory / checkpoints to JobManager)
> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
> 5242880)
> 2020-08-20T20:55:30.2404297Z 2020-08-20 20:55:22,413 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to RUNNING.
> 2020-08-20T20:55:30.2405805Z 2020-08-20 20:55:22,786 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Pinging Elasticsearch cluster via hosts [http://127.0.0.1:9200] ...
> 2020-08-20T20:55:30.2407027Z 2020-08-20 20:55:22,848 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Elasticsearch RestHighLevelClient is connected to
> [http://127.0.0.1:9200]
> 2020-08-20T20:55:30.2409277Z 2020-08-20 20:55:29,205 INFO
> org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl
> [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) discarding 0
> drained requests
> 2020-08-20T20:55:30.2410690Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FINISHED.
> 2020-08-20T20:55:30.2412187Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Freeing
> task resources for Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0).
> 2020-08-20T20:55:30.2414203Z 2020-08-20 20:55:29,224 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
> Un-registering task and sending final execution state FINISHED to JobManager
> for task Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> cbc357ccb763df2852fee8c4fc7d55f2_0_0.
> 2020-08-20T20:55:30.2415602Z 2020-08-20 20:55:29,219 INFO
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1) - asynchronous part of
> checkpoint 1 could not be completed.
> 2020-08-20T20:55:30.2416411Z java.io.UncheckedIOException:
> java.io.IOException: Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument.
> 2020-08-20T20:55:30.2418956Z at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.lambda$registerConsumer$2(SubtaskCheckpointCoordinatorImpl.java:468)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420100Z at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:91)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420927Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421455Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421879Z at
[jira] [Updated] (FLINK-19012) E2E test fails with "Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument."
[
https://issues.apache.org/jira/browse/FLINK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19012:
Fix Version/s: 1.12.0
> E2E test fails with "Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument."
> ---
>
> Key: FLINK-19012
> URL: https://issues.apache.org/jira/browse/FLINK-19012
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> Note: This error occurred in a custom branch with unreviewed changes. I don't
> believe my changes affect this error, but I would keep this in mind when
> investigating the error:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8307=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=0d2e35fc-a330-5cf2-a012-7267e2667b1d
>
> {code}
> 2020-08-20T20:55:30.2400645Z 2020-08-20 20:55:22,373 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Registering
> task at network: Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) [DEPLOYING].
> 2020-08-20T20:55:30.2402392Z 2020-08-20 20:55:22,401 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask [] - No state
> backend has been configured, using default (Memory / JobManager)
> MemoryStateBackend (data in heap memory / checkpoints to JobManager)
> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
> 5242880)
> 2020-08-20T20:55:30.2404297Z 2020-08-20 20:55:22,413 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to RUNNING.
> 2020-08-20T20:55:30.2405805Z 2020-08-20 20:55:22,786 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Pinging Elasticsearch cluster via hosts [http://127.0.0.1:9200] ...
> 2020-08-20T20:55:30.2407027Z 2020-08-20 20:55:22,848 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Elasticsearch RestHighLevelClient is connected to
> [http://127.0.0.1:9200]
> 2020-08-20T20:55:30.2409277Z 2020-08-20 20:55:29,205 INFO
> org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl
> [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) discarding 0
> drained requests
> 2020-08-20T20:55:30.2410690Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FINISHED.
> 2020-08-20T20:55:30.2412187Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Freeing
> task resources for Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0).
> 2020-08-20T20:55:30.2414203Z 2020-08-20 20:55:29,224 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
> Un-registering task and sending final execution state FINISHED to JobManager
> for task Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> cbc357ccb763df2852fee8c4fc7d55f2_0_0.
> 2020-08-20T20:55:30.2415602Z 2020-08-20 20:55:29,219 INFO
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1) - asynchronous part of
> checkpoint 1 could not be completed.
> 2020-08-20T20:55:30.2416411Z java.io.UncheckedIOException:
> java.io.IOException: Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument.
> 2020-08-20T20:55:30.2418956Z at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.lambda$registerConsumer$2(SubtaskCheckpointCoordinatorImpl.java:468)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420100Z at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:91)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420927Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421455Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421879Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2422348Z Caused by: java.io.IOException: Cannot register
>
[jira] [Updated] (FLINK-19012) E2E test fails with "Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument."
[
https://issues.apache.org/jira/browse/FLINK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19012:
Priority: Critical (was: Major)
> E2E test fails with "Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument."
> ---
>
> Key: FLINK-19012
> URL: https://issues.apache.org/jira/browse/FLINK-19012
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> Note: This error occurred in a custom branch with unreviewed changes. I don't
> believe my changes affect this error, but I would keep this in mind when
> investigating the error:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8307=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=0d2e35fc-a330-5cf2-a012-7267e2667b1d
>
> {code}
> 2020-08-20T20:55:30.2400645Z 2020-08-20 20:55:22,373 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Registering
> task at network: Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) [DEPLOYING].
> 2020-08-20T20:55:30.2402392Z 2020-08-20 20:55:22,401 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask [] - No state
> backend has been configured, using default (Memory / JobManager)
> MemoryStateBackend (data in heap memory / checkpoints to JobManager)
> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
> 5242880)
> 2020-08-20T20:55:30.2404297Z 2020-08-20 20:55:22,413 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to RUNNING.
> 2020-08-20T20:55:30.2405805Z 2020-08-20 20:55:22,786 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Pinging Elasticsearch cluster via hosts [http://127.0.0.1:9200] ...
> 2020-08-20T20:55:30.2407027Z 2020-08-20 20:55:22,848 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Elasticsearch RestHighLevelClient is connected to
> [http://127.0.0.1:9200]
> 2020-08-20T20:55:30.2409277Z 2020-08-20 20:55:29,205 INFO
> org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl
> [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) discarding 0
> drained requests
> 2020-08-20T20:55:30.2410690Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FINISHED.
> 2020-08-20T20:55:30.2412187Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Freeing
> task resources for Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0).
> 2020-08-20T20:55:30.2414203Z 2020-08-20 20:55:29,224 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
> Un-registering task and sending final execution state FINISHED to JobManager
> for task Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> cbc357ccb763df2852fee8c4fc7d55f2_0_0.
> 2020-08-20T20:55:30.2415602Z 2020-08-20 20:55:29,219 INFO
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1) - asynchronous part of
> checkpoint 1 could not be completed.
> 2020-08-20T20:55:30.2416411Z java.io.UncheckedIOException:
> java.io.IOException: Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument.
> 2020-08-20T20:55:30.2418956Z at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.lambda$registerConsumer$2(SubtaskCheckpointCoordinatorImpl.java:468)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420100Z at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:91)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420927Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421455Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421879Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2422348Z Caused by: java.io.IOException: Cannot register
> Closeable, this
[GitHub] [flink] zhoulii commented on pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on pull request #12899: URL: https://github.com/apache/flink/pull/12899#issuecomment-679456963 Hi @tillrohrmann , thanks for reviewing this pr. I add some description about how to use Secrets in native_kubernetes.md and update the code on your advice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17274) Maven: Premature end of Content-Length delimited message body
[
https://issues.apache.org/jira/browse/FLINK-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183671#comment-17183671
]
Dian Fu commented on FLINK-17274:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5834=logs=af0c3dd6-ccea-53d1-d352-344c568905e4=f898bece-d8f3-5fab-10f5-eacbefdb2d1b]
> Maven: Premature end of Content-Length delimited message body
> -
>
> Key: FLINK-17274
> URL: https://issues.apache.org/jira/browse/FLINK-17274
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> CI:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=7786=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb
> {code}
> [ERROR] Failed to execute goal on project
> flink-connector-elasticsearch7_2.11: Could not resolve dependencies for
> project
> org.apache.flink:flink-connector-elasticsearch7_2.11:jar:1.11-SNAPSHOT: Could
> not transfer artifact org.apache.lucene:lucene-sandbox:jar:8.3.0 from/to
> alicloud-mvn-mirror
> (http://mavenmirror.alicloud.dak8s.net:/repository/maven-central/): GET
> request of: org/apache/lucene/lucene-sandbox/8.3.0/lucene-sandbox-8.3.0.jar
> from alicloud-mvn-mirror failed: Premature end of Content-Length delimited
> message body (expected: 289920; received: 239832 -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19012) E2E test fails with "Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument."
[
https://issues.apache.org/jira/browse/FLINK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183670#comment-17183670
]
Dian Fu commented on FLINK-19012:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5834=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529]
> E2E test fails with "Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument."
> ---
>
> Key: FLINK-19012
> URL: https://issues.apache.org/jira/browse/FLINK-19012
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> Note: This error occurred in a custom branch with unreviewed changes. I don't
> believe my changes affect this error, but I would keep this in mind when
> investigating the error:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8307=logs=1f3ed471-1849-5d3c-a34c-19792af4ad16=0d2e35fc-a330-5cf2-a012-7267e2667b1d
>
> {code}
> 2020-08-20T20:55:30.2400645Z 2020-08-20 20:55:22,373 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Registering
> task at network: Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) [DEPLOYING].
> 2020-08-20T20:55:30.2402392Z 2020-08-20 20:55:22,401 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask [] - No state
> backend has been configured, using default (Memory / JobManager)
> MemoryStateBackend (data in heap memory / checkpoints to JobManager)
> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
> 5242880)
> 2020-08-20T20:55:30.2404297Z 2020-08-20 20:55:22,413 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to RUNNING.
> 2020-08-20T20:55:30.2405805Z 2020-08-20 20:55:22,786 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Pinging Elasticsearch cluster via hosts [http://127.0.0.1:9200] ...
> 2020-08-20T20:55:30.2407027Z 2020-08-20 20:55:22,848 INFO
> org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge
> [] - Elasticsearch RestHighLevelClient is connected to
> [http://127.0.0.1:9200]
> 2020-08-20T20:55:30.2409277Z 2020-08-20 20:55:29,205 INFO
> org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl
> [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) discarding 0
> drained requests
> 2020-08-20T20:55:30.2410690Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FINISHED.
> 2020-08-20T20:55:30.2412187Z 2020-08-20 20:55:29,218 INFO
> org.apache.flink.runtime.taskmanager.Task[] - Freeing
> task resources for Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> (cbc357ccb763df2852fee8c4fc7d55f2_0_0).
> 2020-08-20T20:55:30.2414203Z 2020-08-20 20:55:29,224 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
> Un-registering task and sending final execution state FINISHED to JobManager
> for task Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1)
> cbc357ccb763df2852fee8c4fc7d55f2_0_0.
> 2020-08-20T20:55:30.2415602Z 2020-08-20 20:55:29,219 INFO
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source:
> Sequence Source -> Flat Map -> Sink: Unnamed (1/1) - asynchronous part of
> checkpoint 1 could not be completed.
> 2020-08-20T20:55:30.2416411Z java.io.UncheckedIOException:
> java.io.IOException: Cannot register Closeable, this
> subtaskCheckpointCoordinator is already closed. Closing argument.
> 2020-08-20T20:55:30.2418956Z at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.lambda$registerConsumer$2(SubtaskCheckpointCoordinatorImpl.java:468)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420100Z at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:91)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-08-20T20:55:30.2420927Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421455Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_265]
> 2020-08-20T20:55:30.2421879Z at
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899: URL: https://github.com/apache/flink/pull/12899#discussion_r476053520 ## File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSecretEnvVar.java ## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.kubernetes.kubeclient.resources; + +import io.fabric8.kubernetes.api.model.EnvVar; +import io.fabric8.kubernetes.api.model.EnvVarBuilder; + +import java.util.Map; + +/** + * Represent EnvVar resource in kubernetes. Review comment: done in https://github.com/apache/flink/pull/12899/files#diff-cdc2fed9ec3c1184d4011356be0bdc65R27 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19041) Add dependency management for ConnectedStream in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuiqiang Chen updated FLINK-19041: --- Description: We failed to set merged configurations into DataStreamTwoInputPythonStatelessFunctionOperator when finally generating the StreamGraph. (was: There is a bug that we forget to set merged configurations into DataStreamTwoInputPythonStatelessFunctionOperator when finally generating the StreamGraph.) > Add dependency management for ConnectedStream in Python DataStream API. > --- > > Key: FLINK-19041 > URL: https://issues.apache.org/jira/browse/FLINK-19041 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Shuiqiang Chen >Priority: Major > Fix For: 1.12.0 > > > We failed to set merged configurations into > DataStreamTwoInputPythonStatelessFunctionOperator when finally generating the > StreamGraph. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899: URL: https://github.com/apache/flink/pull/12899#discussion_r476053086 ## File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesParameters.java ## @@ -110,4 +110,14 @@ * The local directory to locate the custom Hadoop configuration. */ Optional getLocalHadoopConfigurationDirectory(); + + /** +* A collection of Secrets and path that are mounted to the JobManager and TaskManager Container(s). +*/ + Map getSecretNamesToMountPaths(); + + /** +* A collection of customized environments that are attached to the JobManager and TaskManager Container(s). Review comment: done in https://github.com/apache/flink/pull/12899/files#diff-2f05594496213e9ac669e519d48c462dR120 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899: URL: https://github.com/apache/flink/pull/12899#discussion_r476052791 ## File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesParameters.java ## @@ -110,4 +110,14 @@ * The local directory to locate the custom Hadoop configuration. */ Optional getLocalHadoopConfigurationDirectory(); + + /** +* A collection of Secrets and path that are mounted to the JobManager and TaskManager Container(s). Review comment: done in https://github.com/apache/flink/pull/12899/files#diff-2f05594496213e9ac669e519d48c462dR115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899: URL: https://github.com/apache/flink/pull/12899#discussion_r476051896 ## File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/MountSecretsDecorator.java ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.kubernetes.kubeclient.decorators; + +import org.apache.flink.kubernetes.kubeclient.FlinkPod; +import org.apache.flink.kubernetes.kubeclient.parameters.AbstractKubernetesParameters; + +import io.fabric8.kubernetes.api.model.Container; +import io.fabric8.kubernetes.api.model.ContainerBuilder; +import io.fabric8.kubernetes.api.model.Pod; +import io.fabric8.kubernetes.api.model.PodBuilder; +import io.fabric8.kubernetes.api.model.Volume; +import io.fabric8.kubernetes.api.model.VolumeBuilder; +import io.fabric8.kubernetes.api.model.VolumeMount; +import io.fabric8.kubernetes.api.model.VolumeMountBuilder; + +import static org.apache.flink.util.Preconditions.checkNotNull; + +/** + * support mounting Secrets on the JobManager or TaskManager pod.. Review comment: done in https://github.com/apache/flink/pull/12899/files#diff-0e5823ba44ea4819a4c33767020ee54bR36 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899: URL: https://github.com/apache/flink/pull/12899#discussion_r476052224 ## File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/EnvSecretsDecorator.java ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.kubernetes.kubeclient.decorators; + +import org.apache.flink.kubernetes.kubeclient.FlinkPod; +import org.apache.flink.kubernetes.kubeclient.parameters.AbstractKubernetesParameters; +import org.apache.flink.kubernetes.kubeclient.resources.KubernetesSecretEnvVar; + +import io.fabric8.kubernetes.api.model.Container; +import io.fabric8.kubernetes.api.model.ContainerBuilder; +import io.fabric8.kubernetes.api.model.EnvVar; + +import java.util.List; +import java.util.stream.Collectors; + +import static org.apache.flink.util.Preconditions.checkNotNull; + +/** + * support setting environment variables via Secrets. Review comment: done in https://github.com/apache/flink/pull/12899/files#diff-5937442e6c2456441334306f869a6121R35 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899:
URL: https://github.com/apache/flink/pull/12899#discussion_r476051560
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##
@@ -221,6 +221,21 @@
/** Defines the configuration key of that external resource in
Kubernetes. This is used as a suffix in an actual config. */
public static final String
EXTERNAL_RESOURCE_KUBERNETES_CONFIG_KEY_SUFFIX = "kubernetes.config-key";
+ public static final ConfigOption>
KUBERNETES_SECRETS =
+ key("kubernetes.secrets")
+ .mapType()
+ .noDefaultValue()
+ .withDescription("The user-specified secrets that will
be mounted into Flink container. The value should be in " +
+ "the form of
foo:/opt/secrets-foo,bar:/opt/secrets-bar.");
+
+ public static final ConfigOption>>
KUBERNETES_ENV_SECRET_KEY_REF =
+ key("kubernetes.env.secretKeyRef")
+ .mapType()
+ .asList()
+ .noDefaultValue()
+ .withDescription("The user-specified secrets to set env
variables in Flink container. The value should be in " +
+ "the form of
env:FOO_ENV,secret:foo_secret,key:foo_key;env:BAR_ENV,secret:bar_secret,key:bar_key.");
Review comment:
done in
https://github.com/apache/flink/pull/12899/files#diff-1cd32b8ad754e869567bc0d78d39139fR236
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
zhoulii commented on a change in pull request #12899:
URL: https://github.com/apache/flink/pull/12899#discussion_r476050973
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##
@@ -221,6 +221,21 @@
/** Defines the configuration key of that external resource in
Kubernetes. This is used as a suffix in an actual config. */
public static final String
EXTERNAL_RESOURCE_KUBERNETES_CONFIG_KEY_SUFFIX = "kubernetes.config-key";
+ public static final ConfigOption>
KUBERNETES_SECRETS =
+ key("kubernetes.secrets")
+ .mapType()
+ .noDefaultValue()
+ .withDescription("The user-specified secrets that will
be mounted into Flink container. The value should be in " +
+ "the form of
foo:/opt/secrets-foo,bar:/opt/secrets-bar.");
Review comment:
add some description in
[native_kubernetes.md](https://github.com/apache/flink/pull/12899/files#diff-8ac5ff0acb9a268297bd157249c52acdR214)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-19041) Add dependency management for ConnectedStream in Python DataStream API.
Shuiqiang Chen created FLINK-19041: -- Summary: Add dependency management for ConnectedStream in Python DataStream API. Key: FLINK-19041 URL: https://issues.apache.org/jira/browse/FLINK-19041 Project: Flink Issue Type: Bug Components: API / Python Reporter: Shuiqiang Chen Fix For: 1.12.0 There is a bug that we forget to set merged configurations into DataStreamTwoInputPythonStatelessFunctionOperator when finally generating the StreamGraph. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * 67942d0ea538985ec25cecf7cefdc1bec0c1b53b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5833) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19005) used metaspace grow on every execution
[
https://issues.apache.org/jira/browse/FLINK-19005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183623#comment-17183623
]
Chesnay Schepler commented on FLINK-19005:
--
My conclusion is that Flink is not leaking anything, and the errors are due to
unfortunate timings or some JDK issue.
I was able to reproduce the issue when submitting jobs in directly after
another / with 5 seconds in between, but after increasing the backoff to 1
minute the OOM no longer occurred. The GC states also showed that the Metaspace
usage did not continuously increase; the GC created distinct dips that
frequently managed to match or even undercut prior dips.
[Stephans
comment|https://issues.apache.org/jira/browse/FLINK-16408?focusedCommentId=17180577=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17180577]
appears to apply here, at the very least for all the mentioned cases where
Wordcounts are frequently run.
As for the original issue by [~gestevez], this looks like a clear case of
classloaders being leaked. There are (at least) a bunch of
{{oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser}}
threads hanging around preventing the garbage collection.
So technically, this is a thread leak inherent to this library or caused by
improper usage.
> used metaspace grow on every execution
> --
>
> Key: FLINK-19005
> URL: https://issues.apache.org/jira/browse/FLINK-19005
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Runtime / Configuration,
> Runtime / Coordination
>Affects Versions: 1.11.1
>Reporter: Guillermo Sánchez
>Assignee: Chesnay Schepler
>Priority: Major
> Attachments: heap_dump_after_10_executions.zip,
> heap_dump_after_1_execution.zip
>
>
> Hi !
> Im running a 1.11.1 flink cluster, where I execute batch jobs made with
> DataSet API.
> I submit these jobs every day to calculate daily data.
> In every execution, cluster's used metaspace increase by 7MB and its never
> released.
> This ends up with an OutOfMemoryError caused by Metaspace every 15 days and i
> need to restart the cluster to clean the metaspace
> taskmanager.memory.jvm-metaspace.size is set to 512mb
> Any idea of what could be causing this metaspace grow and why is it not
> released ?
>
>
> === Summary ==
>
> Case 1, reported by [~gestevez]:
> * Flink 1.11.1
> * Java 11
> * Maximum Metaspace size set to 512mb
> * Custom Batch job, submitted daily
> * Requires restart every 15 days after an OOM
> Case 2, reported by [~Echo Lee]:
> * Flink 1.11.0
> * Java 11
> * G1GC
> * WordCount Batch job, submitted every second / every 5 minutes
> * eventually fails TaskExecutor with OOM
> Case 3, reported by [~DaDaShen]
> * Flink 1.11.0
> * Java 11
> * WordCount Batch job, submitted every 5 seconds
> * growing Metaspace, eventually OOM
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13205: [FLINK-17330[runtime] Merge cyclic dependent pipelined regions into one region
flinkbot edited a comment on pull request #13205: URL: https://github.com/apache/flink/pull/13205#issuecomment-677463515 ## CI report: * 8617ce6c0c5e0b4e8ba01aae58b1ef5cff2f02e6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5827) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
flinkbot edited a comment on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679234273 ## CI report: * f2538ea4babc43852365b6a3fde86f823873106c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5826) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * f0bb8a255816919a6578a6994499757075ab371b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5819) * 67942d0ea538985ec25cecf7cefdc1bec0c1b53b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5833) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * f0bb8a255816919a6578a6994499757075ab371b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5819) * 67942d0ea538985ec25cecf7cefdc1bec0c1b53b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13231: [FLINK-19031] Remove deprecated StreamExecutionEnvironment#setStateBackend(AbstactStateBackend)
flinkbot edited a comment on pull request #13231: URL: https://github.com/apache/flink/pull/13231#issuecomment-679197180 ## CI report: * faa0726d771536cfdb669d86bc5310e82946f527 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5824) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13217: [FLINK-16866] Make job submission non-blocking
flinkbot edited a comment on pull request #13217: URL: https://github.com/apache/flink/pull/13217#issuecomment-678285884 ## CI report: * 3655fcea1966bfbcb85c86d6a159c354f20d6cc7 UNKNOWN * 52a49f0b0840aa9220de72d64c50a6b33f6adf92 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5832) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13217: [FLINK-16866] Make job submission non-blocking
flinkbot edited a comment on pull request #13217: URL: https://github.com/apache/flink/pull/13217#issuecomment-678285884 ## CI report: * 3655fcea1966bfbcb85c86d6a159c354f20d6cc7 UNKNOWN * 9127b879bec8dc1c05176c1acb89680c0e7baf04 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5807) * 52a49f0b0840aa9220de72d64c50a6b33f6adf92 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13232: [FLINK-18951][python][docs] Add documentation for Configurations in P…
flinkbot edited a comment on pull request #13232: URL: https://github.com/apache/flink/pull/13232#issuecomment-679225070 ## CI report: * dd7743539c7ed5c45d93fc8b3c12e2db8b7af65e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5825) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #13217: [FLINK-16866] Make job submission non-blocking
rmetzger commented on a change in pull request #13217:
URL: https://github.com/apache/flink/pull/13217#discussion_r475791446
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##
@@ -350,71 +341,42 @@ private boolean isPartialResourceConfigured(JobGraph
jobGraph) {
}, getRpcService().getExecutor());
}
- private CompletableFuture persistAndRunJob(JobGraph jobGraph)
throws Exception {
+ private Void persistAndRunJob(JobGraph jobGraph) throws Exception {
Review comment:
Ah, very nice. Thanks a lot!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Reopened] (FLINK-18742) Some configuration args do not take effect at client
[
https://issues.apache.org/jira/browse/FLINK-18742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger reopened FLINK-18742:
I reverted the commit on release-1.11 because it broke the branch:
https://github.com/apache/flink/commit/d9af11d372cc76efefed50f544f67c45223b8bb5
See error details in:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5830=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb
> Some configuration args do not take effect at client
>
>
> Key: FLINK-18742
> URL: https://issues.apache.org/jira/browse/FLINK-18742
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Affects Versions: 1.11.1
>Reporter: Matt Wang
>Assignee: Matt Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Some configuration args from command line will not work at client, for
> example, the job sets the {color:#505f79}_classloader.resolve-order_{color}
> to _{color:#505f79}parent-first,{color}_ it can work at TaskManager, but
> Client doesn't.
> The *FlinkUserCodeClassLoaders* will be created before calling the method of
> _{color:#505f79}getEffectiveConfiguration(){color}_ at
> {color:#505f79}org.apache.flink.client.cli.CliFrontend{color}, so the
> _{color:#505f79}Configuration{color}_ used by
> _{color:#505f79}PackagedProgram{color}_ does not include Configuration args.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18742) Some configuration args do not take effect at client
[
https://issues.apache.org/jira/browse/FLINK-18742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18742:
---
Fix Version/s: (was: 1.11.2)
> Some configuration args do not take effect at client
>
>
> Key: FLINK-18742
> URL: https://issues.apache.org/jira/browse/FLINK-18742
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Affects Versions: 1.11.1
>Reporter: Matt Wang
>Assignee: Matt Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Some configuration args from command line will not work at client, for
> example, the job sets the {color:#505f79}_classloader.resolve-order_{color}
> to _{color:#505f79}parent-first,{color}_ it can work at TaskManager, but
> Client doesn't.
> The *FlinkUserCodeClassLoaders* will be created before calling the method of
> _{color:#505f79}getEffectiveConfiguration(){color}_ at
> {color:#505f79}org.apache.flink.client.cli.CliFrontend{color}, so the
> _{color:#505f79}Configuration{color}_ used by
> _{color:#505f79}PackagedProgram{color}_ does not include Configuration args.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13234: [FLINK-18905][task] Allow SourceOperator chaining with MultipleInputStreamTask
flinkbot edited a comment on pull request #13234: URL: https://github.com/apache/flink/pull/13234#issuecomment-679243226 ## CI report: * e1c13bbc1899c7c2d7e94d26cdeb47deddf76b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5828) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13205: [FLINK-17330[runtime] Merge cyclic dependent pipelined regions into one region
flinkbot edited a comment on pull request #13205: URL: https://github.com/apache/flink/pull/13205#issuecomment-677463515 ## CI report: * b6ec9981ff3880e61df71fc37459026dcaafb950 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5810) * 8617ce6c0c5e0b4e8ba01aae58b1ef5cff2f02e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5827) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18970) Adding Junit TestMarkers
[
https://issues.apache.org/jira/browse/FLINK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-18970.
> Adding Junit TestMarkers
>
>
> Key: FLINK-18970
> URL: https://issues.apache.org/jira/browse/FLINK-18970
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure, Tests
>Affects Versions: 1.11.1
>Reporter: goutham
>Priority: Minor
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> I am planning to add Test Marker to run the Unit test and Integration test
> using markers.
> Currently, if you want to run the complete build locally it takes close to 2
> hours. Based on requirement developers can run unit tests only or just
> integration or both.
> By default, it will run all the tests.
> planning to introduce below markers
> @Tag("IntegrationTest")
> @Tag("UnitTest")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18970) Adding Junit TestMarkers
[
https://issues.apache.org/jira/browse/FLINK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-18970:
-
Fix Version/s: (was: 1.11.1)
> Adding Junit TestMarkers
>
>
> Key: FLINK-18970
> URL: https://issues.apache.org/jira/browse/FLINK-18970
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure, Tests
>Affects Versions: 1.11.1
>Reporter: goutham
>Priority: Minor
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> I am planning to add Test Marker to run the Unit test and Integration test
> using markers.
> Currently, if you want to run the complete build locally it takes close to 2
> hours. Based on requirement developers can run unit tests only or just
> integration or both.
> By default, it will run all the tests.
> planning to introduce below markers
> @Tag("IntegrationTest")
> @Tag("UnitTest")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] StephanEwen commented on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
StephanEwen commented on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679245209 Looks good to me, too. Minor comment: I think it is fine to not put the `close()` in a `finally` block in tests, if this makes the test structure a bit more complex, because we don't need to ensure clean shutdown on unexpected test failures. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13229: [FLINK-19032] Remove deprecated RuntimeContext#getAllAcumullators
flinkbot edited a comment on pull request #13229: URL: https://github.com/apache/flink/pull/13229#issuecomment-679107131 ## CI report: * c7b217fe81e2a5480bb1155422bfa44a4278dc8f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5821) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13205: [FLINK-17330[runtime] Merge cyclic dependent pipelined regions into one region
flinkbot edited a comment on pull request #13205: URL: https://github.com/apache/flink/pull/13205#issuecomment-677463515 ## CI report: * b6ec9981ff3880e61df71fc37459026dcaafb950 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5810) * 8617ce6c0c5e0b4e8ba01aae58b1ef5cff2f02e6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13234: [FLINK-18905][task] Allow SourceOperator chaining with MultipleInputStreamTask
flinkbot commented on pull request #13234: URL: https://github.com/apache/flink/pull/13234#issuecomment-679243226 ## CI report: * e1c13bbc1899c7c2d7e94d26cdeb47deddf76b50 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
flinkbot edited a comment on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679234273 ## CI report: * f2538ea4babc43852365b6a3fde86f823873106c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5826) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13234: [FLINK-18905][task] Allow SourceOperator chaining with MultipleInputStreamTask
flinkbot commented on pull request #13234: URL: https://github.com/apache/flink/pull/13234#issuecomment-679234288 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e1c13bbc1899c7c2d7e94d26cdeb47deddf76b50 (Mon Aug 24 16:31:35 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
flinkbot commented on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679234273 ## CI report: * f2538ea4babc43852365b6a3fde86f823873106c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-675091412 ## CI report: * 9e7d2cb792556814b91b8f580d9a1cbe98433274 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5820) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13232: [FLINK-18951][python][docs] Add documentation for Configurations in P…
flinkbot edited a comment on pull request #13232: URL: https://github.com/apache/flink/pull/13232#issuecomment-679225070 ## CI report: * dd7743539c7ed5c45d93fc8b3c12e2db8b7af65e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5825) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18905) Add runtime support for multiple input operator chained with sources
[ https://issues.apache.org/jira/browse/FLINK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18905: --- Labels: pull-request-available (was: ) > Add runtime support for multiple input operator chained with sources > > > Key: FLINK-18905 > URL: https://issues.apache.org/jira/browse/FLINK-18905 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Raw runtime support, ignoring watermarks, metrics and checkpointing. > Requires rewrite of `OperatorChain` class, dealing with > `StreamTask#headOperator` field and generalisation of > `StreamMultipleInputProcessor`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wangzzu removed a comment on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
wangzzu removed a comment on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679231489 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski opened a new pull request #13234: [FLINK-18905][task] Allow SourceOperator chaining with MultipleInputStreamTask
pnowojski opened a new pull request #13234: URL: https://github.com/apache/flink/pull/13234 This PR adds a possibility to chain `SourceOperator` with `MultipleInputStreamTask`. Currently checkpointing, watermarks or metrics are not tested (part of another ticket). ## Brief change log Please check individual commit messages. ## Verifying this change This change added `MultipleInputStreamTaskChainedSourcesTest` for the new feature. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangzzu commented on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
wangzzu commented on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679231489 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #13205: [FLINK-17330[runtime] Merge cyclic dependent pipelined regions into one region
zhuzhurk commented on a change in pull request #13205:
URL: https://github.com/apache/flink/pull/13205#discussion_r475734143
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/failover/flip1/StronglyConnectedComponentsComputeUtilsTest.java
##
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.executiongraph.failover.flip1;
+
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Random;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static
org.apache.flink.runtime.executiongraph.failover.flip1.StronglyConnectedComponentsComputeUtils.computeStronglyConnectedComponents;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Unit tests for {@link StronglyConnectedComponentsComputeUtils}.
+ */
+public class StronglyConnectedComponentsComputeUtilsTest extends TestLogger {
+
+ @Test
+ public void testWithCycles() {
+ final List> edges = Arrays.asList(
+ Arrays.asList(2, 3),
+ Arrays.asList(0),
+ Arrays.asList(1),
+ Arrays.asList(4),
+ Collections.emptyList());
+
+ final Set> result =
computeStronglyConnectedComponents(5, edges);
+
+ final Set> expected = new HashSet<>();
+ expected.add(new HashSet<>(Arrays.asList(0, 1, 2)));
+ expected.add(Collections.singleton(3));
+ expected.add(Collections.singleton(4));
+
+ assertThat(result, is(expected));
+ }
+
+ @Test
+ public void testWithMultipleCycles() {
+ final List> edges = Arrays.asList(
+ Arrays.asList(1),
+ Arrays.asList(2),
+ Arrays.asList(0),
+ Arrays.asList(1, 2, 4),
+ Arrays.asList(3, 5),
+ Arrays.asList(2, 6),
+ Arrays.asList(5),
+ Arrays.asList(4, 6, 7));
+
+ final Set> result =
computeStronglyConnectedComponents(8, edges);
+
+ final Set> expected = new HashSet<>();
+ expected.add(new HashSet<>(Arrays.asList(0, 1, 2)));
+ expected.add(new HashSet<>(Arrays.asList(3, 4)));
+ expected.add(new HashSet<>(Arrays.asList(5, 6)));
+ expected.add(Collections.singleton(7));
+
+ assertThat(result, is(expected));
+ }
+
+ @Test
+ public void testWithConnectedCycles() {
+ final List> edges = Arrays.asList(
+ Arrays.asList(1),
+ Arrays.asList(2, 4, 5),
+ Arrays.asList(3, 6),
+ Arrays.asList(2, 7),
+ Arrays.asList(0, 5),
+ Arrays.asList(6),
+ Arrays.asList(5),
+ Arrays.asList(3, 6));
+
+ final Set> result =
computeStronglyConnectedComponents(8, edges);
+
+ final Set> expected = new HashSet<>();
+ expected.add(new HashSet<>(Arrays.asList(0, 1, 4)));
+ expected.add(new HashSet<>(Arrays.asList(2, 3, 7)));
+ expected.add(new HashSet<>(Arrays.asList(5, 6)));
+
+ assertThat(result, is(expected));
+ }
+
+ @Test
+ public void testWithNoEdge() {
+ final List> edges = Arrays.asList(
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList());
+
+ final Set> result =
computeStronglyConnectedComponents(5, edges);
+
+ final Set> expected = new HashSet<>();
+ expected.add(Collections.singleton(0));
[jira] [Resolved] (FLINK-18970) Adding Junit TestMarkers
[
https://issues.apache.org/jira/browse/FLINK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
goutham resolved FLINK-18970.
-
Resolution: Not A Problem
> Adding Junit TestMarkers
>
>
> Key: FLINK-18970
> URL: https://issues.apache.org/jira/browse/FLINK-18970
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure, Tests
>Affects Versions: 1.11.1
>Reporter: goutham
>Priority: Minor
> Fix For: 1.11.1
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> I am planning to add Test Marker to run the Unit test and Integration test
> using markers.
> Currently, if you want to run the complete build locally it takes close to 2
> hours. Based on requirement developers can run unit tests only or just
> integration or both.
> By default, it will run all the tests.
> planning to introduce below markers
> @Tag("IntegrationTest")
> @Tag("UnitTest")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18970) Adding Junit TestMarkers
[
https://issues.apache.org/jira/browse/FLINK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183413#comment-17183413
]
goutham commented on FLINK-18970:
-
Thanks [~sewen].. this helps.. we can close this issue
> Adding Junit TestMarkers
>
>
> Key: FLINK-18970
> URL: https://issues.apache.org/jira/browse/FLINK-18970
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure, Tests
>Affects Versions: 1.11.1
>Reporter: goutham
>Priority: Minor
> Fix For: 1.11.1
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> I am planning to add Test Marker to run the Unit test and Integration test
> using markers.
> Currently, if you want to run the complete build locally it takes close to 2
> hours. Based on requirement developers can run unit tests only or just
> integration or both.
> By default, it will run all the tests.
> planning to introduce below markers
> @Tag("IntegrationTest")
> @Tag("UnitTest")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13232: [FLINK-18951][python][docs] Add documentation for Configurations in P…
flinkbot commented on pull request #13232: URL: https://github.com/apache/flink/pull/13232#issuecomment-679225070 ## CI report: * dd7743539c7ed5c45d93fc8b3c12e2db8b7af65e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13230: [FLINK-18950][python][docs] Add documentation for Operations in Pytho…
flinkbot edited a comment on pull request #13230: URL: https://github.com/apache/flink/pull/13230#issuecomment-679164841 ## CI report: * be7e79cede81890f324a979b49651e401dcc329a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5823) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
flinkbot commented on pull request #13233: URL: https://github.com/apache/flink/pull/13233#issuecomment-679223761 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 906a6ad75a30129f73be5c0fb346e6d2d72458ea (Mon Aug 24 16:12:48 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19040) SourceOperator is not closing SourceReader
[
https://issues.apache.org/jira/browse/FLINK-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19040:
---
Labels: pull-request-available (was: )
> SourceOperator is not closing SourceReader
> --
>
> Key: FLINK-19040
> URL: https://issues.apache.org/jira/browse/FLINK-19040
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> {{SourceOperator}} is creating {{SourceReader}} but {{SourceReader}} is never
> closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] pnowojski opened a new pull request #13233: [FLINK-19040][task] Close SourceReader in SourceOperator
pnowojski opened a new pull request #13233: URL: https://github.com/apache/flink/pull/13233 Also properly close operator in the SourceOperatorTest unit tests ## Verifying this change This change added tests an extra assertion to the `SourceOperatorTest#testOpen` unit test, to add test coverage for the bug. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13232: [FLINK-18951][python][docs] Add documentation for Configurations in P…
flinkbot commented on pull request #13232: URL: https://github.com/apache/flink/pull/13232#issuecomment-679211593 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit dd7743539c7ed5c45d93fc8b3c12e2db8b7af65e (Mon Aug 24 15:53:25 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18951) Add documentation for Configurations in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18951: --- Labels: pull-request-available (was: ) > Add documentation for Configurations in Python DataStream API. > --- > > Key: FLINK-18951 > URL: https://issues.apache.org/jira/browse/FLINK-18951 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Shuiqiang Chen >Assignee: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] shuiqiangchen opened a new pull request #13232: [FLINK-18951][python][docs] Add documentation for Configurations in P…
shuiqiangchen opened a new pull request #13232: URL: https://github.com/apache/flink/pull/13232 ## What is the purpose of the change Add documentation for Configurations in Python DataStream API. ## Brief change log - Added a new page named dependency_management.md under dev/python/datastream ## Verifying this change This pull request added documentations for existing Python DataStream API, which has no need for test case coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? ( no) - If yes, how is the feature documented? ( not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-19040) SourceOperator is not closing SourceReader
[
https://issues.apache.org/jira/browse/FLINK-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-19040:
--
Assignee: Piotr Nowojski
> SourceOperator is not closing SourceReader
> --
>
> Key: FLINK-19040
> URL: https://issues.apache.org/jira/browse/FLINK-19040
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0, 1.11.1
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Critical
> Fix For: 1.12.0, 1.11.2
>
>
> {{SourceOperator}} is creating {{SourceReader}} but {{SourceReader}} is never
> closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19040) SourceOperator is not closing SourceReader
Piotr Nowojski created FLINK-19040:
--
Summary: SourceOperator is not closing SourceReader
Key: FLINK-19040
URL: https://issues.apache.org/jira/browse/FLINK-19040
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.11.1, 1.12.0
Reporter: Piotr Nowojski
Fix For: 1.12.0, 1.11.2
{{SourceOperator}} is creating {{SourceReader}} but {{SourceReader}} is never
closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13231: [FLINK-19031] Remove deprecated StreamExecutionEnvironment#setStateBackend(AbstactStateBackend)
flinkbot edited a comment on pull request #13231: URL: https://github.com/apache/flink/pull/13231#issuecomment-679197180 ## CI report: * faa0726d771536cfdb669d86bc5310e82946f527 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5824) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13231: [FLINK-19031] Remove deprecated StreamExecutionEnvironment#setStateBackend(AbstactStateBackend)
flinkbot commented on pull request #13231: URL: https://github.com/apache/flink/pull/13231#issuecomment-679197180 ## CI report: * faa0726d771536cfdb669d86bc5310e82946f527 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 50348849269d2b0edb7e811b9e5e13de45f686cd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5818) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19039) Parallel Flink Kafka Consumers compete with each other
Ayrat Hudaygulov created FLINK-19039:
Summary: Parallel Flink Kafka Consumers compete with each other
Key: FLINK-19039
URL: https://issues.apache.org/jira/browse/FLINK-19039
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.11.1
Reporter: Ayrat Hudaygulov
If I'll run multiple Flink instances with same consumer group id they will not
re-balance partitions with each other, but rather each instance take all
partitions, effectively not working in parallel at all, and multiplying amount
of messages processed.
This is because FlinkKafkaConsumer has its own re-balancing mechanism for
current parallelism level and then just calls:
`consumerTmp.assign(newPartitionAssignments){color:#cc7832};{color}`
[https://github.com/apache/flink/blob/59714b9d6addb1dbf2171cab937a0e3fec52f2b1/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java#L422]
I suppose there has to be a way to fallback to default kafka mechanism of
re-balancing to respect consumer group id, but it's not presented in Flink at
all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13231: [FLINK-19031] Remove deprecated StreamExecutionEnvironment#setStateBackend(AbstactStateBackend)
flinkbot commented on pull request #13231: URL: https://github.com/apache/flink/pull/13231#issuecomment-679188516 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit faa0726d771536cfdb669d86bc5310e82946f527 (Mon Aug 24 15:14:17 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
