[GitHub] [flink] flinkbot commented on issue #9835: [FLINK-14309] [test-stability] Add retries and acks config in producer test
flinkbot commented on issue #9835: [FLINK-14309] [test-stability] Add retries and acks config in producer test URL: https://github.com/apache/flink/pull/9835#issuecomment-537799441 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 49f80fc9c378d0884dd6b411dbe2966ee634f057 (Thu Oct 03 05:55:48 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14309).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] buptljy opened a new pull request #9835: [FLINK-14309] [test-stability] Add retries and acks config in producer test
buptljy opened a new pull request #9835: [FLINK-14309] [test-stability] Add retries and acks config in producer test URL: https://github.com/apache/flink/pull/9835 ## What is the purpose of the change Fix unit test: KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink ## Brief change log Add retries and acks config to reduce the risk of failure of flush() method. ## Verifying this change Unit Testing. ## Does this pull request potentially affect one of the following parts: ## Documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14309) Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14309:
---
Labels: pull-request-available test-stability (was: test-stability)
> Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink
> fails on Travis
> --
>
> Key: FLINK-14309
> URL: https://issues.apache.org/jira/browse/FLINK-14309
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.0
>
>
> The
> {{Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink}}
> fails on Travis with
> {code}
> Test
> testOneToOneAtLeastOnceRegularSink(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
> failed with:
> java.lang.AssertionError: Job should fail!
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnce(KafkaProducerTestBase.java:280)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink(KafkaProducerTestBase.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> https://api.travis-ci.com/v3/job/240747188/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14309) Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943347#comment-16943347
]
Jiayi Liao edited comment on FLINK-14309 at 10/3/19 5:40 AM:
-
I didn't reproduce the error after tens of times trying. But the most possible
reason is that the KafkaProducerSink fails to snapshot(but the checkpoint
succeeds) and the job fails after the first checkpoint. Since the flush()
method in KafkaProducer didn't have a message guarantee, maybe we can increase
the retries config value(default=0) and set acks config value to
"all"(default=0) at first. I'm not completely sure this will help but it would
be better if we add these. (the project already add the configs in consumer's
tests)
Here is the comments from flush() method:
{code:java}
* This example shows how to consume from one Kafka topic and produce to
another Kafka topic:
*
* {@code
* for(ConsumerRecord record: consumer.poll(100))
* producer.send(new ProducerRecord("my-topic", record.key(),
record.value());
* producer.flush();
* consumer.commit();
* }
*
*
* Note that the above example may drop records if the produce request
fails. If we want to ensure that this does not occur
* we need to set retries=large_number in our config.
{code}
cc [~becket_qin] [~trohrmann]
was (Author: wind_ljy):
I didn't reproduce the error after tens of times trying. But the most possible
reason is that the KafkaProducerSink fails to snapshot(but the checkpoint
succeeds) and the job fails after the first checkpoint. Since the flush()
method in KafkaProducer didn't have a message guarantee, maybe we can increase
the retries config value(default=0) and set acks config value to
"all"(default=0) at first. I'm not completely sure this will help but it would
be better if we add these. (the project already add the configs in consumer's
tests)
Here is the comments from flush() method:
{code:java}
* This example shows how to consume from one Kafka topic and produce to
another Kafka topic:
*
* {@code
* for(ConsumerRecord record: consumer.poll(100))
* producer.send(new ProducerRecord("my-topic", record.key(),
record.value());
* producer.flush();
* consumer.commit();
* }
*
*
* Note that the above example may drop records if the produce request
fails. If we want to ensure that this does not occur
* we need to set retries=large_number in our config.
{code}
[~becket_qin] [~trohrmann]
> Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink
> fails on Travis
> --
>
> Key: FLINK-14309
> URL: https://issues.apache.org/jira/browse/FLINK-14309
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The
> {{Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink}}
> fails on Travis with
> {code}
> Test
> testOneToOneAtLeastOnceRegularSink(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
> failed with:
> java.lang.AssertionError: Job should fail!
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnce(KafkaProducerTestBase.java:280)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink(KafkaProducerTestBase.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at
[jira] [Comment Edited] (FLINK-14309) Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943347#comment-16943347
]
Jiayi Liao edited comment on FLINK-14309 at 10/3/19 5:40 AM:
-
I didn't reproduce the error after tens of times trying. But the most possible
reason is that the KafkaProducerSink fails to snapshot(but the checkpoint
succeeds) and the job fails after the first checkpoint. Since the flush()
method in KafkaProducer didn't have a message guarantee, maybe we can increase
the retries config value(default=0) and set acks config value to
"all"(default=0) at first. I'm not completely sure this will help but it would
be better if we add these. (the project already add the configs in consumer's
tests)
Here is the comments from flush() method:
{code:java}
* This example shows how to consume from one Kafka topic and produce to
another Kafka topic:
*
* {@code
* for(ConsumerRecord record: consumer.poll(100))
* producer.send(new ProducerRecord("my-topic", record.key(),
record.value());
* producer.flush();
* consumer.commit();
* }
*
*
* Note that the above example may drop records if the produce request
fails. If we want to ensure that this does not occur
* we need to set retries=large_number in our config.
{code}
[~becket_qin] [~trohrmann]
was (Author: wind_ljy):
I didn't reproduce the error after tens of times trying. But the most possible
reason is that the KafkaProducerSink fails to snapshot(but the checkpoint
succeeds) and the job fails after the first checkpoint. Since the flush()
method in KafkaProducer didn't have a message guarantee, maybe we can increase
the retries config value(default=0) and set acks config value to
"all"(default=0) at first. I'm not completely sure this will help but it would
be better if we add these. (the project already add the configs in consumer's
tests)
Here is the comments from flush() method:
```
* This example shows how to consume from one Kafka topic and produce to
another Kafka topic:
*
* {@code
* for(ConsumerRecord record: consumer.poll(100))
* producer.send(new ProducerRecord("my-topic", record.key(),
record.value());
* producer.flush();
* consumer.commit();
* }
*
*
* Note that the above example may drop records if the produce request
fails. If we want to ensure that this does not occur
* we need to set retries=large_number in our config.
```
[~becket_qin] [~trohrmann]
> Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink
> fails on Travis
> --
>
> Key: FLINK-14309
> URL: https://issues.apache.org/jira/browse/FLINK-14309
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The
> {{Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink}}
> fails on Travis with
> {code}
> Test
> testOneToOneAtLeastOnceRegularSink(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
> failed with:
> java.lang.AssertionError: Job should fail!
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnce(KafkaProducerTestBase.java:280)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink(KafkaProducerTestBase.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at
[jira] [Commented] (FLINK-14309) Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943347#comment-16943347
]
Jiayi Liao commented on FLINK-14309:
I didn't reproduce the error after tens of times trying. But the most possible
reason is that the KafkaProducerSink fails to snapshot(but the checkpoint
succeeds) and the job fails after the first checkpoint. Since the flush()
method in KafkaProducer didn't have a message guarantee, maybe we can increase
the retries config value(default=0) and set acks config value to
"all"(default=0) at first. I'm not completely sure this will help but it would
be better if we add these. (the project already add the configs in consumer's
tests)
Here is the comments from flush() method:
```
* This example shows how to consume from one Kafka topic and produce to
another Kafka topic:
*
* {@code
* for(ConsumerRecord record: consumer.poll(100))
* producer.send(new ProducerRecord("my-topic", record.key(),
record.value());
* producer.flush();
* consumer.commit();
* }
*
*
* Note that the above example may drop records if the produce request
fails. If we want to ensure that this does not occur
* we need to set retries=large_number in our config.
```
[~becket_qin] [~trohrmann]
> Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink
> fails on Travis
> --
>
> Key: FLINK-14309
> URL: https://issues.apache.org/jira/browse/FLINK-14309
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The
> {{Kafka09ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink}}
> fails on Travis with
> {code}
> Test
> testOneToOneAtLeastOnceRegularSink(org.apache.flink.streaming.connectors.kafka.Kafka09ProducerITCase)
> failed with:
> java.lang.AssertionError: Job should fail!
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnce(KafkaProducerTestBase.java:280)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink(KafkaProducerTestBase.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
[jira] [Comment Edited] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943304#comment-16943304
]
Leo Zhang edited comment on FLINK-14055 at 10/3/19 4:23 AM:
Hi, everyone. I have implemented
*\\{Stream\}ExecutionEnvironment::registerUserJarFile()* interfaces both for
java and scala API, and they are tested well. Do you think it is better to
split this task FLINK-14055 into two goals, one for DDL and the other for
*registerUserJarFile()* interface? I see that there are some use cases in which
people want to implement their own SQL application based on loading external
jars for now.
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
was (Author: 50man):
Hi, everyone. I have implemented
*\\{Stream\}ExecutionEnvironment::registerUserJarFile()* interfaces both for
java and scala API, and they are tested well. Do you think it is better to
split this task into two goals, one for DDL and the other for
*registerUserJarFile()* interface? I see that there are some use cases in which
people want to implement their own SQL application based on loading external
jars for now.
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14295) Nightly flink-runtime failed with java 11
[
https://issues.apache.org/jira/browse/FLINK-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943315#comment-16943315
]
ramkrishna.s.vasudevan commented on FLINK-14295:
[~chesnay], [~liyu]
I can help to solve this ticket.
Just one query - while running the test with java11 - is there any profile that
I need to pass to the 'mvn test' command so that the test runs with java 11
(assuming I already have the jdk 11 in my test node).
> Nightly flink-runtime failed with java 11
> -
>
> Key: FLINK-14295
> URL: https://issues.apache.org/jira/browse/FLINK-14295
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Yu Li
>Priority: Critical
> Labels: test-stability
>
> The core-jdk11 part of nightly test failed with below error:
> {noformat}
> 22:09:38.176 [ERROR] Failures:
> 22:09:38.180 [ERROR]
> TaskExecutorSubmissionTest.testRequestStackTraceSample:637
> expected:<[updateTaskExecutionState]> but
> was:<[lambda$updateTaskExecutionState$0]>
> 22:09:38.185 [ERROR] Errors:
> 22:09:38.185 [ERROR]
> RecordWriterTest.testClearBuffersAfterInterruptDuringBlockingBufferRequest:165
> » NoSuchElement
> 22:09:38.185 [INFO]
> 22:09:38.186 [ERROR] Tests run: 3936, Failures: 1, Errors: 1, Skipped: 40
> {noformat}
> Link of the build: https://api.travis-ci.org/v3/job/591086968/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943311#comment-16943311
]
Leo Zhang commented on FLINK-14055:
---
You're right that we should follow the design doc FLINK-10232. And my
implementation exactly obeys to the design doc and chooses the first option
from design alternatives. Should we pose an issue on Github to discuss it
further?
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943307#comment-16943307
]
Bowen Li commented on FLINK-14055:
--
[~50man] this seems to be a quite big effort to me. don't jump right to
implementing it, it definitely needs design doc and has to be discussed in
community dev ml first
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943304#comment-16943304
]
Leo Zhang edited comment on FLINK-14055 at 10/3/19 3:50 AM:
Hi, everyone. I have implemented
*\\{Stream\}ExecutionEnvironment::registerUserJarFile()* interfaces both for
java and scala API, and they are tested well. Do you think it is better to
split this task into two goals, one for DDL and the other for
*registerUserJarFile()* interface? I see that there are some use cases in which
people want to implement their own SQL application based on loading external
jars for now.
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
was (Author: 50man):
Hi, everyone. I have implemented
*{Stream}ExecutionEnvironment::registerUserJarFile()* interfaces both for java
and scala API, and they are tested well. Do you think it is better to split
this task into two goals, one for DDL and the other for *registerUserJarFile()*
interface? I see that there are some use cases in which people want to
implement their own SQL application based on loading external jars for now.
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943304#comment-16943304
]
Leo Zhang edited comment on FLINK-14055 at 10/3/19 3:47 AM:
Hi, everyone. I have implemented
*\\{Stream\}ExecutionEnvironment::registerUserJarFile()* interfaces both for
java and scala API, and they are tested well. Do you think it is better to
split this task into two goals, one for DDL and the other for
*registerUserJarFile()* interface?
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
was (Author: 50man):
Hi, everyone. I have implemented
*{Stream}ExecutionEnvironment::registerUserJarFile()* interfaces both for java
and scala API, and they are tested well. Do you think it is better to split
this task into two goals, one for DDL and the other for *registerUserJarFile()*
interface?
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943304#comment-16943304
]
Leo Zhang edited comment on FLINK-14055 at 10/3/19 3:46 AM:
Hi, everyone. I have implemented
*{Stream}ExecutionEnvironment::registerUserJarFile()* interfaces both for java
and scala API, and they are tested well. Do you think it is better to split
this task into two goals, one for DDL and the other for *registerUserJarFile()*
interface?
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
was (Author: 50man):
Hi, everyone. I have implemented
_{Stream}ExecutionEnvironment::registerUserJarFile()_ interfaces both for java
and scala API, and they are tested well. Do you think it is better to split
this task into two goals, one for DDL and the other for _registerUserJarFile()_
interface?
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943304#comment-16943304
]
Leo Zhang edited comment on FLINK-14055 at 10/3/19 3:45 AM:
Hi, everyone. I have implemented
_{Stream}ExecutionEnvironment::registerUserJarFile()_ interfaces both for java
and scala API, and they are tested well. Do you think it is better to split
this task into two goals, one for DDL and the other for _registerUserJarFile()_
interface?
And I'd like to help here and could you please assign the latter goal to me so
that I can propose a PR on Github as soon as possible?
[~hpeter] [~phoenixjiangnan]
was (Author: 50man):
Hi, everyone. I have implemented
\{Stream}ExecutionEnvironment::registerUserJarFile() interfaces both for java
and scala API, and they are tested well. Do you think its better to split this
task into to goals one for DDL and the other for registerUserJarFile()
interfaces. And I'd like to help here and could you please assign the latter
goal to me so that I can propose a PR on Github as soon as possible?
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14055) Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
[
https://issues.apache.org/jira/browse/FLINK-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943304#comment-16943304
]
Leo Zhang commented on FLINK-14055:
---
Hi, everyone. I have implemented
\{Stream}ExecutionEnvironment::registerUserJarFile() interfaces both for java
and scala API, and they are tested well. Do you think its better to split this
task into to goals one for DDL and the other for registerUserJarFile()
interfaces. And I'd like to help here and could you please assign the latter
goal to me so that I can propose a PR on Github as soon as possible?
> Add advanced function DDL syntax "USING JAR/FILE/ACHIVE"
>
>
> Key: FLINK-14055
> URL: https://issues.apache.org/jira/browse/FLINK-14055
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As FLINK-7151 adds basic function DDL to Flink, this ticket is to support
> dynamically loading functions from external source in function DDL with
> advanced syntax like
>
> {code:java}
> CREATE FUNCTION func_name as class_name USING JAR/FILE/ACHIEVE 'xxx' [,
> JAR/FILE/ACHIEVE 'yyy'] ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9822: [FLINK-14216][table] introduce temp system functions to FunctionCatalog
flinkbot edited a comment on issue #9822: [FLINK-14216][table] introduce temp system functions to FunctionCatalog URL: https://github.com/apache/flink/pull/9822#issuecomment-536774896 ## CI report: * 611ecdea461d9c202e0fd3ad0b33f24e8a4db061 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129772949) * cc357e4d5365aa98db6ebb8a3c1a821080bb202e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129775325) * a06e7bd3f0853e637a7b1cbac52bc70d1ebe3923 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130117578) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-11127) Make metrics query service establish connection to JobManager
[
https://issues.apache.org/jira/browse/FLINK-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943231#comment-16943231
]
Tim commented on FLINK-11127:
-
[~hwanju] - Thanks much for describing this scenario. It is exactly what I
have been experiencing. Am I to take it that the proposal has been implemented
in FLINK-11632 in full? I checked on that ticket, and Till confirmed it is
available in version 1.8.0 onwards (I'm on 1.7.x). Thanks!
> Make metrics query service establish connection to JobManager
> -
>
> Key: FLINK-11127
> URL: https://issues.apache.org/jira/browse/FLINK-11127
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Runtime / Coordination, Runtime
> / Metrics
>Affects Versions: 1.7.0
>Reporter: Ufuk Celebi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of FLINK-10247, the internal metrics query service has been separated
> into its own actor system. Before this change, the JobManager (JM) queried
> TaskManager (TM) metrics via the TM actor. Now, the JM needs to establish a
> separate connection to the TM metrics query service actor.
> In the context of Kubernetes, this is problematic as the JM will typically
> *not* be able to resolve the TMs by name, resulting in warnings as follows:
> {code}
> 2018-12-11 08:32:33,962 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183] has
> failed, address is now gated for [50] ms. Reason: [Association failed with
> [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183]] Caused
> by: [flink-task-manager-64b868487c-x9l4b: Name does not resolve]
> {code}
> In order to expose the TMs by name in Kubernetes, users require a service
> *for each* TM instance which is not practical.
> This currently results in the web UI not being to display some basic metrics
> about number of sent records. You can reproduce this by following the READMEs
> in {{flink-container/kubernetes}}.
> This worked before, because the JM is typically exposed via a service with a
> known name and the TMs establish the connection to it which the metrics query
> service piggybacked on.
> A potential solution to this might be to let the query service connect to the
> JM similar to how the TMs register.
> I tagged this ticket as an improvement, but in the context of Kubernetes I
> would consider this to be a bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-11632) Make TaskManager automatic bind address picking more explicit (by default) and more configurable
[
https://issues.apache.org/jira/browse/FLINK-11632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943226#comment-16943226
]
Tim commented on FLINK-11632:
-
Thank you Till!
> Make TaskManager automatic bind address picking more explicit (by default)
> and more configurable
>
>
> Key: FLINK-11632
> URL: https://issues.apache.org/jira/browse/FLINK-11632
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
>Reporter: Alex
>Assignee: Alex
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, there is an optional {{taskmanager.host}} configuration option in
> {{flink-conf.yaml}} that allows users of Flink to "statically" pre-define
> what should be a bind address for TaskManager to listen on (note: it's also
> possible to override this option by passing corresponding command line option
> to Flink).
> In case when the option is not set, TaskManager would try [heuristically pick
> up a bind
> address|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java#L421-L442].
> The resulting address (hostname) is used to advertise different service
> endpoints (running in TM) to the JobManager. Also it would be resolved to an
> {{[InetAddress|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java#L359]}}
> later that used as binding address for TMs inner node communication.
> This proposal is to minimize usage of heuristics (by default) by introducing
> a new configuration option (for example, {{taskmanager.host.bind-policy}})
> with possible values:
> * {{"hostname"}} - default, use TM's host's name ({{==
> InetAddress.getLocalHost().getHostName()}};
> * {{"ip"}} - use TM's host's ip address ({{==
> InetAddress.getLocalHost().getHostAddress()}});
> * {{"auto-detect-hostname"}} - use the heuristics based detection mechanism.
> *Note:* the configuration key and values could be named better and open for
> proposals.
> *Note 2:* in the future, the configuration option _may_ require to be
> extended to allow choosing some specific network interface, or preference of
> ipv6 vs ipv4.
> h3. Rationale
> [The heuristics
> mechanism|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/net/ConnectionUtils.java#L364-L475]
> tries to establish a probe connection to {{jobmanager.rpc.address}} from
> different network interface addresses.
> In case of parallel setups (when JM and multiple TMs start simultaneously,
> in parallel), this depends on timing, assigned network ip addresses and may
> end up with "non-uniform" address bindings of TMs (some may be "lucky" to
> pick up non default network interface, some would fallback to
> {{InetAddress.getLocalHost().getHostName()}}. At the end, it's less obvious
> and transparent which binding address a TM picks up.
> In practice, it's possible that in majority of cases (in well setup
> environments) the heuristics mechanism returns a result that matches
> {{InetAddress.getLocalHost()}}. The proposal is to stick with this more
> simpler and explicit binding (by default), avoiding non-determinism of
> heuristics.
> The old mechanism is kept available, in case if it is useful in some setups.
> But would require explicit configuration setting.
> Additionally, this proposal extends "auto configuration" option by allowing
> users to choose the host's ip address (instead of hostname). This may be
> convenient in situations where the TMs' machines are not necessary reachable
> via DNS (for example in a Kubernetes setup).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9822: [FLINK-14216][table] introduce temp system functions to FunctionCatalog
flinkbot edited a comment on issue #9822: [FLINK-14216][table] introduce temp system functions to FunctionCatalog URL: https://github.com/apache/flink/pull/9822#issuecomment-536774896 ## CI report: * 611ecdea461d9c202e0fd3ad0b33f24e8a4db061 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129772949) * cc357e4d5365aa98db6ebb8a3c1a821080bb202e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129775325) * a06e7bd3f0853e637a7b1cbac52bc70d1ebe3923 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session URL: https://github.com/apache/flink/pull/9832#issuecomment-537049332 ## CI report: * abaae048fef753455970fac9d6ab421b660b0536 : UNKNOWN * b96c63552ccd322adae7a41a410615e95b538ece : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867595) * 2c95a3939dbf0259d694af6c69451f0ede3c3891 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130066310) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9831: [FLINK-14278] Extend DispatcherResourceManagerComponentFactory.create to take ioExecutor
flinkbot edited a comment on issue #9831: [FLINK-14278] Extend DispatcherResourceManagerComponentFactory.create to take ioExecutor URL: https://github.com/apache/flink/pull/9831#issuecomment-537027384 ## CI report: * e1d37ab03c1bc62b3d45c45ee11d616ed2c78b0c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129858618) * ba5dc1be84ded55f91fad1698e0edb2ca7416add : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130066271) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9830: [FLINK-14307] Extract JobGraphWriter from JobGraphStore
flinkbot edited a comment on issue #9830: [FLINK-14307] Extract JobGraphWriter from JobGraphStore URL: https://github.com/apache/flink/pull/9830#issuecomment-537017897 ## CI report: * ed86e6d768a1353423d969e6c9dcf5fd967c4f19 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129854718) * 9892be4548e684c87dc586b96adbc3fb6052d7c7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130066218) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9813: [FLINK-14287] Decouple leader address from LeaderContender
flinkbot edited a comment on issue #9813: [FLINK-14287] Decouple leader address from LeaderContender URL: https://github.com/apache/flink/pull/9813#issuecomment-536324105 ## CI report: * b7a7e7b6dc8a16d23097911aa98852e2cf4d9c00 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129601641) * 3b8410b82ec366014551e35fb8791c6ad623faf9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707833) * 96ef0ac36eb37792fd7837619cbca7d8a1272090 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837942) * 4c3d9db5f51b815c810be6fe6ae82c62452f14b4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129854608) * 258622b3043245884ed9d567124cef31ffc0571a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867500) * df058e289e348c3f48a1e82479cf90062c542b42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130066162) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9812: [FLINK-14286] Remove Akka specific parsing from LeaderConnectionInfo
flinkbot edited a comment on issue #9812: [FLINK-14286] Remove Akka specific parsing from LeaderConnectionInfo URL: https://github.com/apache/flink/pull/9812#issuecomment-536322371 ## CI report: * c1ca1131b02a2c555cd8386fd07aa1cfccbab161 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600831) * da85c13656e6cf02fcadbadf50df0c201f07970c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707797) * 2abee623fcedcbf8c61ce455cbf59bd59b75dcf4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837903) * c1efb87828fc98fa2f7ff5b07b082b9dfc67c501 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854578) * 9b2e7712bc24fca3f77aa02993aa3afb89de7bfc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129867450) * 2d706c3a58923435de08d0df8a3a938ec6709de2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130066133) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9811: [FLINK-14285] Remove generics from Dispatcher factories
flinkbot edited a comment on issue #9811: [FLINK-14285] Remove generics from Dispatcher factories URL: https://github.com/apache/flink/pull/9811#issuecomment-536320588 ## CI report: * 658cddd4424505c1a904a926ad6e626abbd85cd7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600090) * 5b4df7e3bd41053dcce049edf420ed584a42b98a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707753) * 442ea9d2bcc705ba162ffe6bd7c0d127a372c06e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837865) * 2f46b0debc0a94a969510fbb07b8f1b5e6b50710 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850925) * 65e72a4a5a51920697475dbc3543642b0537cdd5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867404) * e1fd3451aafdd71e80472582a7916a8c24f11376 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130066107) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #9689: [FLINK-7151] add a basic function ddl
bowenli86 commented on issue #9689: [FLINK-7151] add a basic function ddl URL: https://github.com/apache/flink/pull/9689#issuecomment-537617694 @fhueske do you feel we should wait for FLIP-69 to reach consensus before merging this PR, or can we do it anytime soon? Since this ticket has been agreed upon before FLIP-69 came out, I personally feel it's fine to not wait for FLIP-69's voting, but would like to get your opinions on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9689: [FLINK-7151] add a basic function ddl
bowenli86 commented on a change in pull request #9689: [FLINK-7151] add a basic
function ddl
URL: https://github.com/apache/flink/pull/9689#discussion_r330698065
##
File path:
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java
##
@@ -529,6 +529,24 @@ public void testDropIfExists() {
check(sql, "DROP TABLE IF EXISTS `CATALOG1`.`DB1`.`TBL1`");
}
+ @Test
+ public void testCreateFunction() {
Review comment:
add a test for "CREATE FUNCTION XXX IF NOT EXIST"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9810: [FLINK-14284] Add shut down future to Dispatcher
flinkbot edited a comment on issue #9810: [FLINK-14284] Add shut down future to Dispatcher URL: https://github.com/apache/flink/pull/9810#issuecomment-536320582 ## CI report: * 3ea0f47e7dd76ec115db4ef583b416685107604b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600085) * e73e7e0a72952be963e29226bbb9b94487dafde6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707704) * eed7f48de7520824f5674d0959a91b1dd9caa654 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837834) * a4cecd0d80ce9728c96d70195e82a09958a0c90b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850891) * a908e21bc54f2052fc7b4b81261af2324fd32920 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867370) * 0fc658418704bdc15447a18db695551c3521421c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130066083) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9689: [FLINK-7151] add a basic function ddl
bowenli86 commented on a change in pull request #9689: [FLINK-7151] add a basic function ddl URL: https://github.com/apache/flink/pull/9689#discussion_r330697441 ## File path: flink-table/flink-sql-parser/src/main/codegen/config.fmpp ## @@ -10,7 +10,7 @@ # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and +# See the License for the specific language governing permissions andFlinkSqlParserImpl Review comment: by mistake? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9809: [FLINK-14282] Simplify DispatcherResourceManagerComponent hierarchy
flinkbot edited a comment on issue #9809: [FLINK-14282] Simplify DispatcherResourceManagerComponent hierarchy URL: https://github.com/apache/flink/pull/9809#issuecomment-536318695 ## CI report: * 531cd688ceb25b544833025d9b556a0d686b29c4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129599298) * a3ba53aa778e777d8f38750eb6a5e5147fd21654 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707662) * db3d62f7f74f3be33d65588f76a8a082f9fccfd0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837805) * 544d687c37a1597737fd35747d6fa7c5740c2c40 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129850857) * 5fdd2823b2bdbd6dcd7b8172369b41fe5438c3ce : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867317) * 8a8911aa471acb3cf551a5fd12bd14d506f6040a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130066032) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14175) Upgrade KPL version in flink-connector-kinesis to fix application OOM
[ https://issues.apache.org/jira/browse/FLINK-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943046#comment-16943046 ] Bowen Li commented on FLINK-14175: -- I don't feel really necessary to update KPL in Flink 1.9 and past, since users are already building kinesis connector themselves and they can just choose whatever KPL version they like. Feel free to submit a PR though, and I'll try to help review. > Upgrade KPL version in flink-connector-kinesis to fix application OOM > - > > Key: FLINK-14175 > URL: https://issues.apache.org/jira/browse/FLINK-14175 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.6.3, 1.6.4, 1.6.5, 1.7.2, 1.7.3, 1.8.0, 1.8.1, 1.8.2, > 1.9.0 > Environment: [link title|http://example.com][link > title|http://example.com] >Reporter: Abhilasha Seth >Priority: Major > > The [KPL > version|https://github.com/apache/flink/blob/release-1.9/flink-connectors/flink-connector-kinesis/pom.xml#L38] > (0.12.9) used by flink-connector-kinesis in the affected Flink versions has > a thread leak bug that causes applications to run out of memory after > frequent restarts: > KPL Issue - [https://github.com/awslabs/amazon-kinesis-producer/issues/224] > Fix - [https://github.com/awslabs/amazon-kinesis-producer/pull/225/files] > Upgrading KPL to 0.12.10 or higher is necessary to avoid this issue. The > recommended version to upgrade would be the latest (0.13.1) > Note that KPL version in Flink 1.10.0 has been updated to the latest version > (0.13.1): https://issues.apache.org/jira/browse/FLINK-12847 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9807: [FLINK-14280] Introduce DispatcherRunner
flinkbot edited a comment on issue #9807: [FLINK-14280] Introduce DispatcherRunner URL: https://github.com/apache/flink/pull/9807#issuecomment-536314902 ## CI report: * 9ef8fe986e4b06d6ae8512e5051933458363511c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129597654) * 748954650809fd2fa18572d1c925d3b647a55906 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707578) * 1ca7b625300da1a6a522fd1ba8427f5398f61ed5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129834475) * d0bd4cbe32866ed3f6cceafe6dd65133f96350b3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850793) * bf3c8311655872a9def94d46bf1363bc89c4715f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129867229) * a5b8cd87d6c2ce7a41531c34eae8ef78e1a3255f : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130065977) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9808: [FLINK-14281] Add DispatcherRunner#getShutDownFuture
flinkbot edited a comment on issue #9808: [FLINK-14281] Add DispatcherRunner#getShutDownFuture URL: https://github.com/apache/flink/pull/9808#issuecomment-536316924 ## CI report: * 02c93d5d7d69324c33461040fec5c4014eb2b8d6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129598507) * 618f898c4a74c7f35f66ba7ab37e08cf8d66581e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707614) * 4f7e44bebc6f5dd1b222e68658e40b7f2002e5fd : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129834505) * ca5f4450fe3b2a33e39ad1b01b0381b8f7996d78 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850827) * 3d2171647776f5f8537f7e130026f14c99b3fef0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867287) * 79f00b2c099514054118c93214d91a857244c82c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130066015) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session URL: https://github.com/apache/flink/pull/9832#issuecomment-537049332 ## CI report: * abaae048fef753455970fac9d6ab421b660b0536 : UNKNOWN * b96c63552ccd322adae7a41a410615e95b538ece : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867595) * 2c95a3939dbf0259d694af6c69451f0ede3c3891 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066310) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9830: [FLINK-14307] Extract JobGraphWriter from JobGraphStore
flinkbot edited a comment on issue #9830: [FLINK-14307] Extract JobGraphWriter from JobGraphStore URL: https://github.com/apache/flink/pull/9830#issuecomment-537017897 ## CI report: * ed86e6d768a1353423d969e6c9dcf5fd967c4f19 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129854718) * 9892be4548e684c87dc586b96adbc3fb6052d7c7 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066218) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9831: [FLINK-14278] Extend DispatcherResourceManagerComponentFactory.create to take ioExecutor
flinkbot edited a comment on issue #9831: [FLINK-14278] Extend DispatcherResourceManagerComponentFactory.create to take ioExecutor URL: https://github.com/apache/flink/pull/9831#issuecomment-537027384 ## CI report: * e1d37ab03c1bc62b3d45c45ee11d616ed2c78b0c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129858618) * ba5dc1be84ded55f91fad1698e0edb2ca7416add : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066271) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9821: [FLINK-14298] Replace LeaderContender#getAddress with #getDescription
flinkbot edited a comment on issue #9821: [FLINK-14298] Replace LeaderContender#getAddress with #getDescription URL: https://github.com/apache/flink/pull/9821#issuecomment-536618253 ## CI report: * 9ca9df3eaa22965444373a3b6142798ca5559f50 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129712419) * e2b8bb516e06b5337d1f6172ac00c4f4b42f160a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129808007) * 7ded64e1fca1abbbcdeb55604e2ab60e351e23c0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837967) * 4b668e45a1350037e7d738314dd5889f71098a8b : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129854643) * 7e004fc6d0620152b3083019fedad411172b6077 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867555) * 005137c22c57f7e1f92f901571f72916b1d10110 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130066190) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9813: [FLINK-14287] Decouple leader address from LeaderContender
flinkbot edited a comment on issue #9813: [FLINK-14287] Decouple leader address from LeaderContender URL: https://github.com/apache/flink/pull/9813#issuecomment-536324105 ## CI report: * b7a7e7b6dc8a16d23097911aa98852e2cf4d9c00 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129601641) * 3b8410b82ec366014551e35fb8791c6ad623faf9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707833) * 96ef0ac36eb37792fd7837619cbca7d8a1272090 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837942) * 4c3d9db5f51b815c810be6fe6ae82c62452f14b4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129854608) * 258622b3043245884ed9d567124cef31ffc0571a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867500) * df058e289e348c3f48a1e82479cf90062c542b42 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066162) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9809: [FLINK-14282] Simplify DispatcherResourceManagerComponent hierarchy
flinkbot edited a comment on issue #9809: [FLINK-14282] Simplify DispatcherResourceManagerComponent hierarchy URL: https://github.com/apache/flink/pull/9809#issuecomment-536318695 ## CI report: * 531cd688ceb25b544833025d9b556a0d686b29c4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129599298) * a3ba53aa778e777d8f38750eb6a5e5147fd21654 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707662) * db3d62f7f74f3be33d65588f76a8a082f9fccfd0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837805) * 544d687c37a1597737fd35747d6fa7c5740c2c40 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129850857) * 5fdd2823b2bdbd6dcd7b8172369b41fe5438c3ce : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867317) * 8a8911aa471acb3cf551a5fd12bd14d506f6040a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066032) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9811: [FLINK-14285] Remove generics from Dispatcher factories
flinkbot edited a comment on issue #9811: [FLINK-14285] Remove generics from Dispatcher factories URL: https://github.com/apache/flink/pull/9811#issuecomment-536320588 ## CI report: * 658cddd4424505c1a904a926ad6e626abbd85cd7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600090) * 5b4df7e3bd41053dcce049edf420ed584a42b98a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707753) * 442ea9d2bcc705ba162ffe6bd7c0d127a372c06e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837865) * 2f46b0debc0a94a969510fbb07b8f1b5e6b50710 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850925) * 65e72a4a5a51920697475dbc3543642b0537cdd5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867404) * e1fd3451aafdd71e80472582a7916a8c24f11376 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066107) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9810: [FLINK-14284] Add shut down future to Dispatcher
flinkbot edited a comment on issue #9810: [FLINK-14284] Add shut down future to Dispatcher URL: https://github.com/apache/flink/pull/9810#issuecomment-536320582 ## CI report: * 3ea0f47e7dd76ec115db4ef583b416685107604b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600085) * e73e7e0a72952be963e29226bbb9b94487dafde6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707704) * eed7f48de7520824f5674d0959a91b1dd9caa654 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837834) * a4cecd0d80ce9728c96d70195e82a09958a0c90b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850891) * a908e21bc54f2052fc7b4b81261af2324fd32920 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867370) * 0fc658418704bdc15447a18db695551c3521421c : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066083) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9812: [FLINK-14286] Remove Akka specific parsing from LeaderConnectionInfo
flinkbot edited a comment on issue #9812: [FLINK-14286] Remove Akka specific parsing from LeaderConnectionInfo URL: https://github.com/apache/flink/pull/9812#issuecomment-536322371 ## CI report: * c1ca1131b02a2c555cd8386fd07aa1cfccbab161 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600831) * da85c13656e6cf02fcadbadf50df0c201f07970c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707797) * 2abee623fcedcbf8c61ce455cbf59bd59b75dcf4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837903) * c1efb87828fc98fa2f7ff5b07b082b9dfc67c501 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854578) * 9b2e7712bc24fca3f77aa02993aa3afb89de7bfc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129867450) * 2d706c3a58923435de08d0df8a3a938ec6709de2 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066133) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9808: [FLINK-14281] Add DispatcherRunner#getShutDownFuture
flinkbot edited a comment on issue #9808: [FLINK-14281] Add DispatcherRunner#getShutDownFuture URL: https://github.com/apache/flink/pull/9808#issuecomment-536316924 ## CI report: * 02c93d5d7d69324c33461040fec5c4014eb2b8d6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129598507) * 618f898c4a74c7f35f66ba7ab37e08cf8d66581e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707614) * 4f7e44bebc6f5dd1b222e68658e40b7f2002e5fd : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129834505) * ca5f4450fe3b2a33e39ad1b01b0381b8f7996d78 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850827) * 3d2171647776f5f8537f7e130026f14c99b3fef0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867287) * 79f00b2c099514054118c93214d91a857244c82c : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130066015) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9807: [FLINK-14280] Introduce DispatcherRunner
flinkbot edited a comment on issue #9807: [FLINK-14280] Introduce DispatcherRunner URL: https://github.com/apache/flink/pull/9807#issuecomment-536314902 ## CI report: * 9ef8fe986e4b06d6ae8512e5051933458363511c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129597654) * 748954650809fd2fa18572d1c925d3b647a55906 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707578) * 1ca7b625300da1a6a522fd1ba8427f5398f61ed5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129834475) * d0bd4cbe32866ed3f6cceafe6dd65133f96350b3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850793) * bf3c8311655872a9def94d46bf1363bc89c4715f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129867229) * a5b8cd87d6c2ce7a41531c34eae8ef78e1a3255f : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130065977) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14315) NPE with JobMaster.disconnectTaskManager
[
https://issues.apache.org/jira/browse/FLINK-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942975#comment-16942975
]
Zili Chen commented on FLINK-14315:
---
It looks like {{JobMaster#suspend}} races with {{JobMaster#onStop}} which
{{JobMaster#suspend}} set {{taskManagerHeartbeatManager = null}}
while
{{JobMaster#onStop}} calls {{JobMaster#disconnectTaskManager}} calls
{{taskManagerHeartbeatManager.unmonitorTarget(resourceID)}}.
Another perspective is we can properly tolerate zk unstable(connection loss
exception) as discussed in FLINK-10052 (it doesn't solve the problem here but
make it quite more rare)
CC [~trohrmann]
> NPE with JobMaster.disconnectTaskManager
>
>
> Key: FLINK-14315
> URL: https://issues.apache.org/jira/browse/FLINK-14315
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0
>Reporter: Steven Zhen Wu
>Priority: Major
>
> There was some connection issue with zookeeper that caused the job to
> restart. But shutdown failed with this fatal NPE, which seems to cause JVM
> to exit
> {code}
> 2019-10-02 16:16:19,134 INFO
> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Unable
> to read additional data from server sessionid 0x16d83374c4206f8, likely
> server has clo
> sed socket, closing socket connection and attempting reconnect
> 2019-10-02 16:16:19,234 INFO
> org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager
> - State change: SUSPENDED
> 2019-10-02 16:16:19,235 WARN
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Connection to ZooKeeper suspended. Can no longer retrieve the leader from
> ZooKeeper.
> 2019-10-02 16:16:19,235 WARN
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Connection to ZooKeeper suspended. Can no longer retrieve the leader from
> ZooKeeper.
> 2019-10-02 16:16:19,235 WARN
> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService -
> Connection to ZooKeeper suspended. The contender
> akka.tcp://flink@100.122.177.82:42043/u
> ser/dispatcher no longer participates in the leader election.
> 2019-10-02 16:16:19,237 INFO
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint-
> http://100.122.177.82:8081 lost leadership
> 2019-10-02 16:16:19,237 INFO
> com.netflix.spaas.runtime.resourcemanager.TitusResourceManager -
> ResourceManager akka.tcp://flink@100.122.177.82:42043/user/resourcemanager
> was revoked leadershi
> p. Clearing fencing token.
> 2019-10-02 16:16:19,237 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Stopping ZooKeeperLeaderRetrievalService
> /leader/e4e68f2b3fc40c7008cca624b2a2bab0/job_
> manager_lock.
> 2019-10-02 16:16:19,237 WARN
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore -
> ZooKeeper connection SUSPENDING. Changes to the submitted job graphs are not
> monitored (tem
> porarily).
> 2019-10-02 16:16:19,238 INFO
> org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager
> for job ksrouter (e4e68f2b3fc40c7008cca624b2a2bab0) was revoked leadership at
> akka.tcp:
> //flink@100.122.177.82:42043/user/jobmanager_0.
> 2019-10-02 16:16:19,239 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
> 2019-10-02 16:16:19,239 WARN
> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService -
> Connection to ZooKeeper suspended. The contender http://100.122.177.82:8081
> no longer pa
> rticipates in the leader election.
> 2019-10-02 16:16:19,239 WARN
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Connection to ZooKeeper suspended. Can no longer retrieve the leader from
> ZooKeeper.
> 2019-10-02 16:16:19,239 WARN
> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService -
> Connection to ZooKeeper suspended. The contender
> akka.tcp://flink@100.122.177.82:42043/u
> ser/jobmanager_0 no longer participates in the leader election.
> 2019-10-02 16:16:19,239 WARN
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Connection to ZooKeeper suspended. Can no longer retrieve the leader from
> ZooKeeper.
> 2019-10-02 16:16:19,239 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job ksrouter
> (e4e68f2b3fc40c7008cca624b2a2bab0) switched from state RUNNING to SUSPENDED.
> org.apache.flink.util.FlinkException: JobManager is no longer the leader.
> at
> org.apache.flink.runtime.jobmaster.JobManagerRunner.revokeJobMasterLeadership(JobManagerRunner.java:391)
> at
>
[GitHub] [flink] flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session
flinkbot edited a comment on issue #9832: [FLINK-11843] Bind lifespan of Dispatcher to leader session URL: https://github.com/apache/flink/pull/9832#issuecomment-537049332 ## CI report: * abaae048fef753455970fac9d6ab421b660b0536 : UNKNOWN * b96c63552ccd322adae7a41a410615e95b538ece : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867595) * 2c95a3939dbf0259d694af6c69451f0ede3c3891 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9831: [FLINK-14278] Extend DispatcherResourceManagerComponentFactory.create to take ioExecutor
flinkbot edited a comment on issue #9831: [FLINK-14278] Extend DispatcherResourceManagerComponentFactory.create to take ioExecutor URL: https://github.com/apache/flink/pull/9831#issuecomment-537027384 ## CI report: * e1d37ab03c1bc62b3d45c45ee11d616ed2c78b0c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129858618) * ba5dc1be84ded55f91fad1698e0edb2ca7416add : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9830: [FLINK-14307] Extract JobGraphWriter from JobGraphStore
flinkbot edited a comment on issue #9830: [FLINK-14307] Extract JobGraphWriter from JobGraphStore URL: https://github.com/apache/flink/pull/9830#issuecomment-537017897 ## CI report: * ed86e6d768a1353423d969e6c9dcf5fd967c4f19 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129854718) * 9892be4548e684c87dc586b96adbc3fb6052d7c7 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9821: [FLINK-14298] Replace LeaderContender#getAddress with #getDescription
flinkbot edited a comment on issue #9821: [FLINK-14298] Replace LeaderContender#getAddress with #getDescription URL: https://github.com/apache/flink/pull/9821#issuecomment-536618253 ## CI report: * 9ca9df3eaa22965444373a3b6142798ca5559f50 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129712419) * e2b8bb516e06b5337d1f6172ac00c4f4b42f160a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129808007) * 7ded64e1fca1abbbcdeb55604e2ab60e351e23c0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837967) * 4b668e45a1350037e7d738314dd5889f71098a8b : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129854643) * 7e004fc6d0620152b3083019fedad411172b6077 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867555) * 005137c22c57f7e1f92f901571f72916b1d10110 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9813: [FLINK-14287] Decouple leader address from LeaderContender
flinkbot edited a comment on issue #9813: [FLINK-14287] Decouple leader address from LeaderContender URL: https://github.com/apache/flink/pull/9813#issuecomment-536324105 ## CI report: * b7a7e7b6dc8a16d23097911aa98852e2cf4d9c00 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129601641) * 3b8410b82ec366014551e35fb8791c6ad623faf9 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707833) * 96ef0ac36eb37792fd7837619cbca7d8a1272090 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837942) * 4c3d9db5f51b815c810be6fe6ae82c62452f14b4 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/129854608) * 258622b3043245884ed9d567124cef31ffc0571a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867500) * df058e289e348c3f48a1e82479cf90062c542b42 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9812: [FLINK-14286] Remove Akka specific parsing from LeaderConnectionInfo
flinkbot edited a comment on issue #9812: [FLINK-14286] Remove Akka specific parsing from LeaderConnectionInfo URL: https://github.com/apache/flink/pull/9812#issuecomment-536322371 ## CI report: * c1ca1131b02a2c555cd8386fd07aa1cfccbab161 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600831) * da85c13656e6cf02fcadbadf50df0c201f07970c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707797) * 2abee623fcedcbf8c61ce455cbf59bd59b75dcf4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837903) * c1efb87828fc98fa2f7ff5b07b082b9dfc67c501 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854578) * 9b2e7712bc24fca3f77aa02993aa3afb89de7bfc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129867450) * 2d706c3a58923435de08d0df8a3a938ec6709de2 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9810: [FLINK-14284] Add shut down future to Dispatcher
flinkbot edited a comment on issue #9810: [FLINK-14284] Add shut down future to Dispatcher URL: https://github.com/apache/flink/pull/9810#issuecomment-536320582 ## CI report: * 3ea0f47e7dd76ec115db4ef583b416685107604b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600085) * e73e7e0a72952be963e29226bbb9b94487dafde6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707704) * eed7f48de7520824f5674d0959a91b1dd9caa654 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837834) * a4cecd0d80ce9728c96d70195e82a09958a0c90b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850891) * a908e21bc54f2052fc7b4b81261af2324fd32920 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867370) * 0fc658418704bdc15447a18db695551c3521421c : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9811: [FLINK-14285] Remove generics from Dispatcher factories
flinkbot edited a comment on issue #9811: [FLINK-14285] Remove generics from Dispatcher factories URL: https://github.com/apache/flink/pull/9811#issuecomment-536320588 ## CI report: * 658cddd4424505c1a904a926ad6e626abbd85cd7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129600090) * 5b4df7e3bd41053dcce049edf420ed584a42b98a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707753) * 442ea9d2bcc705ba162ffe6bd7c0d127a372c06e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837865) * 2f46b0debc0a94a969510fbb07b8f1b5e6b50710 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850925) * 65e72a4a5a51920697475dbc3543642b0537cdd5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867404) * e1fd3451aafdd71e80472582a7916a8c24f11376 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9808: [FLINK-14281] Add DispatcherRunner#getShutDownFuture
flinkbot edited a comment on issue #9808: [FLINK-14281] Add DispatcherRunner#getShutDownFuture URL: https://github.com/apache/flink/pull/9808#issuecomment-536316924 ## CI report: * 02c93d5d7d69324c33461040fec5c4014eb2b8d6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129598507) * 618f898c4a74c7f35f66ba7ab37e08cf8d66581e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707614) * 4f7e44bebc6f5dd1b222e68658e40b7f2002e5fd : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129834505) * ca5f4450fe3b2a33e39ad1b01b0381b8f7996d78 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850827) * 3d2171647776f5f8537f7e130026f14c99b3fef0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867287) * 79f00b2c099514054118c93214d91a857244c82c : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9809: [FLINK-14282] Simplify DispatcherResourceManagerComponent hierarchy
flinkbot edited a comment on issue #9809: [FLINK-14282] Simplify DispatcherResourceManagerComponent hierarchy URL: https://github.com/apache/flink/pull/9809#issuecomment-536318695 ## CI report: * 531cd688ceb25b544833025d9b556a0d686b29c4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129599298) * a3ba53aa778e777d8f38750eb6a5e5147fd21654 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129707662) * db3d62f7f74f3be33d65588f76a8a082f9fccfd0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129837805) * 544d687c37a1597737fd35747d6fa7c5740c2c40 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129850857) * 5fdd2823b2bdbd6dcd7b8172369b41fe5438c3ce : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129867317) * 8a8911aa471acb3cf551a5fd12bd14d506f6040a : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9807: [FLINK-14280] Introduce DispatcherRunner
flinkbot edited a comment on issue #9807: [FLINK-14280] Introduce DispatcherRunner URL: https://github.com/apache/flink/pull/9807#issuecomment-536314902 ## CI report: * 9ef8fe986e4b06d6ae8512e5051933458363511c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129597654) * 748954650809fd2fa18572d1c925d3b647a55906 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129707578) * 1ca7b625300da1a6a522fd1ba8427f5398f61ed5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129834475) * d0bd4cbe32866ed3f6cceafe6dd65133f96350b3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129850793) * bf3c8311655872a9def94d46bf1363bc89c4715f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129867229) * a5b8cd87d6c2ce7a41531c34eae8ef78e1a3255f : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14271) Deprecate legacy RestartPipelinedRegionStrategy
[
https://issues.apache.org/jira/browse/FLINK-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942945#comment-16942945
]
Steven Zhen Wu commented on FLINK-14271:
[~zhuzh] we don't use the region failover or
`AdaptedRestartPipelinedRegionStrategyNG`. We do use fine-grained recovery
feature for embarrassingly parallel jobs by setting
`jobmanager.execution.failover-strategy: indivisual`
> Deprecate legacy RestartPipelinedRegionStrategy
> ---
>
> Key: FLINK-14271
> URL: https://issues.apache.org/jira/browse/FLINK-14271
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> The legacy {{RestartPipelinedRegionStrategy}} has been superseded by
> {{AdaptedRestartPipelinedRegionStrategyNG}} in Flink 1.9.
> It heavily depends on ExecutionGraph components and becomes a blocker for a
> clean scheduler re-architecture.
> We should deprecate it for further removal.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher
flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher URL: https://github.com/apache/flink/pull/9828#issuecomment-536958155 ## CI report: * 582fcdf7a275b9416d2e5f947352bf1310ff5d88 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129830601) * 12bfe6e05cc2cf1e92d4f99ac9d41964026388ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854671) * 7529618b93035b1238efb7795813a2f550cda04a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130034507) * 5c23b740d76a863ec1022051cccff4d60f9e15c8 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130048881) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher
tillrohrmann commented on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher URL: https://github.com/apache/flink/pull/9828#issuecomment-537563647 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher
tillrohrmann closed pull request #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher URL: https://github.com/apache/flink/pull/9828 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager
tillrohrmann commented on issue #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager URL: https://github.com/apache/flink/pull/9827#issuecomment-537563534 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager
tillrohrmann closed pull request #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager URL: https://github.com/apache/flink/pull/9827 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #9825: [FLINK-14299] Introduce ProcessMetricGroup
tillrohrmann commented on issue #9825: [FLINK-14299] Introduce ProcessMetricGroup URL: https://github.com/apache/flink/pull/9825#issuecomment-537563432 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #9825: [FLINK-14299] Introduce ProcessMetricGroup
tillrohrmann closed pull request #9825: [FLINK-14299] Introduce ProcessMetricGroup URL: https://github.com/apache/flink/pull/9825 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14299) Factor status and system metrics out of JobManagerMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-14299.
-
Resolution: Done
Done via 972576954e9e3d8d6d6c5e23a6fcad9d60b0a6c8
> Factor status and system metrics out of JobManagerMetricGroup
> -
>
> Key: FLINK-14299
> URL: https://issues.apache.org/jira/browse/FLINK-14299
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Metrics
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At the moment, we use the {{JobManagerMetricGroup}} to not only register
> {{Dispatcher}} specific metrics but also process specific metrics such as
> CPU, threads, memory, etc. Due to this fact, it is not possible to close the
> {{JobManagerMetricGroup}} when the life time of the {{Dispatcher}}
> terminates. In order to do this, I suggest to introduce a new
> {{ProcessMetricGroup}} which is used to register the process specific
> metrics.
> In order to guarantee backwards compatibility, I suggest to use the same
> scope format as {{SCOPE_NAMING_JM}} and then appending {{.Status}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14303) Factor resource manager related metrics out of JobManagerMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942932#comment-16942932
]
Till Rohrmann edited comment on FLINK-14303 at 10/2/19 4:02 PM:
Done via
7eda4fc0be962d8af5d87c4120a4b5a2a30b9c11
a682254dbdbe4defa3560ec949401622d93a0a83
was (Author: till.rohrmann):
Done via 7eda4fc0be962d8af5d87c4120a4b5a2a30b9c11
> Factor resource manager related metrics out of JobManagerMetricGroup
>
>
> Key: FLINK-14303
> URL: https://issues.apache.org/jira/browse/FLINK-14303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Metrics
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At the moment, we use the {{JobManagerMetricGroup}} to report resource
> manager specific metrics such as the number of slots or registered
> {{TaskExecutors}}. Since the {{JobManagerMetricGroup}} is shared between the
> {{ResourceManager}} and the {{Dispatcher}} this prevents closing the
> {{JobManagerMetricGroup}} from the {{Dispatcher}}.
> In order to solve this problem, I propose to introduce a
> {{ResourceManagerMetricGroup}} which is used by the {{ResourceManager}} to
> register resource manager specific metrics. In order to not break existing
> third party applications which rely on the current metrics layout, the
> {{ResourceManagerMetricGroup}} will use the scope and group name of the
> {{JobManagerMetricGroup}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14303) Factor resource manager related metrics out of JobManagerMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-14303.
-
Resolution: Done
Done via 7eda4fc0be962d8af5d87c4120a4b5a2a30b9c11
> Factor resource manager related metrics out of JobManagerMetricGroup
>
>
> Key: FLINK-14303
> URL: https://issues.apache.org/jira/browse/FLINK-14303
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Metrics
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> At the moment, we use the {{JobManagerMetricGroup}} to report resource
> manager specific metrics such as the number of slots or registered
> {{TaskExecutors}}. Since the {{JobManagerMetricGroup}} is shared between the
> {{ResourceManager}} and the {{Dispatcher}} this prevents closing the
> {{JobManagerMetricGroup}} from the {{Dispatcher}}.
> In order to solve this problem, I propose to introduce a
> {{ResourceManagerMetricGroup}} which is used by the {{ResourceManager}} to
> register resource manager specific metrics. In order to not break existing
> third party applications which rely on the current metrics layout, the
> {{ResourceManagerMetricGroup}} will use the scope and group name of the
> {{JobManagerMetricGroup}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14305) Move ownership of JobManagerMetricGroup to Dispatcher
[
https://issues.apache.org/jira/browse/FLINK-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-14305.
-
Resolution: Done
Done via 52e5d7c4fa479adbb48cf83a9f09d78978ddd8b7
> Move ownership of JobManagerMetricGroup to Dispatcher
> -
>
> Key: FLINK-14305
> URL: https://issues.apache.org/jira/browse/FLINK-14305
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Metrics
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With FLINK-14303 and FLINK-14299, it is now possible to move the ownership of
> the {{JobManagerMetricGroup}} into the {{Dispatcher}}. This makes the
> lifespan of the metric group shorter and the lifecycle management easier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-14247) Use NoResourceAvailableException to wrap TimeoutException on slot allocation (Scheduler NG)
[
https://issues.apache.org/jira/browse/FLINK-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao resolved FLINK-14247.
--
Resolution: Fixed
1.10: 013f8a14d3e6a46e1565c527dda96d90126425bd
> Use NoResourceAvailableException to wrap TimeoutException on slot allocation
> (Scheduler NG)
>
>
> Key: FLINK-14247
> URL: https://issues.apache.org/jira/browse/FLINK-14247
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This makes the error to be more user friendly.
> It also helps {{MiniClusterITCases}} to pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14247) Use NoResourceAvailableException to wrap TimeoutException on slot allocation (Scheduler NG)
[
https://issues.apache.org/jira/browse/FLINK-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14247:
---
Labels: pull-request-available (was: )
> Use NoResourceAvailableException to wrap TimeoutException on slot allocation
> (Scheduler NG)
>
>
> Key: FLINK-14247
> URL: https://issues.apache.org/jira/browse/FLINK-14247
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
>
> This makes the error to be more user friendly.
> It also helps {{MiniClusterITCases}} to pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14247) Use NoResourceAvailableException to wrap TimeoutException on slot allocation (Scheduler NG)
[
https://issues.apache.org/jira/browse/FLINK-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-14247.
> Use NoResourceAvailableException to wrap TimeoutException on slot allocation
> (Scheduler NG)
>
>
> Key: FLINK-14247
> URL: https://issues.apache.org/jira/browse/FLINK-14247
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This makes the error to be more user friendly.
> It also helps {{MiniClusterITCases}} to pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] GJL closed pull request #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout
GJL closed pull request #9794: [FLINK-14247][runtime] Use NoResourceAvailableException to wrap TimeoutException on slot allocation timeout URL: https://github.com/apache/flink/pull/9794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-7151) Add a basic function SQL DDL
[
https://issues.apache.org/jira/browse/FLINK-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942925#comment-16942925
]
Zhenqiu Huang commented on FLINK-7151:
--
[~phoenixjiangnan][~danny0405][~suez1224]
Resolved comments and added integration tests for CatalogFunctionITCase. Please
review it again.
> Add a basic function SQL DDL
>
>
> Key: FLINK-7151
> URL: https://issues.apache.org/jira/browse/FLINK-7151
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: yuemeng
>Assignee: Zhenqiu Huang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Based on create function and table.we can register a udf,udaf,udtf use sql:
> {code}
> CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS
> class_name;
> DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name;
> ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO
> new_name;
> {code}
> {code}
> CREATE function 'TOPK' AS
> 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF';
> INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP
> BY id;
> {code}
> This ticket can assume that the function class is already loaded in classpath
> by users. Advanced syntax like to how to dynamically load udf libraries from
> external locations can be on a separate ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime]
Let LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330592182
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
It just came to me that it can be problematic if we keep the lazy resetting.
e.g. The job A1--pipelined-->B1 fails. And only A1 will be re-scheduled on
`restartTasks()` since the inputs of B1 are not ready. B1 should be scheduled
later on the partition consumable event from restarted A1. But the terminal
state of B1 will prevent B1 from being scheduled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9717: [FLINK-14044] [runtime] Reducing synchronization in AsyncWaitOperator
flinkbot edited a comment on issue #9717: [FLINK-14044] [runtime] Reducing synchronization in AsyncWaitOperator URL: https://github.com/apache/flink/pull/9717#issuecomment-533068819 ## CI report: * 9cfda801891969ac460f45ea639d636b519f22db : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128317194) * af7f6be848f0bd90a049d5ee9f7a38b1c3e2b972 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128336908) * 43157eb165d9409fbdf4a2f773ef7d52dd74e759 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128365390) * ff400d149ed36c63a90d2f3fcd517bde738e5af1 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128402339) * 02dbf61e50e6c913610df7f586d7eb0f20529c13 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128406923) * 68cf941c7e165a5c6b2a27f7ac716b29f76d1918 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128472867) * 54aa0043ec73b03580eda2e0310b44bdee352a55 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128495989) * e081ea5c536d6ebb828eb8b68c404379d23f5aef : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128502020) * 911bdd80f4b1cbb6cee2ffb517cafc5e3c83ee0b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128622324) * 40fd94b291968062eec85a9436a3a755914d282a : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128904653) * e60ea55f913143611d9162fd79a75cfa575aa9f0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128907420) * c2c7b2cc91961bc2c8707f9077c4d66ea6535b8e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129102072) * 1048cb10486f94983a52582953379afa02a3b350 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129122517) * 773eb07af5ac848ec70cd32663ebf657b0419f3e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129823477) * 7c4191061dbacaf142f2b3e6fa8dec30d7e16595 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/130030060) * d8cee5bd77379a05ba6a501444f7285e73a4be43 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130034555) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher
flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher URL: https://github.com/apache/flink/pull/9828#issuecomment-536958155 ## CI report: * 582fcdf7a275b9416d2e5f947352bf1310ff5d88 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129830601) * 12bfe6e05cc2cf1e92d4f99ac9d41964026388ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854671) * 7529618b93035b1238efb7795813a2f550cda04a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130034507) * 5c23b740d76a863ec1022051cccff4d60f9e15c8 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130048881) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime]
Let LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330592182
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
It just came to me that it can be problematic if we keep the lazy resetting.
e.g. The job A1--pipelined-->B1 fails. And only A1 will be re-scheduled on
`restartTasks()` since the inputs of B1 are not ready. B1 should be scheduled
later on the partition consumable event from restarted A1. But the terminal
state of B1 will prevent B1 from being scheduled.
Seems we lack a test for this case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime]
Let LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330612434
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
And keeping a vertex in FAILED/CANCELED for a long time is also not good to
users, which can happen if it takes a long time for its inputs to become ready
again.
How about closing FLINK-14248 and open a new ticket to change the resetting
to happen before `restartTasks()`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher
flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher URL: https://github.com/apache/flink/pull/9828#issuecomment-536958155 ## CI report: * 582fcdf7a275b9416d2e5f947352bf1310ff5d88 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129830601) * 12bfe6e05cc2cf1e92d4f99ac9d41964026388ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854671) * 7529618b93035b1238efb7795813a2f550cda04a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130034507) * 5c23b740d76a863ec1022051cccff4d60f9e15c8 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9825: [FLINK-14299] Introduce ProcessMetricGroup
flinkbot edited a comment on issue #9825: [FLINK-14299] Introduce ProcessMetricGroup URL: https://github.com/apache/flink/pull/9825#issuecomment-536938289 ## CI report: * 53c41b006ae2155206a7843b67721b598b3591d1 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129823434) * 6039a68165fc94b780c8129bd01844ae7028e97a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/130034403) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime]
Let LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330592182
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
It just came to me that it can be problematic if we keep the lazy resetting.
e.g. The job A1--pipelined-->B1 fails. And only A1 will be re-scheduled on
`restartTasks()` since the inputs of B1 are not ready. B1 should be scheduled
later on the partition consumable event from restarted A1. But the terminal
state of B1 will prevent B1 from being scheduled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
zhuzhurk commented on a change in pull request #9791: [FLINK-14248][runtime]
Let LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330588245
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
I see your concerns.
I'd also prefer to reset vertices before invoking `restartTasks()` on them.
And the contract can be simplified that the schedulingStrategy should only
schedule vertices which is in `CREATED` state.
Previously I thought we have made the decision to do lazy resetting. But now
I think it's reasonable to change this behaviour.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager
flinkbot edited a comment on issue #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager URL: https://github.com/apache/flink/pull/9827#issuecomment-536948437 ## CI report: * fa367aaeb2089366de1d9497f1bbec9b11d75e72 : UNKNOWN * f865d19c9a7c1530d06d194652582ebe7257176a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129826911) * 53201b462b67f199c628367918e19c475204765d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/130034456) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state. I am looking for reasons why this is a bad idea.
Moreover, it seems that we could replace `IS_IN_TERMINAL_STATE` with `x ->
true`, and the algorithm would still work.
The bottom line is, we have to decide on a contract: Before invoking
`restartTasks()`, should all vertices be in `CREATED` or in a terminal state?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state. I am looking for reasons why this is a bad idea.
Moreover, it seems that we could replace `IS_IN_TERMINAL_STATE` with `x ->
true`, and the algorithm would still work.
We have to decide on a contract. Before invoking `restartTasks()`, should
all vertices be in `CREATED` or in a terminal state?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state. I am looking for reasons why this is a bad idea.
Moreover, it seems that we could replace `IS_IN_TERMINAL_STATE` with `x ->
true`, and the algorithm would still work.
The bottom line is, we have to decide on a contract. Before invoking
`restartTasks()`, should all vertices be in `CREATED` or in a terminal state?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state. I am looking for reasons why this is a bad idea.
Moreover, it seems that we could replace `IS_IN_TERMINAL_STATE` with `x ->
true`, and the algorithm would still work.
We have to decide on a contract. Before invoking `restartTasks()`, should
all vertices be in `CREATED` or in a terminal state.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9833: [FLINK-14276][quickstarts] Scala quickstart compiles on JDK 11
tillrohrmann commented on a change in pull request #9833: [FLINK-14276][quickstarts] Scala quickstart compiles on JDK 11 URL: https://github.com/apache/flink/pull/9833#discussion_r330580313 ## File path: flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml ## @@ -171,6 +171,11 @@ under the License. + + + -nobootcp Review comment: Do you know what affects this flag has for Java 8? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942853#comment-16942853 ] Zili Chen edited comment on FLINK-13417 at 10/2/19 2:20 PM: Run against master repo but it seems not trigger corn job. Could you help trigger corn job on zk-3,5-shaded branch [~trohrmann]? Although we should start a discussion about whether or not and if so, how to migrate from zk 3.4 to zk 3.5(since it enforces our users to bump their zk server version), we can concurrently verify that if there is any technical broker to do so. was (Author: tison): Run against master repo but it seems not trigger corn job. Could you help trigger corn job on zk-3,5-shaded branch [~trohrmann]? Although we should take several discussion about whether or not and if so, how to migrate from zk 3.4 to zk 3.5(since it enforces our users to bump their zk server version), we can concurrently verify that if there is any technical broker to do so. > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.10.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942853#comment-16942853 ] Zili Chen commented on FLINK-13417: --- Run against master repo but it seems not trigger corn job. Could you help trigger corn job on zk-3,5-shaded branch [~trohrmann]? Although we should take several discussion about whether or not and if so, how to migrate from zk 3.4 to zk 3.5(since it enforces our users to bump their zk server version), we can concurrently verify that if there is any technical broker to do so. > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.10.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state. I am looking for reasons why this is a bad idea.
Moreover, it seems that we could replace `IS_IN_TERMINAL_STATE` with `x ->
true`, and the algorithm would still work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #9833: [FLINK-14276][quickstarts] Scala quickstart compiles on JDK 11
zentol commented on a change in pull request #9833: [FLINK-14276][quickstarts] Scala quickstart compiles on JDK 11 URL: https://github.com/apache/flink/pull/9833#discussion_r330575826 ## File path: flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml ## @@ -171,6 +171,11 @@ under the License. + + + -nobootcp Review comment: Actually, yes it would be tricky to make it java version dependent and future-proof. Ideally, we'd have it in by default (future-proof) and remove it for java 8, but that's kinda tricky. Removing things based on a profile only works in some cases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9717: [FLINK-14044] [runtime] Reducing synchronization in AsyncWaitOperator
flinkbot edited a comment on issue #9717: [FLINK-14044] [runtime] Reducing synchronization in AsyncWaitOperator URL: https://github.com/apache/flink/pull/9717#issuecomment-533068819 ## CI report: * 9cfda801891969ac460f45ea639d636b519f22db : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128317194) * af7f6be848f0bd90a049d5ee9f7a38b1c3e2b972 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128336908) * 43157eb165d9409fbdf4a2f773ef7d52dd74e759 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128365390) * ff400d149ed36c63a90d2f3fcd517bde738e5af1 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128402339) * 02dbf61e50e6c913610df7f586d7eb0f20529c13 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128406923) * 68cf941c7e165a5c6b2a27f7ac716b29f76d1918 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128472867) * 54aa0043ec73b03580eda2e0310b44bdee352a55 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/128495989) * e081ea5c536d6ebb828eb8b68c404379d23f5aef : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128502020) * 911bdd80f4b1cbb6cee2ffb517cafc5e3c83ee0b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128622324) * 40fd94b291968062eec85a9436a3a755914d282a : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/128904653) * e60ea55f913143611d9162fd79a75cfa575aa9f0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/128907420) * c2c7b2cc91961bc2c8707f9077c4d66ea6535b8e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129102072) * 1048cb10486f94983a52582953379afa02a3b350 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129122517) * 773eb07af5ac848ec70cd32663ebf657b0419f3e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129823477) * 7c4191061dbacaf142f2b3e6fa8dec30d7e16595 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/130030060) * d8cee5bd77379a05ba6a501444f7285e73a4be43 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130034555) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let LazyFromSourcesSchedulingStrategy restart terminated tasks
GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set verticesToRestart) {
+ public void restartTasks(final Set
verticesToRestart) {
+ final Set verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #9833: [FLINK-14276][quickstarts] Scala quickstart compiles on JDK 11
zentol commented on a change in pull request #9833: [FLINK-14276][quickstarts] Scala quickstart compiles on JDK 11 URL: https://github.com/apache/flink/pull/9833#discussion_r330573933 ## File path: flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml ## @@ -171,6 +171,11 @@ under the License. + + + -nobootcp Review comment: well we could add a profile that adds this flag, but it would be inconsistent with the main Flink pom. And given that this is required since Java 9 I think it's fine to have it always on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-13515) ClassLoaderITCase fails on Java 11
[
https://issues.apache.org/jira/browse/FLINK-13515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-13515.
Resolution: Fixed
master: c552f74b0daf76b80e85b6eb062c80e23e532315
> ClassLoaderITCase fails on Java 11
> --
>
> Key: FLINK-13515
> URL: https://issues.apache.org/jira/browse/FLINK-13515
> Project: Flink
> Issue Type: Sub-task
> Components: Command Line Client, Tests
>Reporter: Chesnay Schepler
>Assignee: Haibo Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{ClassLoaderITCas#testCheckpointedStreamingClassloaderJobWithCustomClassLoader}}
> fails on Java 11 because the usercode exception can be serialized in the
> client. This shouldn't be possible since the user-jar isn't on the classpath
> of the client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher
flinkbot edited a comment on issue #9828: [FLINK-14305] Transfer ownership of JobManagerMetricGroup to Dispatcher URL: https://github.com/apache/flink/pull/9828#issuecomment-536958155 ## CI report: * 582fcdf7a275b9416d2e5f947352bf1310ff5d88 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129830601) * 12bfe6e05cc2cf1e92d4f99ac9d41964026388ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/129854671) * 7529618b93035b1238efb7795813a2f550cda04a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130034507) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol merged pull request #9744: [FLINK-13515][test] Fix ClassLoaderITCase fails on Java 11
zentol merged pull request #9744: [FLINK-13515][test] Fix ClassLoaderITCase fails on Java 11 URL: https://github.com/apache/flink/pull/9744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager
flinkbot edited a comment on issue #9827: [FLINK-14303][metrics] Replace JobManagerMetricGroup with ResourceManagerMetricGroup in ResourceManager URL: https://github.com/apache/flink/pull/9827#issuecomment-536948437 ## CI report: * fa367aaeb2089366de1d9497f1bbec9b11d75e72 : UNKNOWN * f865d19c9a7c1530d06d194652582ebe7257176a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/129826911) * 53201b462b67f199c628367918e19c475204765d : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/130034456) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
