[GitHub] zentol commented on issue #7106: [hotfix] [docs] fix error about per-job YARN cluster
zentol commented on issue #7106: [hotfix] [docs] fix error about per-job YARN cluster URL: https://github.com/apache/flink/pull/7106#issuecomment-439800604 @tillrohrmann Is this a bug, or part of this whole "job-cluster is actually a session-cluster" deal? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zentol closed pull request #7137: [hotfix] Fix typo in FlinkKafkaConsumerBase.
zentol closed pull request #7137: [hotfix] Fix typo in FlinkKafkaConsumerBase. URL: https://github.com/apache/flink/pull/7137 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java index cfb5b6d510d..d6686c952be 100644 --- a/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java +++ b/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java @@ -326,7 +326,7 @@ public FlinkKafkaConsumerBase( * * This setting will only have effect if checkpointing is enabled for the job. * If checkpointing isn't enabled, only the "auto.commit.enable" (for 0.8) / "enable.auto.commit" (for 0.9+) -* property settings will be +* property settings will be used. * * @return The consumer object, to allow function chaining. */ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-10916) Include duplicated user-specified uid into error message
[
https://issues.apache.org/jira/browse/FLINK-10916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-10916.
Resolution: Fixed
Fix Version/s: 1.7.0
1.6.3
1.5.6
master: 8eb6cfff9dfcc5a819c03b67db7ce879945341f0
1.7: 3c1f14ebecf74e7e014a11c8f790061dd0a79b2d
1.6: 73f0fd2b4515d2027ba2843c0c61800257bd2699
1.5: b72c8f2e6ae7eed42c5725848516286769e61854
> Include duplicated user-specified uid into error message
>
>
> Key: FLINK-10916
> URL: https://issues.apache.org/jira/browse/FLINK-10916
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Kirill Mikheev
>Priority: Trivial
> Labels: easyfix, newbie, pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0
>
>
> If user provides non-unique uid for a stream, error message does not give a
> clue where does the error reside in client code. Including user-provided uid
> into error message should narrow the search area.
> In 1.6.2 we see the message that was introduced in 1.2.0:
> {noformat}
> java.lang.IllegalArgumentException: Hash collision on user-specified ID. Most
> likely cause is a non-unique ID. Please check that all IDs specified via
> uid(String) are unique. at
> org.apache.flink.streaming.api.graph.StreamGraphHasherV2.generateNodeHash(StreamGraphHasherV2.java:180)
> at
> org.apache.flink.streaming.api.graph.StreamGraphHasherV2.traverseStreamGraphAndGenerateHashes(StreamGraphHasherV2.java:112)
> at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:134)
> at .
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10923) Make SlotPool Status Machine-Readable
Konstantin Knauf created FLINK-10923:
Summary: Make SlotPool Status Machine-Readable
Key: FLINK-10923
URL: https://issues.apache.org/jira/browse/FLINK-10923
Project: Flink
Issue Type: Improvement
Reporter: Konstantin Knauf
{{On DEBUG level org.apache.flink.runtime.jobmaster.slotpool.SlotPool}} logs a
slot pool status message every minute. For large deployments it would be
helpful, if this message was in a machine-readable format (like JSON).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9925) ClientTest.testSimpleRequests fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-9925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691345#comment-16691345
]
Chesnay Schepler commented on FLINK-9925:
-
Another instance on release-1.7@ef683c3ea56b5e82971035790955d895d93ccd5a:
https://travis-ci.org/apache/flink/jobs/456572019
{code}
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.186 sec <<<
FAILURE! - in org.apache.flink.queryablestate.network.ClientTest
testSimpleRequests(org.apache.flink.queryablestate.network.ClientTest) Time
elapsed: 20.004 sec <<< FAILURE!
java.lang.AssertionError: Receive timed out
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:712)
at
org.apache.flink.queryablestate.network.ClientTest.testSimpleRequests(ClientTest.java:170)
{code}
> ClientTest.testSimpleRequests fails on Travis
> -
>
> Key: FLINK-9925
> URL: https://issues.apache.org/jira/browse/FLINK-9925
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
>
> {{ClientTest.testSimpleRequests}} fails on Travis with an {{AssertionError}}:
> https://api.travis-ci.org/v3/job/405690023/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10916) Include duplicated user-specified uid into error message
[
https://issues.apache.org/jira/browse/FLINK-10916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691341#comment-16691341
]
ASF GitHub Bot commented on FLINK-10916:
zentol closed pull request #7133: [FLINK-10916] [DataStream API] Include
duplicated user-specified uid into error message
URL: https://github.com/apache/flink/pull/7133
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
index d7c51bcbdeb..a87f49fc7b8 100644
---
a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
+++
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
@@ -177,7 +177,8 @@ private boolean generateNodeHash(
for (byte[] previousHash : hashes.values()) {
if (Arrays.equals(previousHash, hash)) {
- throw new
IllegalArgumentException("Hash collision on user-specified ID. " +
+ throw new
IllegalArgumentException("Hash collision on user-specified ID " +
+ "\"" +
userSpecifiedHash + "\". " +
"Most likely cause is a
non-unique ID. Please check that all IDs " +
"specified via
`uid(String)` are unique.");
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Include duplicated user-specified uid into error message
>
>
> Key: FLINK-10916
> URL: https://issues.apache.org/jira/browse/FLINK-10916
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Kirill Mikheev
>Priority: Trivial
> Labels: easyfix, newbie, pull-request-available
>
> If user provides non-unique uid for a stream, error message does not give a
> clue where does the error reside in client code. Including user-provided uid
> into error message should narrow the search area.
> In 1.6.2 we see the message that was introduced in 1.2.0:
> {noformat}
> java.lang.IllegalArgumentException: Hash collision on user-specified ID. Most
> likely cause is a non-unique ID. Please check that all IDs specified via
> uid(String) are unique. at
> org.apache.flink.streaming.api.graph.StreamGraphHasherV2.generateNodeHash(StreamGraphHasherV2.java:180)
> at
> org.apache.flink.streaming.api.graph.StreamGraphHasherV2.traverseStreamGraphAndGenerateHashes(StreamGraphHasherV2.java:112)
> at
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:134)
> at .
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol closed pull request #7133: [FLINK-10916] [DataStream API] Include duplicated user-specified uid into error message
zentol closed pull request #7133: [FLINK-10916] [DataStream API] Include
duplicated user-specified uid into error message
URL: https://github.com/apache/flink/pull/7133
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
index d7c51bcbdeb..a87f49fc7b8 100644
---
a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
+++
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphHasherV2.java
@@ -177,7 +177,8 @@ private boolean generateNodeHash(
for (byte[] previousHash : hashes.values()) {
if (Arrays.equals(previousHash, hash)) {
- throw new
IllegalArgumentException("Hash collision on user-specified ID. " +
+ throw new
IllegalArgumentException("Hash collision on user-specified ID " +
+ "\"" +
userSpecifiedHash + "\". " +
"Most likely cause is a
non-unique ID. Please check that all IDs " +
"specified via
`uid(String)` are unique.");
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] link3280 opened a new pull request #7137: [hotfix] Fix typo in FlinkKafkaConsumerBase.
link3280 opened a new pull request #7137: [hotfix] Fix typo in FlinkKafkaConsumerBase. URL: https://github.com/apache/flink/pull/7137 This PR fixes a typo in FlinkKafkaConsumerBase.java. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10791) Provide end-to-end test for Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691305#comment-16691305 ] ASF GitHub Bot commented on FLINK-10791: yanghua commented on issue #7038: [FLINK-10791] Provide end-to-end test for Kafka 0.11 connector URL: https://github.com/apache/flink/pull/7038#issuecomment-439790249 @tillrohrmann I agree with your opinion. I have created a new issue (FLINK-10922) to rework specific modules (0.10, 0.11 and 2.0). Will submit PR as soon as possible. cc @pnowojski This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide end-to-end test for Kafka 0.11 connector > > > Key: FLINK-10791 > URL: https://issues.apache.org/jira/browse/FLINK-10791 > Project: Flink > Issue Type: Test > Components: E2E Tests >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua commented on issue #7038: [FLINK-10791] Provide end-to-end test for Kafka 0.11 connector
yanghua commented on issue #7038: [FLINK-10791] Provide end-to-end test for Kafka 0.11 connector URL: https://github.com/apache/flink/pull/7038#issuecomment-439790249 @tillrohrmann I agree with your opinion. I have created a new issue (FLINK-10922) to rework specific modules (0.10, 0.11 and 2.0). Will submit PR as soon as possible. cc @pnowojski This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10922) Refactor the placement of the Flink Kafka connector end to end test module
vinoyang created FLINK-10922: Summary: Refactor the placement of the Flink Kafka connector end to end test module Key: FLINK-10922 URL: https://issues.apache.org/jira/browse/FLINK-10922 Project: Flink Issue Type: Improvement Components: E2E Tests Reporter: vinoyang Assignee: vinoyang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10904) Expose Classloader before Pipeline execution
[ https://issues.apache.org/jira/browse/FLINK-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691287#comment-16691287 ] Luka Jurukovski commented on FLINK-10904: - Had to go through some old messages to find out exactly what I was experiencing. I'm classloading the entire job graph in the application entrypoint code with the setting of parent first classloading. What I saw when printing out the classloader hierarchy was that there were two different instances of the org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ParentFirstClassLoader's both inheriting from the same AppClassLoader, which is why the problem was resolved by putting the affected class' jars on the /lib directory. > Expose Classloader before Pipeline execution > > > Key: FLINK-10904 > URL: https://issues.apache.org/jira/browse/FLINK-10904 > Project: Flink > Issue Type: Wish >Reporter: Luka Jurukovski >Priority: Minor > > Not sure if this is something that I just have to deal with. However it would > be nice if the classloader can be accessed before the pipeline starts > executing. The case for this is that I am loading classes that contain Flink > operators. I am running into classdef not found issues because the > classloader used by Flink is different then the application program that is > being run. Currently what I have been doing as a work around is adding the > libs that cause these issues in the /lib directory of the Flink cluster and > marking it as provided in the application jar that is uploaded to the Flink > cluster. The issues with this are two fold, first it makes deployment more > complex, as well as there are cases where Classloading causes exceptions to > arise in unusual circumstances. Ie RabbitMQ connector caused issues only when > it was auto-recovering the connection, but not during normal ingest. I can > elaborate further if needed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10918) incremental Keyed state with RocksDB throws cannot create directory error in windows
[
https://issues.apache.org/jira/browse/FLINK-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691264#comment-16691264
]
ASF GitHub Bot commented on FLINK-10918:
imamitsehgal closed pull request #7134: FLINK-10918 fix for checkpoint dir
creation error on window.
URL: https://github.com/apache/flink/pull/7134
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
index 17ba985f883..29ce3814c6d 100644
---
a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
+++
b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
@@ -575,7 +575,7 @@ private RocksDB openDB(
try {
dbRef = RocksDB.open(
Preconditions.checkNotNull(dbOptions),
- Preconditions.checkNotNull(path),
+ Preconditions.checkNotNull(new
Path(path).toString()),
columnFamilyDescriptors,
stateColumnFamilyHandles);
} catch (RocksDBException e) {
@@ -2546,7 +2546,8 @@ void takeSnapshot() throws Exception {
// create hard links of living files in the snapshot
path
try (Checkpoint checkpoint =
Checkpoint.create(stateBackend.db)) {
-
checkpoint.createCheckpoint(localBackupDirectory.getDirectory().getPath());
+ LOG.info("Checkpoint path is {}.",
localBackupDirectory.getDirectory().toString());
+
checkpoint.createCheckpoint(localBackupDirectory.getDirectory().toString());
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> incremental Keyed state with RocksDB throws cannot create directory error in
> windows
>
>
> Key: FLINK-10918
> URL: https://issues.apache.org/jira/browse/FLINK-10918
> Project: Flink
> Issue Type: Bug
> Components: Queryable State, State Backends, Checkpointing
>Affects Versions: 1.6.2
> Environment: windows
> {code:java}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val rocksdb = new RocksDBStateBackend("file:\\C:\\rocksdb\\checkpoint",true)
> rocksdb.setDbStoragePath("file:\\C:\\rocksdb\\storage")
> env.setStateBackend(rocksdb)
> env.enableCheckpointing(10)
> {code}
>
>Reporter: Amit
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
>
>
> Facing error while enabling keyed state with RocksDBBackend with
> checkpointing to a local windows directory
>
> {code:java}
> Caused by: org.rocksdb.RocksDBException: Failed to create dir:
> /c:/tmp/data/job_dbe01128760d4d5cb90809cd94c2a936_op_StreamMap_b5c8d46f3e7b141acf271f12622e752b__3_8__uuid_45c1f62b-a198-44f5-add5-7683079b03f8/chk-1.tmp:
> Invalid argument
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.takeSnapshot(RocksDBKeyedStateBackend.java:2549)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$IncrementalSnapshotStrategy.performSnapshot(RocksDBKeyedStateBackend.java:2008)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:498)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> ... 13 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] imamitsehgal closed pull request #7134: FLINK-10918 fix for checkpoint dir creation error on window.
imamitsehgal closed pull request #7134: FLINK-10918 fix for checkpoint dir
creation error on window.
URL: https://github.com/apache/flink/pull/7134
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
index 17ba985f883..29ce3814c6d 100644
---
a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
+++
b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
@@ -575,7 +575,7 @@ private RocksDB openDB(
try {
dbRef = RocksDB.open(
Preconditions.checkNotNull(dbOptions),
- Preconditions.checkNotNull(path),
+ Preconditions.checkNotNull(new
Path(path).toString()),
columnFamilyDescriptors,
stateColumnFamilyHandles);
} catch (RocksDBException e) {
@@ -2546,7 +2546,8 @@ void takeSnapshot() throws Exception {
// create hard links of living files in the snapshot
path
try (Checkpoint checkpoint =
Checkpoint.create(stateBackend.db)) {
-
checkpoint.createCheckpoint(localBackupDirectory.getDirectory().getPath());
+ LOG.info("Checkpoint path is {}.",
localBackupDirectory.getDirectory().toString());
+
checkpoint.createCheckpoint(localBackupDirectory.getDirectory().toString());
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10918) incremental Keyed state with RocksDB throws cannot create directory error in windows
[
https://issues.apache.org/jira/browse/FLINK-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691243#comment-16691243
]
ASF GitHub Bot commented on FLINK-10918:
imamitsehgal commented on issue #7134: FLINK-10918 fix for checkpoint dir
creation error on window.
URL: https://github.com/apache/flink/pull/7134#issuecomment-439767469
Why are builds failing while the change is not at all related?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> incremental Keyed state with RocksDB throws cannot create directory error in
> windows
>
>
> Key: FLINK-10918
> URL: https://issues.apache.org/jira/browse/FLINK-10918
> Project: Flink
> Issue Type: Bug
> Components: Queryable State, State Backends, Checkpointing
>Affects Versions: 1.6.2
> Environment: windows
> {code:java}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val rocksdb = new RocksDBStateBackend("file:\\C:\\rocksdb\\checkpoint",true)
> rocksdb.setDbStoragePath("file:\\C:\\rocksdb\\storage")
> env.setStateBackend(rocksdb)
> env.enableCheckpointing(10)
> {code}
>
>Reporter: Amit
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
>
>
> Facing error while enabling keyed state with RocksDBBackend with
> checkpointing to a local windows directory
>
> {code:java}
> Caused by: org.rocksdb.RocksDBException: Failed to create dir:
> /c:/tmp/data/job_dbe01128760d4d5cb90809cd94c2a936_op_StreamMap_b5c8d46f3e7b141acf271f12622e752b__3_8__uuid_45c1f62b-a198-44f5-add5-7683079b03f8/chk-1.tmp:
> Invalid argument
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.takeSnapshot(RocksDBKeyedStateBackend.java:2549)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$IncrementalSnapshotStrategy.performSnapshot(RocksDBKeyedStateBackend.java:2008)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:498)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> ... 13 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] imamitsehgal commented on issue #7134: FLINK-10918 fix for checkpoint dir creation error on window.
imamitsehgal commented on issue #7134: FLINK-10918 fix for checkpoint dir creation error on window. URL: https://github.com/apache/flink/pull/7134#issuecomment-439767469 Why are builds failing while the change is not at all related? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ambition119 commented on issue #7120: fix up RowSerializer.copy occur ClassCastException
ambition119 commented on issue #7120: fix up RowSerializer.copy occur ClassCastException URL: https://github.com/apache/flink/pull/7120#issuecomment-439759507 > Please open a JIRA before opening a PR, and include a description as to when this issue can occur. Additionally we require a test to reproduce the issue and to ensure this issue isn't re-introduced later on. https://issues.apache.org/jira/browse/FLINK-10299 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691205#comment-16691205
]
ASF GitHub Bot commented on FLINK-9555:
---
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234481358
##
File path: flink-dist/pom.xml
##
@@ -319,7 +319,6 @@ under the License.
org.apache.flink
flink-table_${scala.binary.version}
${project.version}
- provided
Review comment:
I suggest that revert this change, because `flink-dist` is the whole release
strategy of flink. At present, we package the `flink-table` module to the `opt`
directory. The minimum change I suggest is to modify the scala-shell CLASSPATH
and include `opt/opt/flink-tableXXX.jar` in the CLASSPATH， to support tables in
`scala-shell`, What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[
https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691206#comment-16691206
]
ASF GitHub Bot commented on FLINK-9555:
---
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234478589
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -96,6 +97,12 @@ class FlinkILoop(
(scalaBenv,scalaSenv)
}
+ val (scalaBTEnv: BatchTableEnvironment, scalaSTEnv: StreamTableEnvironment)
= {
Review comment:
I suggest that merge two local environment init logic as follows:
```
// local environment
val (
scalaBenv: ExecutionEnvironment,
scalaSenv: StreamExecutionEnvironment,
scalaBTabEnv: BatchTableEnvironment,
scalaSTabEnv: StreamTableEnvironment) = {
val scalaBenv = new ExecutionEnvironment(remoteBenv)
val scalaSenv = new StreamExecutionEnvironment(remoteSenv)
val scalaBTabEnv = TableEnvironment.getTableEnvironment(scalaBenv)
val scalaSTabEnv = TableEnvironment.getTableEnvironment(scalaSenv)
(scalaBenv, scalaSenv, scalaBTabEnv, scalaSTabEnv)
}```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support table api in scala shell
>
>
> Key: FLINK-9555
> URL: https://issues.apache.org/jira/browse/FLINK-9555
> Project: Flink
> Issue Type: New Feature
> Components: Scala Shell
>Affects Versions: 1.5.0
>Reporter: Jeff Zhang
>Assignee: shuiqiangchen
>Priority: Major
> Labels: pull-request-available
>
> It would be nice to have table api available in scala shell so that user can
> experience table api in interactive way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9555) Support table api in scala shell
[ https://issues.apache.org/jira/browse/FLINK-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691207#comment-16691207 ] ASF GitHub Bot commented on FLINK-9555: --- sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r234478740 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +146,9 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" +"org.apache.flink.streaming.api.windowing.time._", Review comment: Why we should import `windowing.time`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support table api in scala shell > > > Key: FLINK-9555 > URL: https://issues.apache.org/jira/browse/FLINK-9555 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Affects Versions: 1.5.0 >Reporter: Jeff Zhang >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > > It would be nice to have table api available in scala shell so that user can > experience table api in interactive way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234481358
##
File path: flink-dist/pom.xml
##
@@ -319,7 +319,6 @@ under the License.
org.apache.flink
flink-table_${scala.binary.version}
${project.version}
- provided
Review comment:
I suggest that revert this change, because `flink-dist` is the whole release
strategy of flink. At present, we package the `flink-table` module to the `opt`
directory. The minimum change I suggest is to modify the scala-shell CLASSPATH
and include `opt/opt/flink-tableXXX.jar` in the CLASSPATH， to support tables in
`scala-shell`, What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell URL: https://github.com/apache/flink/pull/7121#discussion_r234478740 ## File path: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala ## @@ -139,7 +146,9 @@ class FlinkILoop( "org.apache.flink.api.scala._", "org.apache.flink.api.scala.utils._", "org.apache.flink.streaming.api.scala._", -"org.apache.flink.streaming.api.windowing.time._" +"org.apache.flink.streaming.api.windowing.time._", Review comment: Why we should import `windowing.time`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support table api in scala shell
sunjincheng121 commented on a change in pull request #7121: [FLINK-9555]Support
table api in scala shell
URL: https://github.com/apache/flink/pull/7121#discussion_r234478589
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -96,6 +97,12 @@ class FlinkILoop(
(scalaBenv,scalaSenv)
}
+ val (scalaBTEnv: BatchTableEnvironment, scalaSTEnv: StreamTableEnvironment)
= {
Review comment:
I suggest that merge two local environment init logic as follows:
```
// local environment
val (
scalaBenv: ExecutionEnvironment,
scalaSenv: StreamExecutionEnvironment,
scalaBTabEnv: BatchTableEnvironment,
scalaSTabEnv: StreamTableEnvironment) = {
val scalaBenv = new ExecutionEnvironment(remoteBenv)
val scalaSenv = new StreamExecutionEnvironment(remoteSenv)
val scalaBTabEnv = TableEnvironment.getTableEnvironment(scalaBenv)
val scalaSTabEnv = TableEnvironment.getTableEnvironment(scalaSenv)
(scalaBenv, scalaSenv, scalaBTabEnv, scalaSTabEnv)
}```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-10299) RowSerializer.copy data value cast exception and use checkpoint function Lead to Could not restart this job
[
https://issues.apache.org/jira/browse/FLINK-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ambition updated FLINK-10299:
-
Attachment: image-2018-11-19-10-40-14-577.png
> RowSerializer.copy data value cast exception and use checkpoint function Lead
> to Could not restart this job
> ---
>
> Key: FLINK-10299
> URL: https://issues.apache.org/jira/browse/FLINK-10299
> Project: Flink
> Issue Type: Bug
> Components: Core, Table API SQL
>Affects Versions: 1.6.0
>Reporter: ambition
>Priority: Minor
> Attachments: image-2018-09-07-17-47-04-343.png,
> image-2018-11-19-10-40-14-577.png
>
>
> Flink sql deal with User behavior data collection, such as：
> {code:java}
> {
> "event_id": "session_start",
> "timestamp": "-", // error data,
> "viewport_height": "667",
> "viewport_width": "-" //error data
> }
> {code}
> Causing exception info :
> {code:java}
> 2018-09-07 10:47:01,834 [flink-akka.actor.default-dispatcher-2] INFO
> executiongraph.ExecutionGraph (ExecutionGraph.java:tryRestartOrFail(1511)) -
> Could not restart the job Flink Streaming Job
> (6f0248219c631158f6e38f2dca0beb91) because the restart strategy prevented it.
> java.lang.ClassCastException: java.lang.String cannot be cast to
> java.sql.Timestamp
> at
> org.apache.flink.api.common.typeutils.base.SqlTimestampSerializer.copy(SqlTimestampSerializer.java:27)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:95)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:577)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
> at
> org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:89)
> at
> org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:154)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:738)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-07 10:47:01,834 [flink-akka.actor.default-dispatcher-2] INFO
> checkpoint.CheckpointCoordinator (CheckpointCoordinator.java:shutdown(320)) -
> Stopping checkpoint coordinator for job 6f0248219c631158f6e38f2dca0beb91.
> 2018-09-07 10:47:01,834 [flink-akka.actor.default-dispatcher-2] INFO
> checkpoint.StandaloneCompletedCheckpointStore
> (StandaloneCompletedCheckpointStore.java:shutdown(102)) - Shutting down
> {code}
> Use Flink checkpoint function and Uncatch exception lead to Could not
> restart this job, so just error data happen exception set null, like under
> image.hope flink commiter provide better solution。
> !image-2018-09-07-17-47-04-343.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10875) Add `toTableWithTimestamp` method in `DataStreamConversions`
[
https://issues.apache.org/jira/browse/FLINK-10875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691161#comment-16691161
]
sunjincheng commented on FLINK-10875:
-
Thanks for the feedback [~fhueske] ！
It is a good point to use both `proctime` and `rowtime` on a table. IMO. A way
to solve this problem is add new methods, but too many methods are not the best
solution. It is reasonable to keep the status quo before we come up with a
better way. (I will think about this question again.)
> Add `toTableWithTimestamp` method in `DataStreamConversions`
>
>
> Key: FLINK-10875
> URL: https://issues.apache.org/jira/browse/FLINK-10875
> Project: Flink
> Issue Type: Improvement
> Components: Table API SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Minor
> Fix For: 1.7.1
>
>
> Currently we convert a `DataStream` to a `Table` by
> `DataStreamConversions#toTable`, e.g.:
> {code:java}
> // Without TimeAttribute
> ...
> val stream = env.fromCollection(...)
> val tab = stream.toTable(tEnv, 'a, 'b, 'c)
> val result = tab.select('a, 'b)
>
> // With TimeAttribute
> ...
> val stream = env.fromCollection(...).assignTimestampsAndWatermarks(...)
> val tab = stream.toTable(tEnv, 'a, 'b, 'c, 'ts.rowtime)
> val result = tab.window(Session withGap 5.milli on 'ts as 'w)
> ...{code}
> I think the fieldNames parameter in the `toTable` method is reasonable in the
> conversion without the time attribute, because the fieldNames will actually
> correspond to the fields of the physical table, but when applied to the
> conversion with the time attribute, the time attribute column is silently
> added to the table. This feeling is very Magical, so I recommend adding a
> method that allows the user to display the time attribute added to the
> physical table: `toTableWithTimestamp`, which is automatically named to the
> time attribute column named by user input and TimeCharacteristic, eg:
> {code:java}
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
> ...
> val table = stream.toTableWithTimestamp(tEnv, 'count, 'size, 'name, 'ts)
> .window(Tumble over 2.rows on 'ts as 'w)
> ...
> {code}
> In the example above the flink will mark `ts` ad a `RowtimeAttribute`.
> What do you think ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10921) Prioritize shard consumers in Kinesis Consumer by event time
Thomas Weise created FLINK-10921: Summary: Prioritize shard consumers in Kinesis Consumer by event time Key: FLINK-10921 URL: https://issues.apache.org/jira/browse/FLINK-10921 Project: Flink Issue Type: Sub-task Components: Kinesis Connector Reporter: Thomas Weise Assignee: Thomas Weise Shard consumer threads currently emit records directly. In order to align shards by event time, decouple shard consumer threads and emitter with a queue, as described in [1]. [1] https://lists.apache.org/thread.html/ac41718246ad8f6098efaf7dbf5f7182d60abdc473e8bf3c96ef5968@%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10920) Remove Kafka examples from binary distribution
[ https://issues.apache.org/jira/browse/FLINK-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691000#comment-16691000 ] ASF GitHub Bot commented on FLINK-10920: tillrohrmann opened a new pull request #7135: [FLINK-10920] Remove Kafka examples from flink-dist URL: https://github.com/apache/flink/pull/7135 ## What is the purpose of the change In order to decrease the size of Flink's binary distribution, this commit removes the Kafka examples from it. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove Kafka examples from binary distribution > -- > > Key: FLINK-10920 > URL: https://issues.apache.org/jira/browse/FLINK-10920 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.8.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > > I think we should remove the Kafka examples fro Flink's binary distribution > because they each have a size of at least 50 MB and there are three of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10920) Remove Kafka examples from binary distribution
[ https://issues.apache.org/jira/browse/FLINK-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-10920: --- Labels: pull-request-available (was: ) > Remove Kafka examples from binary distribution > -- > > Key: FLINK-10920 > URL: https://issues.apache.org/jira/browse/FLINK-10920 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.8.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > > I think we should remove the Kafka examples fro Flink's binary distribution > because they each have a size of at least 50 MB and there are three of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] tillrohrmann opened a new pull request #7136: [BP-1.7][FLINK-10920] Remove Kafka examples from flink-dist
tillrohrmann opened a new pull request #7136: [BP-1.7][FLINK-10920] Remove Kafka examples from flink-dist URL: https://github.com/apache/flink/pull/7136 Backport of #7135 to `release-1.7`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tillrohrmann opened a new pull request #7135: [FLINK-10920] Remove Kafka examples from flink-dist
tillrohrmann opened a new pull request #7135: [FLINK-10920] Remove Kafka examples from flink-dist URL: https://github.com/apache/flink/pull/7135 ## What is the purpose of the change In order to decrease the size of Flink's binary distribution, this commit removes the Kafka examples from it. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10920) Remove Kafka examples from binary distribution
Till Rohrmann created FLINK-10920: - Summary: Remove Kafka examples from binary distribution Key: FLINK-10920 URL: https://issues.apache.org/jira/browse/FLINK-10920 Project: Flink Issue Type: Sub-task Components: Build System Affects Versions: 1.8.0 Reporter: Till Rohrmann I think we should remove the Kafka examples fro Flink's binary distribution because they each have a size of at least 50 MB and there are three of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-10920) Remove Kafka examples from binary distribution
[ https://issues.apache.org/jira/browse/FLINK-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann reassigned FLINK-10920: - Assignee: Till Rohrmann > Remove Kafka examples from binary distribution > -- > > Key: FLINK-10920 > URL: https://issues.apache.org/jira/browse/FLINK-10920 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.8.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > > I think we should remove the Kafka examples fro Flink's binary distribution > because they each have a size of at least 50 MB and there are three of them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10919) Flink binary distribution too large
Till Rohrmann created FLINK-10919:
-
Summary: Flink binary distribution too large
Key: FLINK-10919
URL: https://issues.apache.org/jira/browse/FLINK-10919
Project: Flink
Issue Type: Improvement
Components: Build System
Affects Versions: 1.7.0
Reporter: Till Rohrmann
Fix For: 1.8.0
Flink's binary distribution is almost 500 MB large:
{code}
128K./bin
16K./examples/python/streaming
28K./examples/python/batch
44K./examples/python
189M./examples/streaming
240K./examples/gelly
136K./examples/batch
190M./examples
131M./lib
174M./opt
0B./log
56K./conf
494M.
{code}
I think this is far too large and we should try to reduce the size. For
example, the examples directory contains 3 different Kafka example jobs, each
at least 50 MB of size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10845) Support DISTINCT aggregates for batch
[
https://issues.apache.org/jira/browse/FLINK-10845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690918#comment-16690918
]
ASF GitHub Bot commented on FLINK-10845:
hequn8128 commented on issue #7079: [FLINK-10845][table] Support multiple
different DISTINCT aggregates for batch
URL: https://github.com/apache/flink/pull/7079#issuecomment-439697499
@xueyumusic thanks for the update. Looks good from my side.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support DISTINCT aggregates for batch
> -
>
> Key: FLINK-10845
> URL: https://issues.apache.org/jira/browse/FLINK-10845
> Project: Flink
> Issue Type: New Feature
> Components: Table API SQL
>Reporter: Timo Walther
>Assignee: xueyu
>Priority: Major
> Labels: pull-request-available
>
> Currently, we support distinct aggregates for streaming. However, executing
> the same query on batch like the following test:
> {code}
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tEnv = TableEnvironment.getTableEnvironment(env)
> val sqlQuery =
> "SELECT b, " +
> " SUM(DISTINCT (a / 3)), " +
> " COUNT(DISTINCT SUBSTRING(c FROM 1 FOR 2))," +
> " COUNT(DISTINCT c) " +
> "FROM MyTable " +
> "GROUP BY b"
> val data = new mutable.MutableList[(Int, Long, String)]
> data.+=((1, 1L, "Hi"))
> data.+=((2, 2L, "Hello"))
> data.+=((3, 2L, "Hello world"))
> data.+=((4, 3L, "Hello world, how are you?"))
> data.+=((5, 3L, "I am fine."))
> data.+=((6, 3L, "Luke Skywalker"))
> data.+=((7, 4L, "Comment#1"))
> data.+=((8, 4L, "Comment#2"))
> data.+=((9, 4L, "Comment#3"))
> data.+=((10, 4L, "Comment#4"))
> data.+=((11, 5L, "Comment#5"))
> data.+=((12, 5L, "Comment#6"))
> data.+=((13, 5L, "Comment#7"))
> data.+=((14, 5L, "Comment#8"))
> data.+=((15, 5L, "Comment#9"))
> data.+=((16, 6L, "Comment#10"))
> data.+=((17, 6L, "Comment#11"))
> data.+=((18, 6L, "Comment#12"))
> data.+=((19, 6L, "Comment#13"))
> data.+=((20, 6L, "Comment#14"))
> data.+=((21, 6L, "Comment#15"))
> val t = env.fromCollection(data).toTable(tEnv).as('a, 'b, 'c)
> tEnv.registerTable("MyTable", t)
> tEnv.sqlQuery(sqlQuery).toDataSet[Row].print()
> {code}
> Fails with:
> {code}
> org.apache.flink.table.codegen.CodeGenException: Unsupported call: IS NOT
> DISTINCT FROM
> If you think this function should be supported, you can create an issue and
> start a discussion for it.
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$visitCall$3.apply(CodeGenerator.scala:1027)
> at
> org.apache.flink.table.codegen.CodeGenerator$$anonfun$visitCall$3.apply(CodeGenerator.scala:1027)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:1027)
> at
> org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:66)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:107)
> at
> org.apache.flink.table.codegen.CodeGenerator.generateExpression(CodeGenerator.scala:247)
> at
> org.apache.flink.table.plan.nodes.dataset.DataSetJoin.addInnerJoin(DataSetJoin.scala:221)
> at
> org.apache.flink.table.plan.nodes.dataset.DataSetJoin.translateToPlan(DataSetJoin.scala:170)
> at
> org.apache.flink.table.plan.nodes.dataset.DataSetCalc.translateToPlan(DataSetCalc.scala:91)
> at
> org.apache.flink.table.plan.nodes.dataset.DataSetJoin.translateToPlan(DataSetJoin.scala:165)
> at
> org.apache.flink.table.plan.nodes.dataset.DataSetCalc.translateToPlan(DataSetCalc.scala:91)
> at
> org.apache.flink.table.api.BatchTableEnvironment.translate(BatchTableEnvironment.scala:498)
> at
> org.apache.flink.table.api.BatchTableEnvironment.translate(BatchTableEnvironment.scala:476)
> at
> org.apache.flink.table.api.scala.BatchTableEnvironment.toDataSet(BatchTableEnvironment.scala:141)
> at
> org.apache.flink.table.api.scala.TableConversions.toDataSet(TableConversions.scala:50)
> at
> org.apache.flink.table.runtime.stream.sql.SqlITCase.testDistinctGroupBy(SqlITCase.scala:2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hequn8128 commented on issue #7079: [FLINK-10845][table] Support multiple different DISTINCT aggregates for batch
hequn8128 commented on issue #7079: [FLINK-10845][table] Support multiple different DISTINCT aggregates for batch URL: https://github.com/apache/flink/pull/7079#issuecomment-439697499 @xueyumusic thanks for the update. Looks good from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-6756) Provide RichAsyncFunction to Scala API suite
[ https://issues.apache.org/jira/browse/FLINK-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690910#comment-16690910 ] Dawid Wysakowicz commented on FLINK-6756: - Hi [~dangdangdang] thanks for your interest in taking this over. Feel free to do it. I think we should follow the approach described in this issue. > Provide RichAsyncFunction to Scala API suite > > > Key: FLINK-6756 > URL: https://issues.apache.org/jira/browse/FLINK-6756 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Andrea Spina >Assignee: Andrea Spina >Priority: Major > > I can't find any tracking info about the chance to have RichAsyncFunction in > the Scala API suite. I think it'd be nice to have this function in order to > access open/close methods and the RuntimeContext. > I was able to retrieve > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/There-is-no-Open-and-Close-method-in-Async-I-O-API-of-Scala-td11591.html#a11593 > only, so my question is if there are some blocking issues avoiding this > feature. [~till.rohrmann] > If it's possible and nobody already have done it, I can assign the issue to > myself in order to implement it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-10625) Add MATCH_RECOGNIZE documentation
[
https://issues.apache.org/jira/browse/FLINK-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz closed FLINK-10625.
Resolution: Fixed
Fix Version/s: (was: 1.8.0)
1.7.0
Fixed via:
master: 184ba534a4b9675a6b10a88480c82f70b181b96e,
feccce4240e57533cdc2146f10ba0b52dd1de653,
71828fc8540c6693110a1a13850be9bd6e15fab3
1.7: d61f3bc899a53fd77875686fe62ef152c68e10dc,
064281456d493aa12e3a451cd9f3b0cc5334e4fb,
ab86d5d11fe454102fd4b3d0fe92b0b1eb195210
> Add MATCH_RECOGNIZE documentation
> -
>
> Key: FLINK-10625
> URL: https://issues.apache.org/jira/browse/FLINK-10625
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Table API SQL
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> The newly added {{MATCH_RECOGNIZE}} functionality needs to be documented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10625) Add MATCH_RECOGNIZE documentation
[
https://issues.apache.org/jira/browse/FLINK-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690908#comment-16690908
]
ASF GitHub Bot commented on FLINK-10625:
dawidwys closed pull request #7070: [FLINK-10625] Documentation for
MATCH_RECOGNIZE clause
URL: https://github.com/apache/flink/pull/7070
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/sql.md b/docs/dev/table/sql.md
index b1bd572d4d2..90e20065726 100644
--- a/docs/dev/table/sql.md
+++ b/docs/dev/table/sql.md
@@ -163,6 +163,7 @@ joinCondition:
tableReference:
tablePrimary
+ [ matchRecognize ]
[ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]
tablePrimary:
@@ -196,6 +197,45 @@ windowSpec:
]
')'
+matchRecognize:
+ MATCH_RECOGNIZE '('
+ [ PARTITION BY expression [, expression ]* ]
+ [ ORDER BY orderItem [, orderItem ]* ]
+ [ MEASURES measureColumn [, measureColumn ]* ]
+ [ ONE ROW PER MATCH ]
+ [ AFTER MATCH
+( SKIP TO NEXT ROW
+| SKIP PAST LAST ROW
+| SKIP TO FIRST variable
+| SKIP TO LAST variable
+| SKIP TO variable )
+ ]
+ PATTERN '(' pattern ')'
+ DEFINE variable AS condition [, variable AS condition ]*
+ ')'
+
+measureColumn:
+ expression AS alias
+
+pattern:
+ patternTerm [ '|' patternTerm ]*
+
+patternTerm:
+ patternFactor [ patternFactor ]*
+
+patternFactor:
+ variable [ patternQuantifier ]
+
+patternQuantifier:
+ '*'
+ | '*?'
+ | '+'
+ | '+?'
+ | '?'
+ | '??'
+ | '{' { [ minRepeat ], [ maxRepeat ] } '}' ['?']
+ | '{' repeat '}'
+
{% endhighlight %}
Flink SQL uses a lexical policy for identifier (table, attribute, function
names) similar to Java:
@@ -756,7 +796,6 @@ Group windows are defined in the `GROUP BY` clause of a SQL
query. Just like que
-
Time Attributes
For SQL queries on streaming tables, the `time_attr` argument of the group
window function must refer to a valid time attribute that specifies the
processing time or event time of rows. See the [documentation of time
attributes](streaming/time_attributes.html) to learn how to define time
attributes.
@@ -902,6 +941,52 @@ val result4 = tableEnv.sqlQuery(
{% top %}
+### Pattern Recognition
+
+
+
+
+
+ Operation
+ Description
+
+
+
+
+
+MATCH_RECOGNIZE
+Streaming
+
+
+Searches for a given pattern in a streaming table according to the
MATCH_RECOGNIZE https://standards.iso.org/ittf/PubliclyAvailableStandards/c065143_ISO_IEC_TR_19075-5_2016.zip;>ISO
standard. This makes it possible to express complex event processing (CEP)
logic in SQL queries.
+For a more detailed description, see the dedicated page for detecting patterns in tables.
+
+{% highlight sql %}
+SELECT T.aid, T.bid, T.cid
+FROM MyTable
+MATCH_RECOGNIZE (
+ PARTITION BY userid
+ ORDER BY proctime
+ MEASURES
+A.id AS aid,
+B.id AS bid,
+C.id AS cid
+ PATTERN (A B C)
+ DEFINE
+A AS name = 'a',
+B AS name = 'b',
+C AS name = 'c'
+) AS T
+{% endhighlight %}
+
+
+
+
+
+
+
+{% top %}
+
Data Types
--
diff --git a/docs/dev/table/streaming/match_recognize.md
b/docs/dev/table/streaming/match_recognize.md
new file mode 100644
index 000..b12cbe5e0d9
--- /dev/null
+++ b/docs/dev/table/streaming/match_recognize.md
@@ -0,0 +1,842 @@
+---
+title: 'Detecting Patterns in Tables'
+nav-parent_id: streaming_tableapi
+nav-title: 'Detecting Patterns'
+nav-pos: 5
+is_beta: true
+---
+
+
+It is a common use case to search for a set of event patterns, especially in
case of data streams. Flink
+comes with a [complex event processing (CEP) library]({{ site.baseurl
}}/dev/libs/cep.html) which allows for pattern detection in event streams.
Furthermore, Flink's
+SQL API provides a relational way of expressing queries with a large set of
built-in functions and rule-based optimizations that can be used out of the box.
+
+In December 2016, the International Organization for Standardization (ISO)
released a new version of the SQL standard which includes _Row Pattern
Recognition in SQL_ ([ISO/IEC TR
19075-5:2016](https://standards.iso.org/ittf/PubliclyAvailableStandards/c065143_ISO_IEC_TR_19075-5_2016.zip)).
It allows Flink to consolidate CEP and SQL API using the `MATCH_RECOGNIZE`
clause for complex event processing in SQL.
+
+A `MATCH_RECOGNIZE` clause enables the following tasks:
+* Logically partition and order the data that is used with the `PARTITION BY`
and `ORDER BY` clauses.
+* Define patterns of rows to seek using the `PATTERN` clause. These patterns
[GitHub] dawidwys closed pull request #7070: [FLINK-10625] Documentation for MATCH_RECOGNIZE clause
dawidwys closed pull request #7070: [FLINK-10625] Documentation for
MATCH_RECOGNIZE clause
URL: https://github.com/apache/flink/pull/7070
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/sql.md b/docs/dev/table/sql.md
index b1bd572d4d2..90e20065726 100644
--- a/docs/dev/table/sql.md
+++ b/docs/dev/table/sql.md
@@ -163,6 +163,7 @@ joinCondition:
tableReference:
tablePrimary
+ [ matchRecognize ]
[ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]
tablePrimary:
@@ -196,6 +197,45 @@ windowSpec:
]
')'
+matchRecognize:
+ MATCH_RECOGNIZE '('
+ [ PARTITION BY expression [, expression ]* ]
+ [ ORDER BY orderItem [, orderItem ]* ]
+ [ MEASURES measureColumn [, measureColumn ]* ]
+ [ ONE ROW PER MATCH ]
+ [ AFTER MATCH
+( SKIP TO NEXT ROW
+| SKIP PAST LAST ROW
+| SKIP TO FIRST variable
+| SKIP TO LAST variable
+| SKIP TO variable )
+ ]
+ PATTERN '(' pattern ')'
+ DEFINE variable AS condition [, variable AS condition ]*
+ ')'
+
+measureColumn:
+ expression AS alias
+
+pattern:
+ patternTerm [ '|' patternTerm ]*
+
+patternTerm:
+ patternFactor [ patternFactor ]*
+
+patternFactor:
+ variable [ patternQuantifier ]
+
+patternQuantifier:
+ '*'
+ | '*?'
+ | '+'
+ | '+?'
+ | '?'
+ | '??'
+ | '{' { [ minRepeat ], [ maxRepeat ] } '}' ['?']
+ | '{' repeat '}'
+
{% endhighlight %}
Flink SQL uses a lexical policy for identifier (table, attribute, function
names) similar to Java:
@@ -756,7 +796,6 @@ Group windows are defined in the `GROUP BY` clause of a SQL
query. Just like que
-
Time Attributes
For SQL queries on streaming tables, the `time_attr` argument of the group
window function must refer to a valid time attribute that specifies the
processing time or event time of rows. See the [documentation of time
attributes](streaming/time_attributes.html) to learn how to define time
attributes.
@@ -902,6 +941,52 @@ val result4 = tableEnv.sqlQuery(
{% top %}
+### Pattern Recognition
+
+
+
+
+
+ Operation
+ Description
+
+
+
+
+
+MATCH_RECOGNIZE
+Streaming
+
+
+Searches for a given pattern in a streaming table according to the
MATCH_RECOGNIZE https://standards.iso.org/ittf/PubliclyAvailableStandards/c065143_ISO_IEC_TR_19075-5_2016.zip;>ISO
standard. This makes it possible to express complex event processing (CEP)
logic in SQL queries.
+For a more detailed description, see the dedicated page for detecting patterns in tables.
+
+{% highlight sql %}
+SELECT T.aid, T.bid, T.cid
+FROM MyTable
+MATCH_RECOGNIZE (
+ PARTITION BY userid
+ ORDER BY proctime
+ MEASURES
+A.id AS aid,
+B.id AS bid,
+C.id AS cid
+ PATTERN (A B C)
+ DEFINE
+A AS name = 'a',
+B AS name = 'b',
+C AS name = 'c'
+) AS T
+{% endhighlight %}
+
+
+
+
+
+
+
+{% top %}
+
Data Types
--
diff --git a/docs/dev/table/streaming/match_recognize.md
b/docs/dev/table/streaming/match_recognize.md
new file mode 100644
index 000..b12cbe5e0d9
--- /dev/null
+++ b/docs/dev/table/streaming/match_recognize.md
@@ -0,0 +1,842 @@
+---
+title: 'Detecting Patterns in Tables'
+nav-parent_id: streaming_tableapi
+nav-title: 'Detecting Patterns'
+nav-pos: 5
+is_beta: true
+---
+
+
+It is a common use case to search for a set of event patterns, especially in
case of data streams. Flink
+comes with a [complex event processing (CEP) library]({{ site.baseurl
}}/dev/libs/cep.html) which allows for pattern detection in event streams.
Furthermore, Flink's
+SQL API provides a relational way of expressing queries with a large set of
built-in functions and rule-based optimizations that can be used out of the box.
+
+In December 2016, the International Organization for Standardization (ISO)
released a new version of the SQL standard which includes _Row Pattern
Recognition in SQL_ ([ISO/IEC TR
19075-5:2016](https://standards.iso.org/ittf/PubliclyAvailableStandards/c065143_ISO_IEC_TR_19075-5_2016.zip)).
It allows Flink to consolidate CEP and SQL API using the `MATCH_RECOGNIZE`
clause for complex event processing in SQL.
+
+A `MATCH_RECOGNIZE` clause enables the following tasks:
+* Logically partition and order the data that is used with the `PARTITION BY`
and `ORDER BY` clauses.
+* Define patterns of rows to seek using the `PATTERN` clause. These patterns
use a syntax similar to that of regular expressions.
+* The logical components of the row pattern variables are specified in the
`DEFINE` clause.
+* Define measures, which are expressions usable in other parts of the SQL
query, in the
[jira] [Updated] (FLINK-10918) incremental Keyed state with RocksDB throws cannot create directory error in windows
[
https://issues.apache.org/jira/browse/FLINK-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10918:
---
Labels: pull-request-available (was: )
> incremental Keyed state with RocksDB throws cannot create directory error in
> windows
>
>
> Key: FLINK-10918
> URL: https://issues.apache.org/jira/browse/FLINK-10918
> Project: Flink
> Issue Type: Bug
> Components: Queryable State, State Backends, Checkpointing
>Affects Versions: 1.6.2
> Environment: windows
> {code:java}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val rocksdb = new RocksDBStateBackend("file:\\C:\\rocksdb\\checkpoint",true)
> rocksdb.setDbStoragePath("file:\\C:\\rocksdb\\storage")
> env.setStateBackend(rocksdb)
> env.enableCheckpointing(10)
> {code}
>
>Reporter: Amit
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
>
>
> Facing error while enabling keyed state with RocksDBBackend with
> checkpointing to a local windows directory
>
> {code:java}
> Caused by: org.rocksdb.RocksDBException: Failed to create dir:
> /c:/tmp/data/job_dbe01128760d4d5cb90809cd94c2a936_op_StreamMap_b5c8d46f3e7b141acf271f12622e752b__3_8__uuid_45c1f62b-a198-44f5-add5-7683079b03f8/chk-1.tmp:
> Invalid argument
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.takeSnapshot(RocksDBKeyedStateBackend.java:2549)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$IncrementalSnapshotStrategy.performSnapshot(RocksDBKeyedStateBackend.java:2008)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:498)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> ... 13 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10918) incremental Keyed state with RocksDB throws cannot create directory error in windows
[
https://issues.apache.org/jira/browse/FLINK-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690907#comment-16690907
]
ASF GitHub Bot commented on FLINK-10918:
imamitsehgal opened a new pull request #7134: FLINK-10918 fix for checkpoint
dir creation error on window.
URL: https://github.com/apache/flink/pull/7134
FLINK-10918 [flink-state-backaends] fix for checkpoint dir creation error on
window. 1 of the 4 failing test on windows are now passing
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> incremental Keyed state with RocksDB throws cannot create directory error in
> windows
>
>
> Key: FLINK-10918
> URL: https://issues.apache.org/jira/browse/FLINK-10918
> Project: Flink
> Issue Type: Bug
> Components: Queryable State, State Backends, Checkpointing
>Affects Versions: 1.6.2
> Environment: windows
> {code:java}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val rocksdb = new RocksDBStateBackend("file:\\C:\\rocksdb\\checkpoint",true)
> rocksdb.setDbStoragePath("file:\\C:\\rocksdb\\storage")
> env.setStateBackend(rocksdb)
> env.enableCheckpointing(10)
> {code}
>
>Reporter: Amit
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
>
>
> Facing error while enabling keyed state with RocksDBBackend with
> checkpointing to a local windows directory
>
> {code:java}
> Caused by: org.rocksdb.RocksDBException: Failed to create dir:
> /c:/tmp/data/job_dbe01128760d4d5cb90809cd94c2a936_op_StreamMap_b5c8d46f3e7b141acf271f12622e752b__3_8__uuid_45c1f62b-a198-44f5-add5-7683079b03f8/chk-1.tmp:
> Invalid argument
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.takeSnapshot(RocksDBKeyedStateBackend.java:2549)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$IncrementalSnapshotStrategy.performSnapshot(RocksDBKeyedStateBackend.java:2008)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:498)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> ... 13 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] imamitsehgal opened a new pull request #7134: FLINK-10918 fix for checkpoint dir creation error on window.
imamitsehgal opened a new pull request #7134: FLINK-10918 fix for checkpoint dir creation error on window. URL: https://github.com/apache/flink/pull/7134 FLINK-10918 [flink-state-backaends] fix for checkpoint dir creation error on window. 1 of the 4 failing test on windows are now passing This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10918) incremental Keyed state with RocksDB throws cannot create directory error in windows
[
https://issues.apache.org/jira/browse/FLINK-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690899#comment-16690899
]
Amit commented on FLINK-10918:
--
Attaching a patch file.
[^0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch]
> incremental Keyed state with RocksDB throws cannot create directory error in
> windows
>
>
> Key: FLINK-10918
> URL: https://issues.apache.org/jira/browse/FLINK-10918
> Project: Flink
> Issue Type: Bug
> Components: Queryable State, State Backends, Checkpointing
>Affects Versions: 1.6.2
> Environment: windows
> {code:java}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val rocksdb = new RocksDBStateBackend("file:\\C:\\rocksdb\\checkpoint",true)
> rocksdb.setDbStoragePath("file:\\C:\\rocksdb\\storage")
> env.setStateBackend(rocksdb)
> env.enableCheckpointing(10)
> {code}
>
>Reporter: Amit
>Priority: Major
> Attachments:
> 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
>
>
> Facing error while enabling keyed state with RocksDBBackend with
> checkpointing to a local windows directory
>
> {code:java}
> Caused by: org.rocksdb.RocksDBException: Failed to create dir:
> /c:/tmp/data/job_dbe01128760d4d5cb90809cd94c2a936_op_StreamMap_b5c8d46f3e7b141acf271f12622e752b__3_8__uuid_45c1f62b-a198-44f5-add5-7683079b03f8/chk-1.tmp:
> Invalid argument
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.takeSnapshot(RocksDBKeyedStateBackend.java:2549)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$IncrementalSnapshotStrategy.performSnapshot(RocksDBKeyedStateBackend.java:2008)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:498)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> ... 13 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10918) incremental Keyed state with RocksDB throws cannot create directory error in windows
[
https://issues.apache.org/jira/browse/FLINK-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated FLINK-10918:
-
Attachment: 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
> incremental Keyed state with RocksDB throws cannot create directory error in
> windows
>
>
> Key: FLINK-10918
> URL: https://issues.apache.org/jira/browse/FLINK-10918
> Project: Flink
> Issue Type: Bug
> Components: Queryable State, State Backends, Checkpointing
>Affects Versions: 1.6.2
> Environment: windows
> {code:java}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val rocksdb = new RocksDBStateBackend("file:\\C:\\rocksdb\\checkpoint",true)
> rocksdb.setDbStoragePath("file:\\C:\\rocksdb\\storage")
> env.setStateBackend(rocksdb)
> env.enableCheckpointing(10)
> {code}
>
>Reporter: Amit
>Priority: Major
> Attachments:
> 0001-FLINK-10918-Fix-for-checkpoint-creation-on-windows-1.patch
>
>
> Facing error while enabling keyed state with RocksDBBackend with
> checkpointing to a local windows directory
>
> {code:java}
> Caused by: org.rocksdb.RocksDBException: Failed to create dir:
> /c:/tmp/data/job_dbe01128760d4d5cb90809cd94c2a936_op_StreamMap_b5c8d46f3e7b141acf271f12622e752b__3_8__uuid_45c1f62b-a198-44f5-add5-7683079b03f8/chk-1.tmp:
> Invalid argument
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.takeSnapshot(RocksDBKeyedStateBackend.java:2549)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$IncrementalSnapshotStrategy.performSnapshot(RocksDBKeyedStateBackend.java:2008)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:498)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> ... 13 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9720) Introduce ResourceTag class for tag support in scheduling
[ https://issues.apache.org/jira/browse/FLINK-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9720: - Fix Version/s: (was: 1.7.0) 1.8.0 > Introduce ResourceTag class for tag support in scheduling > - > > Key: FLINK-9720 > URL: https://issues.apache.org/jira/browse/FLINK-9720 > Project: Flink > Issue Type: New Feature > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10503) Periodically check for new resources
[ https://issues.apache.org/jira/browse/FLINK-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10503: -- Fix Version/s: (was: 1.7.0) > Periodically check for new resources > > > Key: FLINK-10503 > URL: https://issues.apache.org/jira/browse/FLINK-10503 > Project: Flink > Issue Type: Sub-task > Components: Distributed Coordination >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Shimin Yang >Priority: Major > Fix For: 1.8.0 > > > In order to decide when to start scheduling or to rescale, we need to > periodically check for new resources (slots). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8418) Kafka08ITCase.testStartFromLatestOffsets() times out on Travis
[ https://issues.apache.org/jira/browse/FLINK-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-8418: - Fix Version/s: 1.8.0 > Kafka08ITCase.testStartFromLatestOffsets() times out on Travis > -- > > Key: FLINK-8418 > URL: https://issues.apache.org/jira/browse/FLINK-8418 > Project: Flink > Issue Type: Bug > Components: Kafka Connector, Tests >Affects Versions: 1.4.0, 1.5.0 >Reporter: Tzu-Li (Gordon) Tai >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > Labels: test-stability > Fix For: 1.5.6, 1.6.3, 1.7.0, 1.8.0 > > > Instance: https://travis-ci.org/kl0u/flink/builds/327733085 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10435) Client sporadically hangs after Ctrl + C
[
https://issues.apache.org/jira/browse/FLINK-10435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10435:
--
Fix Version/s: 1.7.1
> Client sporadically hangs after Ctrl + C
>
>
> Key: FLINK-10435
> URL: https://issues.apache.org/jira/browse/FLINK-10435
> Project: Flink
> Issue Type: Bug
> Components: Client, YARN
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Gary Yao
>Priority: Major
> Fix For: 1.5.6, 1.6.3, 1.7.0, 1.8.0, 1.7.1
>
>
> When submitting a YARN job cluster in attached mode, the client hangs
> indefinitely if Ctrl + C is pressed at the right time. One can recover from
> this by sending SIGKILL.
> *Command to submit job*
> {code}
> HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster
> examples/streaming/WordCount.jar
> {code}
>
> *Output/Stacktrace*
> {code}
> [hadoop@ip-172-31-45-22 flink-1.5.4]$ HADOOP_CLASSPATH=`hadoop classpath`
> bin/flink run -m yarn-cluster examples/streaming/WordCount.jar
> Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/hadoop/flink-1.5.4/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2018-09-26 12:01:04,241 INFO org.apache.hadoop.yarn.client.RMProxy
> - Connecting to ResourceManager at
> ip-172-31-45-22.eu-central-1.compute.internal/172.31.45.22:8032
> 2018-09-26 12:01:04,386 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-09-26 12:01:04,386 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2018-09-26 12:01:04,402 WARN
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Neither the
> HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink
> YARN Client needs one of these to be set to properly load the Hadoop
> configuration for accessing YARN.
> 2018-09-26 12:01:04,598 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster
> specification: ClusterSpecification{masterMemoryMB=1024,
> taskManagerMemoryMB=1024, numberTaskManagers=1, slotsPerTaskManager=1}
> 2018-09-26 12:01:04,972 WARN
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - The
> configuration directory ('/home/hadoop/flink-1.5.4/conf') contains both LOG4J
> and Logback configuration files. Please delete or rename one of them.
> 2018-09-26 12:01:07,857 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting
> application master application_1537944258063_0017
> 2018-09-26 12:01:07,913 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1537944258063_0017
> 2018-09-26 12:01:07,913 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for
> the cluster to be allocated
> 2018-09-26 12:01:07,916 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying
> cluster, current state ACCEPTED
> ^C2018-09-26 12:01:08,851 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cancelling
> deployment from Deployment Failure Hook
> 2018-09-26 12:01:08,854 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Killing YARN
> application
>
> The program finished with the following exception:
> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't
> deploy Yarn session cluster
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:410)
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:258)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
[jira] [Updated] (FLINK-10347) Kafka09SecuredRunITCase failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10347:
--
Fix Version/s: 1.8.0
> Kafka09SecuredRunITCase failed on Travis
>
>
> Key: FLINK-10347
> URL: https://issues.apache.org/jira/browse/FLINK-10347
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector, Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.7.0, 1.8.0
>
>
> The {{Kafka09SecuredRunITCase}} failed on Travis:
> https://api.travis-ci.org/v3/job/428662252/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10842) Waiting loops are broken in e2e/common.sh
[
https://issues.apache.org/jira/browse/FLINK-10842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10842:
--
Fix Version/s: 1.8.0
> Waiting loops are broken in e2e/common.sh
> -
>
> Key: FLINK-10842
> URL: https://issues.apache.org/jira/browse/FLINK-10842
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.8.0
>
>
> There are 3 loops in flink-end-to-end-tests/test-scripts/common.sh where the
> script waits for some event to happen (for i in \{1..10}; do):
> - wait_dispatcher_running
> - start_and_wait_for_tm
> - wait_job_running
> All loops have 10 iterations and the loop breaks if the awaited event
> happens. If timeout occurs then the script does not fail and the function
> just continues after 10 iterations ignoring that the awaited event did not
> happen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10252) Handle oversized metric messges
[
https://issues.apache.org/jira/browse/FLINK-10252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10252:
--
Fix Version/s: 1.7.1
> Handle oversized metric messges
> ---
>
> Key: FLINK-10252
> URL: https://issues.apache.org/jira/browse/FLINK-10252
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0, 1.8.0, 1.7.1
>
>
> Since the {{MetricQueryService}} is implemented as an Akka actor, it can only
> send messages of a smaller size then the current {{akka.framesize}}. We
> should check similarly to FLINK-10251 whether the payload exceeds the maximum
> framesize and fail fast if it is true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9812) SpanningRecordSerializationTest fails on travis
[
https://issues.apache.org/jira/browse/FLINK-9812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9812:
-
Fix Version/s: (was: 1.7.0)
> SpanningRecordSerializationTest fails on travis
> ---
>
> Key: FLINK-9812
> URL: https://issues.apache.org/jira/browse/FLINK-9812
> Project: Flink
> Issue Type: Bug
> Components: Network, Tests, Type Serialization System
>Affects Versions: 1.6.0
>Reporter: Chesnay Schepler
>Assignee: Nico Kruber
>Priority: Critical
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> https://travis-ci.org/zentol/flink/jobs/402744191
> {code}
> testHandleMixedLargeRecords(org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest)
> Time elapsed: 6.113 sec <<< ERROR!
> java.nio.channels.ClosedChannelException: null
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:199)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer$SpanningWrapper.addNextChunkFromMemorySegment(SpillingAdaptiveSpanningRecordDeserializer.java:529)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer$SpanningWrapper.access$200(SpillingAdaptiveSpanningRecordDeserializer.java:431)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.setNextBuffer(SpillingAdaptiveSpanningRecordDeserializer.java:76)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest.testSerializationRoundTrip(SpanningRecordSerializationTest.java:149)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest.testSerializationRoundTrip(SpanningRecordSerializationTest.java:115)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializationTest.testHandleMixedLargeRecords(SpanningRecordSerializationTest.java:104)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9002) Add operators with input type that goes through Avro serialization (with schema/generic)
[ https://issues.apache.org/jira/browse/FLINK-9002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9002: - Fix Version/s: (was: 1.7.0) > Add operators with input type that goes through Avro serialization (with > schema/generic) > - > > Key: FLINK-9002 > URL: https://issues.apache.org/jira/browse/FLINK-9002 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.5.0 >Reporter: Stefan Richter >Assignee: Tzu-Li (Gordon) Tai >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-7829) Remove (or at least deprecate) DataStream.writeToFile/Csv
[
https://issues.apache.org/jira/browse/FLINK-7829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-7829:
-
Fix Version/s: 1.8.0
> Remove (or at least deprecate) DataStream.writeToFile/Csv
> -
>
> Key: FLINK-7829
> URL: https://issues.apache.org/jira/browse/FLINK-7829
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Aljoscha Krettek
>Priority: Major
> Fix For: 1.7.0, 1.8.0
>
>
> These methods are seductive for users but they should never actually use them
> in a production streaming job. For those cases the {{StreamingFileSink}}
> should be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10504) Decide actual parallelism based on available resources
[
https://issues.apache.org/jira/browse/FLINK-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10504:
--
Fix Version/s: (was: 1.7.0)
> Decide actual parallelism based on available resources
> --
>
> Key: FLINK-10504
> URL: https://issues.apache.org/jira/browse/FLINK-10504
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: blues Zheng
>Priority: Major
> Fix For: 1.8.0
>
>
> Check if a {{JobGraph}} can be scheduled with the available set of resources
> (slots). If the minimum parallelism is fulfilled, then distribute the
> available set of slots across all available slot sharing groups in order to
> decide on the actual runtime parallelism. In the absence of minimum, target
> and maximum parallelism, assume minimum = target = maximum = parallelism
> defined in the {{JobGraph}}.
> Ideally, we make the slot assignment strategy pluggable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9054) IllegalStateException: Buffer pool is destroyed
[ https://issues.apache.org/jira/browse/FLINK-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9054: - Fix Version/s: (was: 1.7.0) > IllegalStateException: Buffer pool is destroyed > --- > > Key: FLINK-9054 > URL: https://issues.apache.org/jira/browse/FLINK-9054 > Project: Flink > Issue Type: Bug > Components: Cluster Management, Configuration, Core >Affects Versions: 1.4.2 >Reporter: dhiraj prajapati >Priority: Major > Fix For: 1.8.0 > > Attachments: flink-conf.yaml > > > Hi, > I have a flink cluster running on 2 machines, say A and B. > Job manager is running on A. There are 2 TaksManagers, one on each node. > So effectively, A has a job manager and a task manager, while B has a task > manager. > When I submit a job to the cluster, I see below exception and the job fails: > 2018-03-22 17:16:52,205 WARN > org.apache.flink.streaming.api.operators.AbstractStreamOperator - Error while > emitting latency marker. > org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: > Could not forward element to next operator > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.emitLatencyMarker(OperatorChain.java:489) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.emitLatencyMarker(AbstractStreamOperator.java:824) > at > org.apache.flink.streaming.api.operators.StreamSource$LatencyMarksEmitter$1.onProcessingTime(StreamSource.java:150) > at > org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:294) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: > Could not forward element to next operator > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.emitLatencyMarker(OperatorChain.java:489) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.emitLatencyMarker(AbstractStreamOperator.java:824) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.reportOrForwardLatencyMarker(AbstractStreamOperator.java:679) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.processLatencyMarker(AbstractStreamOperator.java:662) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.emitLatencyMarker(OperatorChain.java:486) > ... 10 more > Caused by: java.lang.RuntimeException: Buffer pool is destroyed. > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitLatencyMarker(RecordWriterOutput.java:141) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.emitLatencyMarker(OperatorChain.java:604) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.emitLatencyMarker(AbstractStreamOperator.java:824) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.reportOrForwardLatencyMarker(AbstractStreamOperator.java:679) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.processLatencyMarker(AbstractStreamOperator.java:662) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.emitLatencyMarker(OperatorChain.java:486) > ... 14 more > Caused by: java.lang.IllegalStateException: Buffer pool is destroyed. > at > org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBuffer(LocalBufferPool.java:203) > at > org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:191) > at > org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:132) > at > org.apache.flink.runtime.io.network.api.writer.RecordWriter.randomEmit(RecordWriter.java:107) > at > org.apache.flink.streaming.runtime.io.StreamRecordWriter.randomEmit(StreamRecordWriter.java:102) > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitLatencyMarker(RecordWriterOutput.java:138) > ... 19 more > > The exception does not come when I run only one JobManager (only on machine > B). > > I am attaching flink-conf.yaml -- This message was sent by
[jira] [Updated] (FLINK-10610) Port slot sharing cases to new codebase
[
https://issues.apache.org/jira/browse/FLINK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10610:
--
Fix Version/s: (was: 1.7.0)
> Port slot sharing cases to new codebase
> ---
>
> Key: FLINK-10610
> URL: https://issues.apache.org/jira/browse/FLINK-10610
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> Port {{CoLocationConstraintITCase}} and {{SlotSharingITCase}} to new codebase
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9798) Drop canEqual() from TypeInformation, TypeSerializer, etc.
[
https://issues.apache.org/jira/browse/FLINK-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9798:
-
Fix Version/s: (was: 1.7.0)
> Drop canEqual() from TypeInformation, TypeSerializer, etc.
> --
>
> Key: FLINK-9798
> URL: https://issues.apache.org/jira/browse/FLINK-9798
> Project: Flink
> Issue Type: Sub-task
> Components: Core, Type Serialization System
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.8.0
>
>
> See discussion from
> https://lists.apache.org/thread.html/7cc6cfd66e96e8d33c768629b55481b6c951c68128f10256abb328fe@%3Cdev.flink.apache.org%3E
> {quote}
> Hi all!
> As part of an attempt to simplify some code in the TypeInfo and
> TypeSerializer area, I would like to drop the "canEqual" methods for the
> following reason:
> "canEqual()" is necessary to make proper equality checks across hierarchies
> of types. This is for example useful in a collection API, stating for
> example whether a List can be equal to a Collection if they have the same
> contents. We don't have that here.
> A certain type information (and serializer) is equal to another one if they
> describe the same type, strictly. There is no necessity for cross hierarchy
> checks.
> This has also let to the situation that most type infos and serializers
> implement just a dummy/default version of "canEqual". Many "equals()"
> methods do not even call the other object's "canEqual", etc.
> As a first step, we could simply deprecate the method and implement an
> empty default, and remove all calls to that method.
> Best,
> Stephan
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10500) Let ExecutionGraphDriver react to fail signal
[
https://issues.apache.org/jira/browse/FLINK-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10500:
--
Fix Version/s: (was: 1.7.0)
> Let ExecutionGraphDriver react to fail signal
> -
>
> Key: FLINK-10500
> URL: https://issues.apache.org/jira/browse/FLINK-10500
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Shimin Yang
>Priority: Major
> Fix For: 1.8.0
>
>
> In order to scale down when there are not enough resources available or if
> TMs died, the {{ExecutionGraphDriver}} needs to learn about a failure.
> Depending on the failure type and the available set of resources, it can then
> decide to scale the job down or simply restart. In the scope of this issue,
> the {{ExecutionGraphDriver}} should simply call into the {{RestartStrategy}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[
https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-6763:
-
Fix Version/s: (was: 1.7.0)
> Inefficient PojoSerializerConfigSnapshot serialization format
> -
>
> Key: FLINK-6763
> URL: https://issues.apache.org/jira/browse/FLINK-6763
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing, Type Serialization System
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Till Rohrmann
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning
> offset and ending offset in the serialization stream. This information is
> also written if the serializer serialization is supposed to be ignored. The
> beginning and ending offsets are stored as a sequence of integers at the
> beginning of the serialization stream. We store this information to skip
> broken serializers.
> I think we don't need both offsets. Instead I would suggest to write the
> length of the serialized serializer first into the serialization stream and
> then the serialized serializer. This can be done in
> {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the
> serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can
> try to deserialize the serializer. If this operation fails, then we can skip
> the number of serialized serializer because we know how long it was.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9957) Rescale job with respect to available slots
[
https://issues.apache.org/jira/browse/FLINK-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9957:
-
Fix Version/s: (was: 1.7.0)
> Rescale job with respect to available slots
> ---
>
> Key: FLINK-9957
> URL: https://issues.apache.org/jira/browse/FLINK-9957
> Project: Flink
> Issue Type: Sub-task
> Components: JobManager
>Reporter: Till Rohrmann
>Assignee: blues Zheng
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{JobMaster}} which runs in the reactive container mode, needs to react
> to additionally offered slots in order to make use of newly started
> {{TaskExecutors}}. This could mean that the {{JobMaster}} tries to scale the
> job wrt the available number of slots after some grace period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10346) MemoryStateBackend does not clean up checkpoint directory
[
https://issues.apache.org/jira/browse/FLINK-10346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10346:
--
Fix Version/s: (was: 1.7.0)
> MemoryStateBackend does not clean up checkpoint directory
> -
>
> Key: FLINK-10346
> URL: https://issues.apache.org/jira/browse/FLINK-10346
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{StateBackendLoader}} creates a random subdirectory under
> {{HighAvailabilityOptions.HA_STORAGE_PATH}} for the {{MemoryStateBackend}} if
> no checkpointing directory has been specified (see
> {{StateBackendLoader.java:246}}). The problem is that this directory gets
> never removed after the {{MemoryStateBackend}} is closed. It is important to
> clean up these created directories.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10842) Waiting loops are broken in e2e/common.sh
[
https://issues.apache.org/jira/browse/FLINK-10842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10842:
--
Fix Version/s: (was: 1.7.0)
> Waiting loops are broken in e2e/common.sh
> -
>
> Key: FLINK-10842
> URL: https://issues.apache.org/jira/browse/FLINK-10842
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> There are 3 loops in flink-end-to-end-tests/test-scripts/common.sh where the
> script waits for some event to happen (for i in \{1..10}; do):
> - wait_dispatcher_running
> - start_and_wait_for_tm
> - wait_job_running
> All loops have 10 iterations and the loop breaks if the awaited event
> happens. If timeout occurs then the script does not fail and the function
> just continues after 10 iterations ignoring that the awaited event did not
> happen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9193) Deprecate non-well-defined output methods on DataStream
[
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9193:
-
Fix Version/s: 1.8.0
> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0, 1.8.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe
> to use in a streaming program as they have no consistency guarantees. They
> are:
> - {{writeAsText()}}
> - {{writeAsCsv()}}
> - {{writeToSocket()}}
> - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10481) Wordcount end-to-end test in docker env unstable
[
https://issues.apache.org/jira/browse/FLINK-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10481:
--
Fix Version/s: (was: 1.7.0)
> Wordcount end-to-end test in docker env unstable
>
>
> Key: FLINK-10481
> URL: https://issues.apache.org/jira/browse/FLINK-10481
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.5.6, 1.6.3, 1.8.0, 1.7.1
>
>
> The {{Wordcount end-to-end test in docker env}} fails sometimes on Travis
> with the following problem:
> {code}
> Status: Downloaded newer image for java:8-jre-alpine
> ---> fdc893b19a14
> Step 2/16 : RUN apk add --no-cache bash snappy
> ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
> ---> Running in 4329ebcd8a77
> fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
> WARNING: Ignoring
> http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz:
> temporary error (try again later)
> fetch
> http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
> WARNING: Ignoring
> http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz:
> temporary error (try again later)
> ERROR: unsatisfiable constraints:
> bash (missing):
> required by: world[bash]
> snappy (missing):
> required by: world[snappy]
> The command '/bin/sh -c apk add --no-cache bash snappy' returned a non-zero
> code: 2
> {code}
> https://api.travis-ci.org/v3/job/434909395/log.txt
> It seems as if it is related to
> https://github.com/gliderlabs/docker-alpine/issues/264 and
> https://github.com/gliderlabs/docker-alpine/issues/279.
> We might want to switch to a different base image to avoid these problems in
> the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9205) Delete non-well-defined SinkFunctions
[
https://issues.apache.org/jira/browse/FLINK-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9205:
-
Fix Version/s: (was: 1.7.0)
> Delete non-well-defined SinkFunctions
> -
>
> Key: FLINK-9205
> URL: https://issues.apache.org/jira/browse/FLINK-9205
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.8.0
>
>
> Specifically, these are:
> {code}
> OutputFormatSinkFunction.java
> WriteFormat.java
> WriteFormatAsCsv.java
> WriteFormatAsText.java
> WriteSinkFunction.java
> WriteSinkFunctionByMillis.java
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9469) Add tests that cover PatternStream#flatSelect
[ https://issues.apache.org/jira/browse/FLINK-9469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9469: - Fix Version/s: 1.7.1 > Add tests that cover PatternStream#flatSelect > - > > Key: FLINK-9469 > URL: https://issues.apache.org/jira/browse/FLINK-9469 > Project: Flink > Issue Type: Improvement > Components: CEP >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Fix For: 1.6.3, 1.7.0, 1.8.0, 1.7.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8799) Make AbstractYarnClusterDescriptor immutable
[
https://issues.apache.org/jira/browse/FLINK-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-8799:
-
Fix Version/s: (was: 1.7.0)
> Make AbstractYarnClusterDescriptor immutable
>
>
> Key: FLINK-8799
> URL: https://issues.apache.org/jira/browse/FLINK-8799
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> {{AbstractYarnClusterDescriptor}} should be made immutable. Currently, its
> internal configuration is modified from different places which makes it
> difficult to reason about the code. For example, it should not be possible to
> modify the {{zookeeperNamespace}} using a setter method. A user of this class
> should be forced to provide all information prior to creating the instance,
> e.g., by passing a {{org.apache.flink.configuration.Configuration}} object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9980) wiki-edits quickstart docs aren't clear enough
[ https://issues.apache.org/jira/browse/FLINK-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9980: - Fix Version/s: (was: 1.7.0) > wiki-edits quickstart docs aren't clear enough > -- > > Key: FLINK-9980 > URL: https://issues.apache.org/jira/browse/FLINK-9980 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.5.1 >Reporter: David Anderson >Priority: Major > Fix For: 1.8.0 > > > The docs wiki-edit quickstart example should emphasize the need to remove the > provided scope from the flink dependencies. Otherwise the example can't be > run from the command line, and this is confusing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9055) WebUI shows job as Running although not enough resources are available
[ https://issues.apache.org/jira/browse/FLINK-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9055: - Fix Version/s: (was: 1.7.0) > WebUI shows job as Running although not enough resources are available > -- > > Key: FLINK-9055 > URL: https://issues.apache.org/jira/browse/FLINK-9055 > Project: Flink > Issue Type: Bug > Components: JobManager, Webfrontend >Affects Versions: 1.5.0 > Environment: * FLIP-6 enabled > * Local Flink instance with fixed number of TMs > * Job parallelism exceeds available slots >Reporter: Fabian Hueske >Assignee: Sihua Zhou >Priority: Major > Fix For: 1.8.0 > > > The WebUI shows a (batch) job as "Running" although not enough resources have > been allocated to actually run the job with the requested parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-5125) ContinuousFileProcessingCheckpointITCase is Flaky
[
https://issues.apache.org/jira/browse/FLINK-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-5125:
-
Fix Version/s: (was: 1.7.0)
> ContinuousFileProcessingCheckpointITCase is Flaky
> -
>
> Key: FLINK-5125
> URL: https://issues.apache.org/jira/browse/FLINK-5125
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Aljoscha Krettek
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: test-stability
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> This is the travis log:
> https://api.travis-ci.org/jobs/177402367/log.txt?deansi=true
> The relevant sections is:
> {code}
> Running org.apache.flink.test.checkpointing.CoStreamCheckpointingITCase
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.571 sec -
> in org.apache.flink.test.exampleJavaPrograms.EnumTriangleBasicITCase
> Running org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.704 sec -
> in org.apache.flink.test.checkpointing.CoStreamCheckpointingITCase
> Running
> org.apache.flink.test.checkpointing.EventTimeAllWindowCheckpointingITCase
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.805 sec -
> in org.apache.flink.test.checkpointing.EventTimeAllWindowCheckpointingITCase
> Running
> org.apache.flink.test.checkpointing.ContinuousFileProcessingCheckpointITCase
> org.apache.flink.client.program.ProgramInvocationException: The program
> execution failed: Job execution failed.
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
> at
> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:101)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:392)
> at
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:209)
> at
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:173)
> at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:32)
> at
> org.apache.flink.test.checkpointing.StreamFaultToleranceTestBase.runCheckpointedProgram(StreamFaultToleranceTestBase.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
>
[jira] [Updated] (FLINK-10381) concurrent submit job get ProgramAbortException
[
https://issues.apache.org/jira/browse/FLINK-10381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10381:
--
Fix Version/s: (was: 1.7.0)
> concurrent submit job get ProgramAbortException
> ---
>
> Key: FLINK-10381
> URL: https://issues.apache.org/jira/browse/FLINK-10381
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 1.4.0, 1.5.1, 1.6.0
> Environment: Flink 1.4.0, standardalone cluster.
>Reporter: Youjun Yuan
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: image-2018-09-20-22-40-31-846.png
>
>
> if submit multiple jobs concurrently, some the them are likely to fail, and
> return following exception:
> _java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkException: Could not run the jar._
> _at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:90)_
>
> _at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler$$Lambda$47/713642705.get(Unknown
> Source)_
> _at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1582)_
>
> _at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)_
> _at java.util.concurrent.FutureTask.run(FutureTask.java:266)_
> _at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)_
>
> _at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)_
>
> _at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)_
>
> _at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)_
>
> _at java.lang.Thread.run(Thread.java:745)_
> _Caused by: org.apache.flink.util.FlinkException: Could not run the jar. ...
> 10 more_
> _Caused by: org.apache.flink.client.program.ProgramInvocationException: The
> program caused an error:_
> _at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)_
>
> _at
> org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)_
>
> _at
> org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76)_
>
> _at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69)
> ... 9 more_
> _Caused by:
> org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException_
>
> _at
> org.apache.flink.streaming.api.environment.StreamPlanEnvironment.execute(StreamPlanEnvironment.java:72)_
>
> _..._
> _at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_
> _at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)_
>
> _at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
>
> _at java.lang.reflect.Method.invoke(Method.java:497)_
> _at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)_
>
> _at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:417)_
>
> _at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83_
>
> h2. Possible Cause:
> in OptimizerPlanEnvironment.getOptimizerPlan(), setAsContext() will set a
> static variable named contextEnvironmentFactory in ExecutionEnviroment, which
> will eventually cause ExecutionEnviroment.getExecutionEnvironment() returns
> the currently OptimizerPlanEnvironment instance, and capture the
> optimizerPlan and save to a instance vairable in OptimizerPlanEnvironment.
> However, if multiple jobs are submitted at the same time, the static variable
> contextEnvironmentFactory in ExecutionEnvironment will be set again by a
> following job, hence force ExecutionEnviroment.getExecutionEnvironment()
> return another new instance of OptimizerPlanEnvironment, therefore, the first
> intance of OptimizerPlanEnvironment will not caputre the optimizerPlan, and
> throws ProgramInvocationException. The spot is copied below for you convience:
> setAsContext();
> try {
> prog.invokeInteractiveModeForExecution();
> }
> catch (ProgramInvocationException e) {
> throw e;
> }
> catch (Throwable t) {
> // the invocation gets aborted with the preview plan
> if (optimizerPlan != null) {
> return optimizerPlan;
> } else {
> throw new ProgramInvocationException("The program caused an error: ", t);
> }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8997) Add sliding window aggregation to the job
[ https://issues.apache.org/jira/browse/FLINK-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-8997: - Fix Version/s: (was: 1.7.0) > Add sliding window aggregation to the job > - > > Key: FLINK-8997 > URL: https://issues.apache.org/jira/browse/FLINK-8997 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.5.0 >Reporter: Stefan Richter >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > The test job should also test windowing. Sliding windows are probably the > most demanding form, so this would be a good pick for the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10428) AbstractServerBase should bind to any address per default
[
https://issues.apache.org/jira/browse/FLINK-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10428:
--
Fix Version/s: (was: 1.7.0)
> AbstractServerBase should bind to any address per default
> -
>
> Key: FLINK-10428
> URL: https://issues.apache.org/jira/browse/FLINK-10428
> Project: Flink
> Issue Type: Bug
> Components: Queryable State
>Affects Versions: 1.5.4, 1.6.1, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{AbstractServerBase}}, the base class of the queryable state servers, is
> setup to bind to the same hostname as the {{RpcService}} of the
> {{TaskExecutor}}. People ran into the problem that in certain setups where
> you start Flink components locally or in the same container, that the
> queryable state server binds to 127.0.0.1. Therefore it is not possible to
> connect from the outside to this machine.
> I propose to bind per default to any hostname (0.0.0.0) and make it
> configurable in case that a user wants to bind to a specific hostname.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10721) kafkaFetcher runFetchLoop throw exception will cause follow-up code not execute in FlinkKafkaConsumerBase run method
[
https://issues.apache.org/jira/browse/FLINK-10721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10721:
--
Fix Version/s: (was: 1.7.0)
> kafkaFetcher runFetchLoop throw exception will cause follow-up code not
> execute in FlinkKafkaConsumerBase run method
> -
>
> Key: FLINK-10721
> URL: https://issues.apache.org/jira/browse/FLINK-10721
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Affects Versions: 1.6.2
>Reporter: zhaoshijie
>Assignee: Chesnay Schepler
>Priority: Major
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> In FlinkKafkaConsumerBase run method on line 721(master branch), if
> kafkaFetcher.runFetchLoop() throw exception(by discoveryLoopThread throw
> exception then finally execute cancel method, cancel method will execute
> kafkaFetcher.cancel, this implemented Kafka09Fetcher will execute
> handover.close, then result in handover.pollNext throw ClosedException),then
> next code will not execute,especially discoveryLoopError not be throwed,so,
> real culprit exception will be Swallowed.
> failed log like this:
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$10.apply$mcV$sp(JobManager.scala:1229)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$10.apply(JobManager.scala:1172)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$10.apply(JobManager.scala:1172)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by:
> org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
> at
> org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:180)
> at
> org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.cancel(Kafka09Fetcher.java:174)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:753)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase$2.run(FlinkKafkaConsumerBase.java:695)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Shoud we modify it as follows?
> {code:java}
> try {
> kafkaFetcher.runFetchLoop();
> } catch (Exception e) {
> // if discoveryLoopErrorRef not null ,we should
> throw real culprit exception
> if (discoveryLoopErrorRef.get() != null){
> throw new
> RuntimeException(discoveryLoopErrorRef.get());
> } else {
> throw e;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10344) Rethink SubmittedJobGraphListener
[
https://issues.apache.org/jira/browse/FLINK-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10344:
--
Fix Version/s: (was: 1.7.0)
> Rethink SubmittedJobGraphListener
> -
>
> Key: FLINK-10344
> URL: https://issues.apache.org/jira/browse/FLINK-10344
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: Shimin Yang
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{SubmittedJobGraphListener}} in {{ZooKeeperSubmittedJobGraphStore}} can
> return false positives. This is obviously problematic, because it causes the
> subsequent recovery operation to fail. Ideally we would not require the
> {{SubmittedJobGraphListener}}. One could, for example, periodically check
> from the main thread whether there are new jobs. That way we would know which
> jobs are currently running and which are being cleaned up.
> Alternatively it is necessary to tolerate false positives :-(
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10756) TaskManagerProcessFailureBatchRecoveryITCase did not finish on time
[
https://issues.apache.org/jira/browse/FLINK-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10756:
--
Fix Version/s: (was: 1.7.0)
> TaskManagerProcessFailureBatchRecoveryITCase did not finish on time
> ---
>
> Key: FLINK-10756
> URL: https://issues.apache.org/jira/browse/FLINK-10756
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.2, 1.7.0
>Reporter: Bowen Li
>Priority: Critical
> Labels: test-stability
> Fix For: 1.8.0
>
>
> {code:java}
> Failed tests:
>
> TaskManagerProcessFailureBatchRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure:207
> The program did not finish in time
> {code}
> https://travis-ci.org/apache/flink/jobs/449439623
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10252) Handle oversized metric messges
[
https://issues.apache.org/jira/browse/FLINK-10252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10252:
--
Fix Version/s: (was: 1.7.0)
> Handle oversized metric messges
> ---
>
> Key: FLINK-10252
> URL: https://issues.apache.org/jira/browse/FLINK-10252
> Project: Flink
> Issue Type: Sub-task
> Components: Metrics
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.8.0, 1.7.1
>
>
> Since the {{MetricQueryService}} is implemented as an Akka actor, it can only
> send messages of a smaller size then the current {{akka.framesize}}. We
> should check similarly to FLINK-10251 whether the payload exceeds the maximum
> framesize and fail fast if it is true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9142) Lower the minimum number of buffers for incoming channels to 1
[
https://issues.apache.org/jira/browse/FLINK-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9142:
-
Fix Version/s: (was: 1.7.0)
> Lower the minimum number of buffers for incoming channels to 1
> --
>
> Key: FLINK-9142
> URL: https://issues.apache.org/jira/browse/FLINK-9142
> Project: Flink
> Issue Type: Sub-task
> Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Priority: Major
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> Even if we make the floating buffers optional, we still require
> {{taskmanager.network.memory.buffers-per-channel}} number of (exclusive)
> buffers per incoming channel with credit-based flow control while without,
> the minimum was 1 and only the maximum number of buffers was influenced by
> this parameter.
> {{taskmanager.network.memory.buffers-per-channel}} is set to {{2}} by default
> with the argumentation that this way we will have one buffer available for
> netty to process while a worker thread is processing/deserializing the other
> buffer. While this seems reasonable, it does increase our minimum
> requirements. Instead, we could probably live with {{1}} exclusive buffer and
> up to {{gate.getNumberOfInputChannels() * (networkBuffersPerChannel - 1) +
> extraNetworkBuffersPerGate}} floating buffers. That way we will have the same
> memory footprint as before with only slightly changed behaviour.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10819) The instability problem of CI, JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure test fail.
[ https://issues.apache.org/jira/browse/FLINK-10819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10819: -- Fix Version/s: (was: 1.7.0) > The instability problem of CI, > JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure test > fail. > --- > > Key: FLINK-10819 > URL: https://issues.apache.org/jira/browse/FLINK-10819 > Project: Flink > Issue Type: Test > Components: Tests >Affects Versions: 1.7.0 >Reporter: sunjincheng >Priority: Critical > Labels: test-stability > Fix For: 1.8.0 > > > Found the following error in the process of CI： > Results : > Tests in error: > JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure:331 » > IllegalArgument > Tests run: 1463, Failures: 0, Errors: 1, Skipped: 29 > 18:40:55.828 [INFO] > > 18:40:55.829 [INFO] BUILD FAILURE > 18:40:55.829 [INFO] > > 18:40:55.830 [INFO] Total time: 30:19 min > 18:40:55.830 [INFO] Finished at: 2018-11-07T18:40:55+00:00 > 18:40:56.294 [INFO] Final Memory: 92M/678M > 18:40:56.294 [INFO] > > 18:40:56.294 [WARNING] The requested profile "include-kinesis" could not be > activated because it does not exist. > 18:40:56.295 [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test > (integration-tests) on project flink-tests_2.11: There are test failures. > 18:40:56.295 [ERROR] > 18:40:56.295 [ERROR] Please refer to > /home/travis/build/sunjincheng121/flink/flink-tests/target/surefire-reports > for the individual test results. > 18:40:56.295 [ERROR] -> [Help 1] > 18:40:56.295 [ERROR] > 18:40:56.295 [ERROR] To see the full stack trace of the errors, re-run Maven > with the -e switch. > 18:40:56.295 [ERROR] Re-run Maven using the -X switch to enable full debug > logging. > 18:40:56.295 [ERROR] > 18:40:56.295 [ERROR] For more information about the errors and possible > solutions, please read the following articles: > 18:40:56.295 [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > MVN exited with EXIT CODE: 1. > Trying to KILL watchdog (11329). > ./tools/travis_mvn_watchdog.sh: line 269: 11329 Terminated watchdog > PRODUCED build artifacts. > But after the rerun, the error disappeared. > Currently，no specific reasons are found, and will continue to pay attention. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10522) Check if RecoverableWriter supportsResume and act accordingly.
[ https://issues.apache.org/jira/browse/FLINK-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10522: -- Fix Version/s: (was: 1.7.0) > Check if RecoverableWriter supportsResume and act accordingly. > -- > > Key: FLINK-10522 > URL: https://issues.apache.org/jira/browse/FLINK-10522 > Project: Flink > Issue Type: Sub-task > Components: filesystem-connector >Affects Versions: 1.6.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > So far we assumed that all `RecoverableWriters` support "resuming", i.e. > after recovering from a failure or from a savepoint they could keep writing > to the previously "in-progress" file. This assumption holds for all current > writers, but in order to be able to accommodate also filesystems that may not > support this operation, we should check upon initialization if the writer > supports resuming and if yes, we go as before, if not, we recover for commit > and commit the previously in-progress file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10694) ZooKeeperHaServices Cleanup
[
https://issues.apache.org/jira/browse/FLINK-10694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10694:
--
Fix Version/s: (was: 1.7.0)
> ZooKeeperHaServices Cleanup
> ---
>
> Key: FLINK-10694
> URL: https://issues.apache.org/jira/browse/FLINK-10694
> Project: Flink
> Issue Type: Bug
> Components: DataStream API
>Affects Versions: 1.6.1, 1.7.0
>Reporter: Mikhail Pryakhin
>Assignee: vinoyang
>Priority: Critical
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> When a streaming job with Zookeeper-HA enabled gets cancelled all the
> job-related Zookeeper nodes are not removed. Is there a reason behind that?
> I noticed that Zookeeper paths are created of type "Container Node" (an
> Ephemeral node that can have nested nodes) and fall back to Persistent node
> type in case Zookeeper doesn't support this sort of nodes.
> But anyway, it is worth removing the job Zookeeper node when a job is
> cancelled, isn't it?
> zookeeper version 3.4.10
> flink version 1.6.1
> # The job is deployed as a YARN cluster with the following properties set
> {noformat}
> high-availability: zookeeper
> high-availability.zookeeper.quorum:
> high-availability.zookeeper.storageDir: hdfs:///
> high-availability.zookeeper.path.root:
> high-availability.zookeeper.path.namespace:
> {noformat}
> # The job is cancelled via flink cancel command.
> What I've noticed:
> when the job is running the following directory structure is created in
> zookeeper
> {noformat}
> ///leader/resource_manager_lock
> ///leader/rest_server_lock
> ///leader/dispatcher_lock
> ///leader/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
> ///leaderlatch/resource_manager_lock
> ///leaderlatch/rest_server_lock
> ///leaderlatch/dispatcher_lock
> ///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
> ///checkpoints/5c21f00b9162becf5ce25a1cf0e67cde/041
> ///checkpoint-counter/5c21f00b9162becf5ce25a1cf0e67cde
> ///running_job_registry/5c21f00b9162becf5ce25a1cf0e67cde
> {noformat}
> when the job is cancelled some ephemeral nodes disappear, but most of them
> are still there:
> {noformat}
> ///leader/5c21f00b9162becf5ce25a1cf0e67cde
> ///leaderlatch/resource_manager_lock
> ///leaderlatch/rest_server_lock
> ///leaderlatch/dispatcher_lock
> ///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
> ///checkpoints/
> ///checkpoint-counter/
> ///running_job_registry/
> {noformat}
> Here is the method [1] responsible for cleaning zookeeper folders up [1]
> which is called when a job manager has stopped [2].
> And it seems it only cleans up the *running_job_registry* folder, other
> folders stay untouched. I suppose that everything under the
> *///* folder should be cleaned up when the
> job is cancelled.
> [1]
> [https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperRunningJobsRegistry.java#L107]
> [2]
> [https://github.com/apache/flink/blob/f087f57749004790b6f5b823d66822c36ae09927/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L332]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8899) Submitting YARN job with FLIP-6 may lead to ApplicationAttemptNotFoundException
[
https://issues.apache.org/jira/browse/FLINK-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-8899:
-
Fix Version/s: (was: 1.7.0)
> Submitting YARN job with FLIP-6 may lead to
> ApplicationAttemptNotFoundException
> ---
>
> Key: FLINK-8899
> URL: https://issues.apache.org/jira/browse/FLINK-8899
> Project: Flink
> Issue Type: Bug
> Components: ResourceManager, YARN
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Priority: Major
> Labels: flip-6
> Fix For: 1.5.6, 1.6.3, 1.8.0, 1.7.1
>
>
> Occasionally, running a simple word count as this
> {code}
> ./bin/flink run -m yarn-cluster -yjm 768 -ytm 3072 -ys 2 -p 20 -c
> org.apache.flink.streaming.examples.wordcount.WordCount
> ./examples/streaming/WordCount.jar --input /usr/share/doc/rsync-3.0.6/COPYING
> {code}
> leads to an {{ApplicationAttemptNotFoundException}} in the logs:
> {code}
> 2018-03-08 16:18:08,507 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job Streaming
> WordCount (df707a3c9817ddf5936efe56d427e2bd) switched from state RUNNING to
> FINISHED.
> 2018-03-08 16:18:08,508 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping
> checkpoint coordinator for job df707a3c9817ddf5936efe56d427e2bd
> 2018-03-08 16:18:08,508 INFO
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore -
> Shutting down
> 2018-03-08 16:18:08,536 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job
> df707a3c9817ddf5936efe56d427e2bd reached globally terminal state FINISHED.
> 2018-03-08 16:18:08,611 INFO org.apache.flink.runtime.jobmaster.JobMaster
> - Stopping the JobMaster for job Streaming
> WordCount(df707a3c9817ddf5936efe56d427e2bd).
> 2018-03-08 16:18:08,634 INFO org.apache.flink.runtime.jobmaster.JobMaster
> - Close ResourceManager connection
> dcfdc329d61aae0ace2de26292c8916b: JobManager is shutting down..
> 2018-03-08 16:18:08,634 INFO org.apache.flink.yarn.YarnResourceManager
> - Disconnect job manager
> 0...@akka.tcp://fl...@ip-172-31-2-0.eu-west-1.compute.internal:38555/user/jobmanager_0
> for job df707a3c9817ddf5936efe56d427e2bd from the resource manager.
> 2018-03-08 16:18:08,664 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Suspending
> SlotPool.
> 2018-03-08 16:18:08,664 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Stopping
> SlotPool.
> 2018-03-08 16:18:08,664 INFO
> org.apache.flink.runtime.jobmaster.JobManagerRunner -
> JobManagerRunner already shutdown.
> 2018-03-08 16:18:09,650 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Register
> TaskManager adc8090bdb3f7052943ff86bde7d2a7b at the SlotManager.
> 2018-03-08 16:18:09,654 INFO org.apache.flink.yarn.YarnResourceManager
> - Replacing old instance of worker for ResourceID
> container_1519984124671_0090_01_05
> 2018-03-08 16:18:09,654 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager -
> Unregister TaskManager adc8090bdb3f7052943ff86bde7d2a7b from the SlotManager.
> 2018-03-08 16:18:09,654 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Register
> TaskManager b975dbd16e0fd59c1168d978490a4b76 at the SlotManager.
> 2018-03-08 16:18:09,654 INFO org.apache.flink.yarn.YarnResourceManager
> - The target with resource ID
> container_1519984124671_0090_01_05 is already been monitored.
> 2018-03-08 16:18:09,992 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Register
> TaskManager 73c258a0dbad236501b8391971c330ba at the SlotManager.
> 2018-03-08 16:18:10,000 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2018-03-08 16:18:10,028 ERROR
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Exception
> on heartbeat
> org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException:
> Application attempt appattempt_1519984124671_0090_01 doesn't exist in
> ApplicationMasterService cache.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:403)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
>
[jira] [Updated] (FLINK-10625) Add MATCH_RECOGNIZE documentation
[
https://issues.apache.org/jira/browse/FLINK-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10625:
--
Fix Version/s: (was: 1.7.0)
> Add MATCH_RECOGNIZE documentation
> -
>
> Key: FLINK-10625
> URL: https://issues.apache.org/jira/browse/FLINK-10625
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Table API SQL
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> The newly added {{MATCH_RECOGNIZE}} functionality needs to be documented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10751) Checkpoints should be retained when job reaches suspended state
[
https://issues.apache.org/jira/browse/FLINK-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10751:
--
Fix Version/s: (was: 1.7.0)
> Checkpoints should be retained when job reaches suspended state
> ---
>
> Key: FLINK-10751
> URL: https://issues.apache.org/jira/browse/FLINK-10751
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.6.2, 1.7.0
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> {{CheckpointProperties}} define in which terminal job status a checkpoint
> should be disposed.
> I've noticed that the properties for {{CHECKPOINT_NEVER_RETAINED}},
> {{CHECKPOINT_RETAINED_ON_FAILURE}} prescribe checkpoint disposal in (locally)
> terminal job status {{SUSPENDED}}.
> Since a job reaches the {{SUSPENDED}} state when its {{JobMaster}} looses
> leadership, this would result in the checkpoint to be cleaned up and not
> being available for recovery by the new leader. Therefore, we should rather
> retain checkpoints when reaching job status {{SUSPENDED}}.
> *BUT:* Because we special case this terminal state in the only highly
> available {{CompletedCheckpointStore}} implementation (see
> [ZooKeeperCompletedCheckpointStore|https://github.com/apache/flink/blob/e7ac3ba/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java#L315])
> and don't use regular checkpoint disposal, this issue has not surfaced yet.
> I think we should proactively fix the properties to indicate to retain
> checkpoints in {{SUSPENDED}} state. We might actually completely remove this
> case since with this change, all properties will indicate to retain on
> suspension.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10140) Log files are not available via web UI in containerized environment
[
https://issues.apache.org/jira/browse/FLINK-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10140:
--
Fix Version/s: (was: 1.7.0)
> Log files are not available via web UI in containerized environment
> ---
>
> Key: FLINK-10140
> URL: https://issues.apache.org/jira/browse/FLINK-10140
> Project: Flink
> Issue Type: Bug
> Components: Docker, Kubernetes
>Affects Versions: 1.5.2, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.8.0
>
>
> Since we start Flink components in the foreground (see
> {{flink-contrib/docker-flink/docker-entrypoint.sh}} and
> {{flink-container/docker/docker-entrypoint.sh}} we print the log statements
> to STDOUT and don't write them into a file. Consequently, the web UI cannot
> server the log files since they don't exist.
> A simple way to fix the problem is to also create a log file like in daemon
> mode
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10712) RestartPipelinedRegionStrategy does not restore state
[ https://issues.apache.org/jira/browse/FLINK-10712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10712: -- Fix Version/s: (was: 1.7.0) > RestartPipelinedRegionStrategy does not restore state > - > > Key: FLINK-10712 > URL: https://issues.apache.org/jira/browse/FLINK-10712 > Project: Flink > Issue Type: Bug > Components: Distributed Coordination >Affects Versions: 1.3.3, 1.4.2, 1.5.5, 1.6.2, 1.7.0 >Reporter: Stefan Richter >Assignee: Yun Tang >Priority: Critical > Labels: pull-request-available > Fix For: 1.8.0 > > > RestartPipelinedRegionStrategy does not perform any state restore. This is > big problem because all restored regions will be restarted with empty state. > We need to take checkpoints into account when restoring. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10507) Set target parallelism to maximum when using the standalone job cluster mode
[ https://issues.apache.org/jira/browse/FLINK-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10507: -- Fix Version/s: (was: 1.7.0) > Set target parallelism to maximum when using the standalone job cluster mode > > > Key: FLINK-10507 > URL: https://issues.apache.org/jira/browse/FLINK-10507 > Project: Flink > Issue Type: Sub-task > Components: JobManager >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: vinoyang >Priority: Major > Fix For: 1.8.0 > > > In order to enable the reactive container mode, we should set the target > value to the maximum parallelism if we run in standalone job cluster mode. > That way, we will always use all available resources and scale up if new > resources are being added. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10335) Create common ZooKeeperStateStore based on ZooKeeperStateHandleStore and RetrievableStateStorageHelper
[
https://issues.apache.org/jira/browse/FLINK-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10335:
--
Fix Version/s: (was: 1.7.0)
> Create common ZooKeeperStateStore based on ZooKeeperStateHandleStore and
> RetrievableStateStorageHelper
> --
>
> Key: FLINK-10335
> URL: https://issues.apache.org/jira/browse/FLINK-10335
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Create a common {{ZooKeeperStateStore}} which is based on
> {{ZooKeeperStateHandleStore}} and the {{RetrievableStateStorageHelper}} which
> encapsulates the storage logic of large state objects whose handles are
> persisted to ZooKeeper. This could be used by the
> {{ZooKeeperSubmittedJobGraphStore}}, {{ZooKeeperCompletedCheckpointStore}}
> and {{ZooKeeperMesosWorkerStore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10510) Bump asm version to 6
[ https://issues.apache.org/jira/browse/FLINK-10510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10510: -- Fix Version/s: (was: 1.7.0) > Bump asm version to 6 > - > > Key: FLINK-10510 > URL: https://issues.apache.org/jira/browse/FLINK-10510 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.7.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: shaded-5.0, 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9646) ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-9646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9646:
-
Fix Version/s: (was: 1.7.0)
> ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on
> Travis
>
>
> Key: FLINK-9646
> URL: https://issues.apache.org/jira/browse/FLINK-9646
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.5.6, 1.6.3, 1.8.0, 1.7.1
>
>
> {{ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart}} fails on
> Travis.
> https://api.travis-ci.org/v3/job/395394863/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9207) Client returns SUCCESS(0) return code for canceled job
[ https://issues.apache.org/jira/browse/FLINK-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-9207: - Fix Version/s: (was: 1.7.0) > Client returns SUCCESS(0) return code for canceled job > -- > > Key: FLINK-9207 > URL: https://issues.apache.org/jira/browse/FLINK-9207 > Project: Flink > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0 > Environment: Version: 1.5.0, Commit : 2af481a >Reporter: Amit Jain >Priority: Major > Fix For: 1.8.0 > > > Flink Client returns zero return code when a job is deliberately canceled. > Steps to reproduced it: > 1. bin/flink run -p 10 -m yarn-cluster -yjm 1024 -ytm 12288 WordCount.jar > 2. User externally canceled the job. > 3. Job Manager marked the job as CANCELED. > 4. Although client code emits following logs, still returns zero return code. > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Killed > application application_1523726493647_. > Job scheduler like Airflow would have hard-time detecting whether the > submitted job was canceled or not. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10505) Treat fail signal as scheduling event
[
https://issues.apache.org/jira/browse/FLINK-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10505:
--
Fix Version/s: (was: 1.7.0)
> Treat fail signal as scheduling event
> -
>
> Key: FLINK-10505
> URL: https://issues.apache.org/jira/browse/FLINK-10505
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Instead of simply calling into the {{RestartStrategy}} which restarts the
> existing {{ExecutionGraph}} with the same parallelism, the
> {{ExecutionGraphDriver}} should treat a recovery similar to the initial
> scheduling operation. First, one needs to decide on the new parallelism of
> the {{ExecutionGraph}} (scale up/scale down) wrt to the available set of
> resources. Only if the minimum configuration is fulfilled, the potentially
> rescaled {{ExecutionGraph}} will be restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10333) Rethink ZooKeeper based stores (SubmittedJobGraph, MesosWorker, CompletedCheckpoints)
[
https://issues.apache.org/jira/browse/FLINK-10333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10333:
--
Fix Version/s: (was: 1.7.0)
> Rethink ZooKeeper based stores (SubmittedJobGraph, MesosWorker,
> CompletedCheckpoints)
> -
>
> Key: FLINK-10333
> URL: https://issues.apache.org/jira/browse/FLINK-10333
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.5.3, 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.8.0
>
>
> While going over the ZooKeeper based stores
> ({{ZooKeeperSubmittedJobGraphStore}}, {{ZooKeeperMesosWorkerStore}},
> {{ZooKeeperCompletedCheckpointStore}}) and the underlying
> {{ZooKeeperStateHandleStore}} I noticed several inconsistencies which were
> introduced with past incremental changes.
> * Depending whether {{ZooKeeperStateHandleStore#getAllSortedByNameAndLock}}
> or {{ZooKeeperStateHandleStore#getAllAndLock}} is called, deserialization
> problems will either lead to removing the Znode or not
> * {{ZooKeeperStateHandleStore}} leaves inconsistent state in case of
> exceptions (e.g. {{#getAllAndLock}} won't release the acquired locks in case
> of a failure)
> * {{ZooKeeperStateHandleStore}} has too many responsibilities. It would be
> better to move {{RetrievableStateStorageHelper}} out of it for a better
> separation of concerns
> * {{ZooKeeperSubmittedJobGraphStore}} overwrites a stored {{JobGraph}} even
> if it is locked. This should not happen since it could leave another system
> in an inconsistent state (imagine a changed {{JobGraph}} which restores from
> an old checkpoint)
> * Redundant but also somewhat inconsistent put logic in the different stores
> * Shadowing of ZooKeeper specific exceptions in {{ZooKeeperStateHandleStore}}
> which were expected to be caught in {{ZooKeeperSubmittedJobGraphStore}}
> * Getting rid of the {{SubmittedJobGraphListener}} would be helpful
> These problems made me think how reliable these components actually work.
> Since these components are very important, I propose to refactor them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-6437) Move history server configuration to a separate file
[
https://issues.apache.org/jira/browse/FLINK-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-6437:
-
Fix Version/s: (was: 1.7.0)
> Move history server configuration to a separate file
>
>
> Key: FLINK-6437
> URL: https://issues.apache.org/jira/browse/FLINK-6437
> Project: Flink
> Issue Type: Improvement
> Components: History Server
>Affects Versions: 1.3.0
>Reporter: Stephan Ewen
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> I suggest to keep the {{flink-conf.yaml}} leaner by moving configuration of
> the History Server to a different file.
> In general, I would propose to move configurations of separate, independent
> and optional components to individual config files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-7981) Bump commons-lang3 version to 3.6
[
https://issues.apache.org/jira/browse/FLINK-7981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-7981:
-
Fix Version/s: (was: 1.7.0)
> Bump commons-lang3 version to 3.6
> -
>
> Key: FLINK-7981
> URL: https://issues.apache.org/jira/browse/FLINK-7981
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Hai Zhou
>Assignee: Hai Zhou
>Priority: Trivial
> Fix For: 1.8.0
>
>
> Update commons-lang3 from 3.3.2 to 3.6.
> {{SerializationUtils.clone()}} of commons-lang3 (<3.5) has a bug that break
> thread safety, which gets stack sometimes caused by race condition of
> initializing hash map.
> See https://issues.apache.org/jira/browse/LANG-1251.
> **other**
> [BEAM-2481:Update commons-lang3 dependency to version
> 3.6|https://issues.apache.org/jira/browse/BEAM-2481]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10583) Add support for state retention to the Processing Time versioned joins.
[ https://issues.apache.org/jira/browse/FLINK-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-10583: -- Fix Version/s: (was: 1.7.0) > Add support for state retention to the Processing Time versioned joins. > --- > > Key: FLINK-10583 > URL: https://issues.apache.org/jira/browse/FLINK-10583 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Affects Versions: 1.7.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9462) Disentangle flink-json and flink-table
[
https://issues.apache.org/jira/browse/FLINK-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9462:
-
Fix Version/s: (was: 1.7.0)
> Disentangle flink-json and flink-table
> --
>
> Key: FLINK-9462
> URL: https://issues.apache.org/jira/browse/FLINK-9462
> Project: Flink
> Issue Type: Improvement
> Components: Build System, Table API SQL
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{flink-json}} module defines Json serialization and deserialization
> schemas. Additionally, it defines Json table descriptor. Due to this, it has
> a dependency on {{flink-table}}. We should either rename this module into
> {{flink-json-table}} or move the table API specific classes into a different
> module. That way we could remove the dependency on {{flink-table}} which
> decouples the Json serialization and deserialization schemas from the Table
> API on which the schemas should not depend.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10304) Remove deprecated AbstractYarnClusterDescriptor field
[
https://issues.apache.org/jira/browse/FLINK-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10304:
--
Fix Version/s: (was: 1.7.0)
> Remove deprecated AbstractYarnClusterDescriptor field
> -
>
> Key: FLINK-10304
> URL: https://issues.apache.org/jira/browse/FLINK-10304
> Project: Flink
> Issue Type: Improvement
> Components: Client, YARN
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> Depend on [~trohrm...@apache.org]'s
> [commit|https://github.com/apache/flink/commit/6356128865bff7463bf03185d18b129ed3633bc2],
> {{AbstractYarnClusterDescriptor}} should not care whether it is in DETACHED
> mode.
> After digging I found the main usages of it are
> 1. {{FlinkYarnSessionCli#run}}, this can be resolved by checking whether
> {{allOptions}} has {{DETACHED_OPTION}} locally.
> 2. when AbstractYarnClusterDescriptor start a AM, it sets
> {{appMasterEnv.put(YarnConfigKeys.ENV_DETACHED, String.valueOf(detached));}}.
> At this point it seems that YarnClusterDescriptor should know whether or not
> it is in detached mode.
> If usage 2 is irrelevant now, we can get rid of deprecated method in FLIP-6
> codebase.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10665) Port YARNSessionFIFOITCase#testJavaAPI to new codebase
[
https://issues.apache.org/jira/browse/FLINK-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10665:
--
Fix Version/s: (was: 1.7.0)
> Port YARNSessionFIFOITCase#testJavaAPI to new codebase
> --
>
> Key: FLINK-10665
> URL: https://issues.apache.org/jira/browse/FLINK-10665
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.7.0
>Reporter: TisonKun
>Assignee: TisonKun
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> Port {{YARNSessionFIFOITCase#testJavaAPI}} to new codebase
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
