[jira] [Commented] (FLINK-9514) Create wrapper with TTL logic for value state
[
https://issues.apache.org/jira/browse/FLINK-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541073#comment-16541073
]
ASF GitHub Bot commented on FLINK-9514:
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6186
Hi, after read the whole implementation, i found that the state is expired
when it is accessed, When there is the dirty data store to state and never be
queried, how does it can be expired. Or is there an undergoing work for this ?
@azagrebin
> Create wrapper with TTL logic for value state
> -
>
> Key: FLINK-9514
> URL: https://issues.apache.org/jira/browse/FLINK-9514
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> TTL state decorator uses original state with packed TTL and add TTL logic
> using time provider:
> {code:java}
> TtlValueState implements ValueState {
> ValueState> underlyingState;
> InternalTimeService timeProvider;
> V value() {
> TtlValue valueWithTtl = underlyingState.get();
> // ttl logic here (e.g. update timestamp)
> return valueWithTtl.getValue();
> }
> void update() { ... underlyingState.update(valueWithTtl) ... }
> }
> {code}
> TTL decorators are apply to state produced by normal state binder in its TTL
> wrapper from FLINK-9513
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9827) ResourceManager may receive outdate report of slots status from task manager
shuai.xu created FLINK-9827: --- Summary: ResourceManager may receive outdate report of slots status from task manager Key: FLINK-9827 URL: https://issues.apache.org/jira/browse/FLINK-9827 Project: Flink Issue Type: Bug Components: Cluster Management Affects Versions: 1.5.0 Reporter: shuai.xu Assignee: shuai.xu TaskExecutor will report its slot status to resource manager in heartbeat, but this is in a different thread with the main rpc thread. So it may happen that rm request a slot from task executor but then receive a heartbeat saying the slot not assigned. This will cause the slot be freed and assigned again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9817) Version not correctly bumped to 5.0
[
https://issues.apache.org/jira/browse/FLINK-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9817:
--
Labels: pull-request-available (was: )
> Version not correctly bumped to 5.0
> ---
>
> Key: FLINK-9817
> URL: https://issues.apache.org/jira/browse/FLINK-9817
> Project: Flink
> Issue Type: Bug
> Components: flink-shaded.git
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
>
> Current {{master}} of {{flink-shaded}} only made half of the changes needed
> when updating version numbers: the suffix in each sub-module's version number
> also needs to be adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6300: [FLINK-9692][Kinesis Connector] Adaptive reads fro...
Github user tweise commented on a diff in the pull request:
https://github.com/apache/flink/pull/6300#discussion_r201792650
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
---
@@ -224,10 +232,19 @@ public void run() {
subscribedShard.getShard().getHashKeyRange().getStartingHashKey(),
subscribedShard.getShard().getHashKeyRange().getEndingHashKey());
+ long recordBatchSizeBytes = 0L;
+ long averageRecordSizeBytes = 0L;
+
for (UserRecord record :
fetchedRecords) {
+ recordBatchSizeBytes +=
record.getData().remaining();
deserializeRecordForCollectionAndUpdateState(record);
}
+ if (useAdaptiveReads &&
fetchedRecords.size() != 0) {
--- End diff --
nit: && !isEmpty()
---
[jira] [Commented] (FLINK-9692) Adapt maxRecords parameter in the getRecords call to optimize bytes read from Kinesis
[
https://issues.apache.org/jira/browse/FLINK-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540706#comment-16540706
]
ASF GitHub Bot commented on FLINK-9692:
---
Github user tweise commented on a diff in the pull request:
https://github.com/apache/flink/pull/6300#discussion_r201792650
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
---
@@ -224,10 +232,19 @@ public void run() {
subscribedShard.getShard().getHashKeyRange().getStartingHashKey(),
subscribedShard.getShard().getHashKeyRange().getEndingHashKey());
+ long recordBatchSizeBytes = 0L;
+ long averageRecordSizeBytes = 0L;
+
for (UserRecord record :
fetchedRecords) {
+ recordBatchSizeBytes +=
record.getData().remaining();
deserializeRecordForCollectionAndUpdateState(record);
}
+ if (useAdaptiveReads &&
fetchedRecords.size() != 0) {
--- End diff --
nit: && !isEmpty()
> Adapt maxRecords parameter in the getRecords call to optimize bytes read from
> Kinesis
> --
>
> Key: FLINK-9692
> URL: https://issues.apache.org/jira/browse/FLINK-9692
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Lakshmi Rao
>Assignee: Lakshmi Rao
>Priority: Major
> Labels: performance, pull-request-available
>
> The Kinesis connector currently has a [constant
> value|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java#L213]
> set for maxRecords that it can fetch from a single Kinesis getRecords call.
> However, in most realtime scenarios, the average size of the Kinesis record
> (in bytes) changes depending on the situation i.e. you could be in a
> transient scenario where you are reading large sized records and would hence
> like to fetch fewer records in each getRecords call (so as to not exceed the
> 2 Mb/sec [per shard
> limit|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html]
> on the getRecords call).
> The idea here is to adapt the Kinesis connector to identify an average batch
> size prior to making the getRecords call, so that the maxRecords parameter
> can be appropriately tuned before making the call.
> This feature can be behind a
> [ConsumerConfigConstants|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java]
> flag that defaults to false.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9692) Adapt maxRecords parameter in the getRecords call to optimize bytes read from Kinesis
[
https://issues.apache.org/jira/browse/FLINK-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540707#comment-16540707
]
ASF GitHub Bot commented on FLINK-9692:
---
Github user tweise commented on a diff in the pull request:
https://github.com/apache/flink/pull/6300#discussion_r201793459
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
---
@@ -330,4 +347,24 @@ private GetRecordsResult getRecords(String shardItr,
int maxNumberOfRecords) thr
protected static List deaggregateRecords(List
records, String startingHashKey, String endingHashKey) {
return UserRecord.deaggregate(records, new
BigInteger(startingHashKey), new BigInteger(endingHashKey));
}
+
+ /**
+* Adapts the maxNumberOfRecordsPerFetch based on the current average
record size
+* to optimize 2 Mb / sec read limits.
+*
+* @param averageRecordSizeBytes
+* @return adaptedMaxRecordsPerFetch
+*/
+
+ private int getAdaptiveMaxRecordsPerFetch(long averageRecordSizeBytes) {
--- End diff --
Make this protected to allow for override? (Currently the shard consumer as
a whole cannot be customized, but I think it should.)
> Adapt maxRecords parameter in the getRecords call to optimize bytes read from
> Kinesis
> --
>
> Key: FLINK-9692
> URL: https://issues.apache.org/jira/browse/FLINK-9692
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Lakshmi Rao
>Assignee: Lakshmi Rao
>Priority: Major
> Labels: performance, pull-request-available
>
> The Kinesis connector currently has a [constant
> value|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java#L213]
> set for maxRecords that it can fetch from a single Kinesis getRecords call.
> However, in most realtime scenarios, the average size of the Kinesis record
> (in bytes) changes depending on the situation i.e. you could be in a
> transient scenario where you are reading large sized records and would hence
> like to fetch fewer records in each getRecords call (so as to not exceed the
> 2 Mb/sec [per shard
> limit|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html]
> on the getRecords call).
> The idea here is to adapt the Kinesis connector to identify an average batch
> size prior to making the getRecords call, so that the maxRecords parameter
> can be appropriately tuned before making the call.
> This feature can be behind a
> [ConsumerConfigConstants|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java]
> flag that defaults to false.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6300: [FLINK-9692][Kinesis Connector] Adaptive reads fro...
Github user tweise commented on a diff in the pull request:
https://github.com/apache/flink/pull/6300#discussion_r201793459
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
---
@@ -330,4 +347,24 @@ private GetRecordsResult getRecords(String shardItr,
int maxNumberOfRecords) thr
protected static List deaggregateRecords(List
records, String startingHashKey, String endingHashKey) {
return UserRecord.deaggregate(records, new
BigInteger(startingHashKey), new BigInteger(endingHashKey));
}
+
+ /**
+* Adapts the maxNumberOfRecordsPerFetch based on the current average
record size
+* to optimize 2 Mb / sec read limits.
+*
+* @param averageRecordSizeBytes
+* @return adaptedMaxRecordsPerFetch
+*/
+
+ private int getAdaptiveMaxRecordsPerFetch(long averageRecordSizeBytes) {
--- End diff --
Make this protected to allow for override? (Currently the shard consumer as
a whole cannot be customized, but I think it should.)
---
[jira] [Commented] (FLINK-9819) Create start up scripts for the StandaloneJobClusterEntryPoint
[
https://issues.apache.org/jira/browse/FLINK-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540777#comment-16540777
]
ASF GitHub Bot commented on FLINK-9819:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/6316
[FLINK-9819] Add startup scripts for standalone job cluster entry point
## What is the purpose of the change
Add startup shell scripts for the `StandaloneJobClusterEntryPoint`.
This PR is based on #6315.
cc @GJL
## Verifying this change
- Tested manually
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink standaloneJobStartupScripts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6316.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6316
commit b798af824433657ca215f8598112094a23819ee0
Author: Till Rohrmann
Date: 2018-07-11T15:30:53Z
[hotfix] Make PackagedProgram(Class, String...) constructor public
commit d373b6b01ec9d5b63513718a8e6b7db87629a477
Author: Till Rohrmann
Date: 2018-07-11T15:41:27Z
[FLINK-9818] Add cluster component command line parser
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922
Author: Till Rohrmann
Date: 2018-07-09T21:54:55Z
[FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint
The StandaloneJobClusterEntryPoint is the basic entry point for containers.
It is started with
the user code jar in its classpath and the classname of the user program.
The entrypoint will
then load this user program via the classname and execute its main method.
This will generate
a JobGraph which is then used to start the MiniDispatcher.
commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa
Author: Till Rohrmann
Date: 2018-07-10T09:24:26Z
[FLINK-9819] Add startup scripts for standalone job cluster entry point
> Create start up scripts for the StandaloneJobClusterEntryPoint
> --
>
> Key: FLINK-9819
> URL: https://issues.apache.org/jira/browse/FLINK-9819
> Project: Flink
> Issue Type: New Feature
> Components: Startup Shell Scripts
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> In order to start the {{StandaloneJobClusterEntryPoint}} we need start up
> scripts in {{flink-dist}}. We should extend the {{flink-daemon.sh}} and the
> {{flink-console.sh}} scripts to support this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6316: [FLINK-9819] Add startup scripts for standalone jo...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6316 [FLINK-9819] Add startup scripts for standalone job cluster entry point ## What is the purpose of the change Add startup shell scripts for the `StandaloneJobClusterEntryPoint`. This PR is based on #6315. cc @GJL ## Verifying this change - Tested manually ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink standaloneJobStartupScripts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6316.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6316 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa Author: Till Rohrmann Date: 2018-07-10T09:24:26Z [FLINK-9819] Add startup scripts for standalone job cluster entry point ---
[jira] [Updated] (FLINK-9819) Create start up scripts for the StandaloneJobClusterEntryPoint
[
https://issues.apache.org/jira/browse/FLINK-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9819:
--
Labels: pull-request-available (was: )
> Create start up scripts for the StandaloneJobClusterEntryPoint
> --
>
> Key: FLINK-9819
> URL: https://issues.apache.org/jira/browse/FLINK-9819
> Project: Flink
> Issue Type: New Feature
> Components: Startup Shell Scripts
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> In order to start the {{StandaloneJobClusterEntryPoint}} we need start up
> scripts in {{flink-dist}}. We should extend the {{flink-daemon.sh}} and the
> {{flink-console.sh}} scripts to support this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9820) Let dynamic properties overwrite configuration settings in ClusterEntrypoint
Till Rohrmann created FLINK-9820:
Summary: Let dynamic properties overwrite configuration settings
in ClusterEntrypoint
Key: FLINK-9820
URL: https://issues.apache.org/jira/browse/FLINK-9820
Project: Flink
Issue Type: Improvement
Components: Distributed Coordination
Affects Versions: 1.6.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.6.0
The dynamic properties which are passed to the {{ClusterEntrypoint}} should
overwrite values in the loaded {{Configuration}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6317: [FLINK-9820] Forward dynamic properties to Flink c...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6317 [FLINK-9820] Forward dynamic properties to Flink configuration in ClusterEntrypoint ## What is the purpose of the change With this commit we can use dynamic properties to overwrite configuration values in the `ClusterEntrypoint`. This PR is based on #6316. cc @GJL ## Verifying this change - Added `ConfigurationUtilsTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink forwardDynamicPropertiesClusterEntryPoint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6317.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6317 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa Author: Till Rohrmann Date: 2018-07-10T09:24:26Z [FLINK-9819] Add startup scripts for standalone job cluster entry point commit 75fb8125e3ec270994628f46457b281cdb587874 Author: Till Rohrmann Date: 2018-07-10T21:23:59Z [FLINK-9820] Forward dynamic properties to Flink configuration in ClusterEntrypoint With this commit we can use dynamic properties to overwrite configuration values in the ClusterEntrypoint. ---
[jira] [Commented] (FLINK-9820) Let dynamic properties overwrite configuration settings in ClusterEntrypoint
[
https://issues.apache.org/jira/browse/FLINK-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540801#comment-16540801
]
ASF GitHub Bot commented on FLINK-9820:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/6317
[FLINK-9820] Forward dynamic properties to Flink configuration in
ClusterEntrypoint
## What is the purpose of the change
With this commit we can use dynamic properties to overwrite configuration
values in the
`ClusterEntrypoint`.
This PR is based on #6316.
cc @GJL
## Verifying this change
- Added `ConfigurationUtilsTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink
forwardDynamicPropertiesClusterEntryPoint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6317.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6317
commit b798af824433657ca215f8598112094a23819ee0
Author: Till Rohrmann
Date: 2018-07-11T15:30:53Z
[hotfix] Make PackagedProgram(Class, String...) constructor public
commit d373b6b01ec9d5b63513718a8e6b7db87629a477
Author: Till Rohrmann
Date: 2018-07-11T15:41:27Z
[FLINK-9818] Add cluster component command line parser
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922
Author: Till Rohrmann
Date: 2018-07-09T21:54:55Z
[FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint
The StandaloneJobClusterEntryPoint is the basic entry point for containers.
It is started with
the user code jar in its classpath and the classname of the user program.
The entrypoint will
then load this user program via the classname and execute its main method.
This will generate
a JobGraph which is then used to start the MiniDispatcher.
commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa
Author: Till Rohrmann
Date: 2018-07-10T09:24:26Z
[FLINK-9819] Add startup scripts for standalone job cluster entry point
commit 75fb8125e3ec270994628f46457b281cdb587874
Author: Till Rohrmann
Date: 2018-07-10T21:23:59Z
[FLINK-9820] Forward dynamic properties to Flink configuration in
ClusterEntrypoint
With this commit we can use dynamic properties to overwrite configuration
values in the
ClusterEntrypoint.
> Let dynamic properties overwrite configuration settings in ClusterEntrypoint
>
>
> Key: FLINK-9820
> URL: https://issues.apache.org/jira/browse/FLINK-9820
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> The dynamic properties which are passed to the {{ClusterEntrypoint}} should
> overwrite values in the loaded {{Configuration}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9820) Let dynamic properties overwrite configuration settings in ClusterEntrypoint
[
https://issues.apache.org/jira/browse/FLINK-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9820:
--
Labels: pull-request-available (was: )
> Let dynamic properties overwrite configuration settings in ClusterEntrypoint
>
>
> Key: FLINK-9820
> URL: https://issues.apache.org/jira/browse/FLINK-9820
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> The dynamic properties which are passed to the {{ClusterEntrypoint}} should
> overwrite values in the loaded {{Configuration}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9823) Add Kubernetes deployment files for standalone job cluster
[ https://issues.apache.org/jira/browse/FLINK-9823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-9823: -- Labels: pull-request-available (was: ) > Add Kubernetes deployment files for standalone job cluster > -- > > Key: FLINK-9823 > URL: https://issues.apache.org/jira/browse/FLINK-9823 > Project: Flink > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > > Similar to FLINK-9822, it would be helpful for the user to have example > Kubernetes deployment files to start a standalone job cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9823) Add Kubernetes deployment files for standalone job cluster
[ https://issues.apache.org/jira/browse/FLINK-9823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540869#comment-16540869 ] ASF GitHub Bot commented on FLINK-9823: --- GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6320 [FLINK-9823] Add Kubernetes deployment ymls ## What is the purpose of the change The Kubernetes files contain a job-cluster service specification, a job specification for the StandaloneJobClusterEntryPoint and a deployment for TaskManagers. This PR is based on #6319. cc @GJL ## Verifying this change - Tested manually ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (README) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink containerEntrypoint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6320.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6320 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa Author: Till Rohrmann Date: 2018-07-10T09:24:26Z [FLINK-9819] Add startup scripts for standalone job cluster entry point commit 75fb8125e3ec270994628f46457b281cdb587874 Author: Till Rohrmann Date: 2018-07-10T21:23:59Z [FLINK-9820] Forward dynamic properties to Flink configuration in ClusterEntrypoint With this commit we can use dynamic properties to overwrite configuration values in the ClusterEntrypoint. commit b38683205961c625e8c99eff1552ef5a8142ee89 Author: Till Rohrmann Date: 2018-07-10T21:43:34Z [FLINK-9821] Forward dynamic properties to configuration in TaskManagerRunner With this commit we can use dynamic properties to overwrite configuration values in the TaskManagerRunner. commit 339a24fb2508c7f3cd041bc2cf9b15fd62980fcf Author: Till Rohrmann Date: 2018-07-10T13:41:18Z [FLINK-9822] Add Dockerfile for StandaloneJobClusterEntryPoint image This commit adds a Dockerfile for a standalone job cluster image. The image contains the Flink distribution and a specified user code jar. The entrypoint will start the StandaloneJobClusterEntryPoint with the provided job classname. commit c0f8ce88a1e5ce877add31214d9b2674acfbc90f Author: Till Rohrmann Date: 2018-07-10T22:52:08Z [FLINK-9823] Add Kubernetes deployment ymls The Kubernetes files contain a job-cluster service specification, a job specification for the StandaloneJobClusterEntryPoint and a deployment for TaskManagers. commit b32a8f4149fecf953d24bd62af56f8620b360610 Author: Till Rohrmann Date: 2018-07-11T14:13:30Z [hotfix] Support building a job image from a Flink archive Extend the flink-container/docker/build.sh script to also accept a Flink archive to build the image from. This makes it easier to build an image from one of the convenience releases. > Add Kubernetes deployment files for standalone job cluster > -- > > Key: FLINK-9823 >
[jira] [Assigned] (FLINK-9814) CsvTableSource lack of column warning
[ https://issues.apache.org/jira/browse/FLINK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned FLINK-9814: --- Assignee: vinoyang > CsvTableSource lack of column warning > - > > Key: FLINK-9814 > URL: https://issues.apache.org/jira/browse/FLINK-9814 > Project: Flink > Issue Type: Wish > Components: Table API SQL >Affects Versions: 1.5.0 >Reporter: François Lacombe >Assignee: vinoyang >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > The CsvTableSource class is built by defining expected columns to be find in > the corresponding csv file. > > It would be great to throw an Exception when the csv file doesn't have the > same structure as defined in the source. > It can be easilly checked with file header if it exists. > Is this possible ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9817) Version not correctly bumped to 5.0
[
https://issues.apache.org/jira/browse/FLINK-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540625#comment-16540625
]
ASF GitHub Bot commented on FLINK-9817:
---
NicoK commented on issue #45: [FLINK-9817] update version to 5.0, part 2
URL: https://github.com/apache/flink-shaded/pull/45#issuecomment-404303250
ah - that's how the change was done...I'll adapt the script and update this
PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Version not correctly bumped to 5.0
> ---
>
> Key: FLINK-9817
> URL: https://issues.apache.org/jira/browse/FLINK-9817
> Project: Flink
> Issue Type: Bug
> Components: flink-shaded.git
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
>
> Current {{master}} of {{flink-shaded}} only made half of the changes needed
> when updating version numbers: the suffix in each sub-module's version number
> also needs to be adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9818) Add cluster component command line parser
[
https://issues.apache.org/jira/browse/FLINK-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9818:
--
Labels: pull-request-available (was: )
> Add cluster component command line parser
> -
>
> Key: FLINK-9818
> URL: https://issues.apache.org/jira/browse/FLINK-9818
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> In order to parse command line options for the cluster components
> ({{TaskManagerRunner}}, {{ClusterEntrypoints}}), we should add a
> {{CommandLineParser}} which supports the common command line options
> ({{--configDir}}, {{--webui-port}} and dynamic properties which can override
> configuration values).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9818) Add cluster component command line parser
[
https://issues.apache.org/jira/browse/FLINK-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540744#comment-16540744
]
ASF GitHub Bot commented on FLINK-9818:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/6314
[FLINK-9818] Add cluster component command line parser
## What is the purpose of the change
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
cc @GJL
## Verifying this change
- Added `EntrypointClusterConfigurationParserFactoryTest` and
`ClusterConfigurationParserFactoryTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink commandLineParser
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6314.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6314
commit b798af824433657ca215f8598112094a23819ee0
Author: Till Rohrmann
Date: 2018-07-11T15:30:53Z
[hotfix] Make PackagedProgram(Class, String...) constructor public
commit b577425b67dd5b33a8094989f7a25145ac60d542
Author: Till Rohrmann
Date: 2018-07-11T15:41:27Z
[FLINK-9818] Add cluster component command line parser
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
> Add cluster component command line parser
> -
>
> Key: FLINK-9818
> URL: https://issues.apache.org/jira/browse/FLINK-9818
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> In order to parse command line options for the cluster components
> ({{TaskManagerRunner}}, {{ClusterEntrypoints}}), we should add a
> {{CommandLineParser}} which supports the common command line options
> ({{--configDir}}, {{--webui-port}} and dynamic properties which can override
> configuration values).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9821) Let dynamic properties overwrite configuration settings in TaskManagerRunner
Till Rohrmann created FLINK-9821:
Summary: Let dynamic properties overwrite configuration settings
in TaskManagerRunner
Key: FLINK-9821
URL: https://issues.apache.org/jira/browse/FLINK-9821
Project: Flink
Issue Type: Improvement
Components: Distributed Coordination
Affects Versions: 1.6.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.6.0
Similar to FLINK-9820 we should also allow dynamic properties to overwrite
configuration values in the {{TaskManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9822) Add Dockerfile for StandaloneJobClusterEntryPoint image
[
https://issues.apache.org/jira/browse/FLINK-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540863#comment-16540863
]
ASF GitHub Bot commented on FLINK-9822:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/6319
[FLINK-9822] Add Dockerfile for StandaloneJobClusterEntryPoint image
## What is the purpose of the change
This commit adds a Dockerfile for a standalone job cluster image. The image
contains the Flink distribution and a specified user code jar. The
entrypoint
will start the StandaloneJobClusterEntryPoint with the provided job
classname.
This PR is based on #6318.
cc @GJL
## Verifying this change
- Manually tested
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (yes)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink standaloneJobDockerfile
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6319.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6319
commit b798af824433657ca215f8598112094a23819ee0
Author: Till Rohrmann
Date: 2018-07-11T15:30:53Z
[hotfix] Make PackagedProgram(Class, String...) constructor public
commit d373b6b01ec9d5b63513718a8e6b7db87629a477
Author: Till Rohrmann
Date: 2018-07-11T15:41:27Z
[FLINK-9818] Add cluster component command line parser
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922
Author: Till Rohrmann
Date: 2018-07-09T21:54:55Z
[FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint
The StandaloneJobClusterEntryPoint is the basic entry point for containers.
It is started with
the user code jar in its classpath and the classname of the user program.
The entrypoint will
then load this user program via the classname and execute its main method.
This will generate
a JobGraph which is then used to start the MiniDispatcher.
commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa
Author: Till Rohrmann
Date: 2018-07-10T09:24:26Z
[FLINK-9819] Add startup scripts for standalone job cluster entry point
commit 75fb8125e3ec270994628f46457b281cdb587874
Author: Till Rohrmann
Date: 2018-07-10T21:23:59Z
[FLINK-9820] Forward dynamic properties to Flink configuration in
ClusterEntrypoint
With this commit we can use dynamic properties to overwrite configuration
values in the
ClusterEntrypoint.
commit b38683205961c625e8c99eff1552ef5a8142ee89
Author: Till Rohrmann
Date: 2018-07-10T21:43:34Z
[FLINK-9821] Forward dynamic properties to configuration in
TaskManagerRunner
With this commit we can use dynamic properties to overwrite configuration
values in the
TaskManagerRunner.
commit 339a24fb2508c7f3cd041bc2cf9b15fd62980fcf
Author: Till Rohrmann
Date: 2018-07-10T13:41:18Z
[FLINK-9822] Add Dockerfile for StandaloneJobClusterEntryPoint image
This commit adds a Dockerfile for a standalone job cluster image. The image
contains the Flink distribution and a specified user code jar. The
entrypoint
will start the StandaloneJobClusterEntryPoint with the provided job
classname.
> Add Dockerfile for StandaloneJobClusterEntryPoint image
> ---
>
> Key: FLINK-9822
> URL: https://issues.apache.org/jira/browse/FLINK-9822
> Project: Flink
> Issue Type: New Feature
> Components: Docker
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> Add a {{Dockerfile}} to create an image which contains the
> {{StandaloneJobClusterEntryPoint}} and a specified user code jar. The
> entrypoint of this image should start the {{StandaloneJobClusterEntryPoint}}
>
[jira] [Updated] (FLINK-9822) Add Dockerfile for StandaloneJobClusterEntryPoint image
[
https://issues.apache.org/jira/browse/FLINK-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9822:
--
Labels: pull-request-available (was: )
> Add Dockerfile for StandaloneJobClusterEntryPoint image
> ---
>
> Key: FLINK-9822
> URL: https://issues.apache.org/jira/browse/FLINK-9822
> Project: Flink
> Issue Type: New Feature
> Components: Docker
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> Add a {{Dockerfile}} to create an image which contains the
> {{StandaloneJobClusterEntryPoint}} and a specified user code jar. The
> entrypoint of this image should start the {{StandaloneJobClusterEntryPoint}}
> with the added user code jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6319: [FLINK-9822] Add Dockerfile for StandaloneJobClust...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6319 [FLINK-9822] Add Dockerfile for StandaloneJobClusterEntryPoint image ## What is the purpose of the change This commit adds a Dockerfile for a standalone job cluster image. The image contains the Flink distribution and a specified user code jar. The entrypoint will start the StandaloneJobClusterEntryPoint with the provided job classname. This PR is based on #6318. cc @GJL ## Verifying this change - Manually tested ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (yes) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink standaloneJobDockerfile Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6319.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6319 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa Author: Till Rohrmann Date: 2018-07-10T09:24:26Z [FLINK-9819] Add startup scripts for standalone job cluster entry point commit 75fb8125e3ec270994628f46457b281cdb587874 Author: Till Rohrmann Date: 2018-07-10T21:23:59Z [FLINK-9820] Forward dynamic properties to Flink configuration in ClusterEntrypoint With this commit we can use dynamic properties to overwrite configuration values in the ClusterEntrypoint. commit b38683205961c625e8c99eff1552ef5a8142ee89 Author: Till Rohrmann Date: 2018-07-10T21:43:34Z [FLINK-9821] Forward dynamic properties to configuration in TaskManagerRunner With this commit we can use dynamic properties to overwrite configuration values in the TaskManagerRunner. commit 339a24fb2508c7f3cd041bc2cf9b15fd62980fcf Author: Till Rohrmann Date: 2018-07-10T13:41:18Z [FLINK-9822] Add Dockerfile for StandaloneJobClusterEntryPoint image This commit adds a Dockerfile for a standalone job cluster image. The image contains the Flink distribution and a specified user code jar. The entrypoint will start the StandaloneJobClusterEntryPoint with the provided job classname. ---
[GitHub] flink issue #6281: [FLINK-9750] Add new StreamingFileSink with ResumableWrit...
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/6281 Hi @aljoscha and @StephanEwen . I have updated the PR, please have a look. ---
[jira] [Assigned] (FLINK-9488) Create common entry point for master and workers
[
https://issues.apache.org/jira/browse/FLINK-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-9488:
Assignee: Till Rohrmann
> Create common entry point for master and workers
>
>
> Key: FLINK-9488
> URL: https://issues.apache.org/jira/browse/FLINK-9488
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> To make the container setup easier, we should provide a single cluster entry
> point which uses leader election to become either the master or a worker
> which runs the {{TaskManager}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9819) Create start up scripts for the StandaloneJobClusterEntryPoint
Till Rohrmann created FLINK-9819:
Summary: Create start up scripts for the
StandaloneJobClusterEntryPoint
Key: FLINK-9819
URL: https://issues.apache.org/jira/browse/FLINK-9819
Project: Flink
Issue Type: New Feature
Components: Startup Shell Scripts
Affects Versions: 1.6.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.6.0
In order to start the {{StandaloneJobClusterEntryPoint}} we need start up
scripts in {{flink-dist}}. We should extend the {{flink-daemon.sh}} and the
{{flink-console.sh}} scripts to support this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9823) Add Kubernetes deployment files for standalone job cluster
Till Rohrmann created FLINK-9823: Summary: Add Kubernetes deployment files for standalone job cluster Key: FLINK-9823 URL: https://issues.apache.org/jira/browse/FLINK-9823 Project: Flink Issue Type: New Feature Components: Kubernetes Affects Versions: 1.6.0 Reporter: Till Rohrmann Assignee: Till Rohrmann Fix For: 1.6.0 Similar to FLINK-9822, it would be helpful for the user to have example Kubernetes deployment files to start a standalone job cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink pull request #6279: [FLINK-9706] Properly wait for termination of JobM...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/6279 ---
[jira] [Commented] (FLINK-9706) DispatcherTest#testSubmittedJobGraphListener fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-9706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540886#comment-16540886
]
ASF GitHub Bot commented on FLINK-9706:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/6279
> DispatcherTest#testSubmittedJobGraphListener fails on Travis
>
>
> Key: FLINK-9706
> URL: https://issues.apache.org/jira/browse/FLINK-9706
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination, Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chesnay Schepler
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.5.2, 1.6.0
>
>
> https://travis-ci.org/apache/flink/jobs/399331775
> {code:java}
> testSubmittedJobGraphListener(org.apache.flink.runtime.dispatcher.DispatcherTest)
> Time elapsed: 0.103 sec <<< FAILURE!
> java.lang.AssertionError:
> Expected: a collection with size <1>
> but: collection size was <0>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.runtime.dispatcher.DispatcherTest.testSubmittedJobGraphListener(DispatcherTest.java:294)
> testSubmittedJobGraphListener(org.apache.flink.runtime.dispatcher.DispatcherTest)
> Time elapsed: 0.11 sec <<< ERROR!
> org.apache.flink.runtime.util.TestingFatalErrorHandler$TestingException:
> org.apache.flink.runtime.dispatcher.DispatcherException: Could not start the
> added job b8ab3b7fa8a929bf608a5b65896a2b17
> at
> org.apache.flink.runtime.util.TestingFatalErrorHandler.rethrowError(TestingFatalErrorHandler.java:51)
> at
> org.apache.flink.runtime.dispatcher.DispatcherTest.tearDown(DispatcherTest.java:219)
> Caused by: org.apache.flink.runtime.dispatcher.DispatcherException: Could not
> start the added job b8ab3b7fa8a929bf608a5b65896a2b17
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onAddedJobGraph$28(Dispatcher.java:845)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
> at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:332)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:158)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
> at
> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.util.FlinkException: Failed to submit job
> b8ab3b7fa8a929bf608a5b65896a2b17.
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:254)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onAddedJobGraph$27(Dispatcher.java:836)
> at
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
> at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at
>
[jira] [Created] (FLINK-9824) Support IPv6 literal
Ted Yu created FLINK-9824: - Summary: Support IPv6 literal Key: FLINK-9824 URL: https://issues.apache.org/jira/browse/FLINK-9824 Project: Flink Issue Type: Bug Reporter: Ted Yu Currently we use colon as separator when parsing host and port. We should support the usage of IPv6 literals in parsing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9817) Version not correctly bumped to 5.0
[
https://issues.apache.org/jira/browse/FLINK-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540623#comment-16540623
]
ASF GitHub Bot commented on FLINK-9817:
---
zentol commented on issue #45: [FLINK-9817] update version to 5.0, part 2
URL: https://github.com/apache/flink-shaded/pull/45#issuecomment-404302844
we have to fix the `tools/releasing/update_branch_version_sh` script to
prevent this from happening again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Version not correctly bumped to 5.0
> ---
>
> Key: FLINK-9817
> URL: https://issues.apache.org/jira/browse/FLINK-9817
> Project: Flink
> Issue Type: Bug
> Components: flink-shaded.git
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
>
> Current {{master}} of {{flink-shaded}} only made half of the changes needed
> when updating version numbers: the suffix in each sub-module's version number
> also needs to be adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9792) Cannot add html tags in options description
[
https://issues.apache.org/jira/browse/FLINK-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540695#comment-16540695
]
ASF GitHub Bot commented on FLINK-9792:
---
Github user dawidwys commented on a diff in the pull request:
https://github.com/apache/flink/pull/6312#discussion_r201844991
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/description/LineBreakElement.java
---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration.description;
+
+/**
+ * Represents a line break in the {@link Description}.
+ */
+public class LineBreakElement implements BlockElement {
+
+ /**
+* Creates a line break in the description.
+*/
+ public static LineBreakElement linebreak() {
+ return new LineBreakElement();
+ }
+
+ private LineBreakElement() {
+ }
+
+ @Override
+ public String format(Formatter formatter) {
+ return formatter.format(this);
--- End diff --
It is a basic Visitor Pattern approach. This way we have a typesafe way of
implementing it. With the enum approach we would need to do casting if we have
any custom parameters e.g. in link where beside link we also want to have a
visible name for the link.
Also I would be in favour of splitting into inline and block elements to
control complexity of the structure. I don't think we should support highly
nested structures just to minimize future efforts.
> Cannot add html tags in options description
> ---
>
> Key: FLINK-9792
> URL: https://issues.apache.org/jira/browse/FLINK-9792
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.5.1, 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
>
> Right now it is impossible to add any html tags in options description,
> because all "<" and ">" are escaped. Therefore some links there do not work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6312: [FLINK-9792] Added custom Description class for Co...
Github user dawidwys commented on a diff in the pull request:
https://github.com/apache/flink/pull/6312#discussion_r201844991
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/description/LineBreakElement.java
---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration.description;
+
+/**
+ * Represents a line break in the {@link Description}.
+ */
+public class LineBreakElement implements BlockElement {
+
+ /**
+* Creates a line break in the description.
+*/
+ public static LineBreakElement linebreak() {
+ return new LineBreakElement();
+ }
+
+ private LineBreakElement() {
+ }
+
+ @Override
+ public String format(Formatter formatter) {
+ return formatter.format(this);
--- End diff --
It is a basic Visitor Pattern approach. This way we have a typesafe way of
implementing it. With the enum approach we would need to do casting if we have
any custom parameters e.g. in link where beside link we also want to have a
visible name for the link.
Also I would be in favour of splitting into inline and block elements to
control complexity of the structure. I don't think we should support highly
nested structures just to minimize future efforts.
---
[jira] [Commented] (FLINK-9692) Adapt maxRecords parameter in the getRecords call to optimize bytes read from Kinesis
[ https://issues.apache.org/jira/browse/FLINK-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540708#comment-16540708 ] ASF GitHub Bot commented on FLINK-9692: --- Github user tweise commented on the issue: https://github.com/apache/flink/pull/6300 @tzulitai can you please take a look - would be good if we can get this into v1.6.0 > Adapt maxRecords parameter in the getRecords call to optimize bytes read from > Kinesis > -- > > Key: FLINK-9692 > URL: https://issues.apache.org/jira/browse/FLINK-9692 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector >Affects Versions: 1.5.0, 1.4.2 >Reporter: Lakshmi Rao >Assignee: Lakshmi Rao >Priority: Major > Labels: performance, pull-request-available > > The Kinesis connector currently has a [constant > value|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java#L213] > set for maxRecords that it can fetch from a single Kinesis getRecords call. > However, in most realtime scenarios, the average size of the Kinesis record > (in bytes) changes depending on the situation i.e. you could be in a > transient scenario where you are reading large sized records and would hence > like to fetch fewer records in each getRecords call (so as to not exceed the > 2 Mb/sec [per shard > limit|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html] > on the getRecords call). > The idea here is to adapt the Kinesis connector to identify an average batch > size prior to making the getRecords call, so that the maxRecords parameter > can be appropriately tuned before making the call. > This feature can be behind a > [ConsumerConfigConstants|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java] > flag that defaults to false. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #6300: [FLINK-9692][Kinesis Connector] Adaptive reads from Kines...
Github user tweise commented on the issue: https://github.com/apache/flink/pull/6300 @tzulitai can you please take a look - would be good if we can get this into v1.6.0 ---
[jira] [Commented] (FLINK-9735) Potential resource leak in RocksDBStateBackend#getDbOptions
[
https://issues.apache.org/jira/browse/FLINK-9735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540754#comment-16540754
]
Ted Yu commented on FLINK-9735:
---
Short term, we should fix the leaked DBOptions instance.
> Potential resource leak in RocksDBStateBackend#getDbOptions
> ---
>
> Key: FLINK-9735
> URL: https://issues.apache.org/jira/browse/FLINK-9735
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Here is related code:
> {code}
> if (optionsFactory != null) {
> opt = optionsFactory.createDBOptions(opt);
> }
> {code}
> opt, an DBOptions instance, should be closed before being rewritten.
> getColumnOptions has similar issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9675) Avoid FileInputStream/FileOutputStream
[ https://issues.apache.org/jira/browse/FLINK-9675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-9675: -- Description: They rely on finalizers (before Java 11), which create unnecessary GC load. The alternatives, Files.newInputStream, are as easy to use and don't have this issue. was:They rely on finalizers (before Java 11), which create unnecessary GC load. The alternatives, Files.newInputStream, are as easy to use and don't have this issue. > Avoid FileInputStream/FileOutputStream > -- > > Key: FLINK-9675 > URL: https://issues.apache.org/jira/browse/FLINK-9675 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu >Assignee: zhangminglei >Priority: Minor > Labels: filesystem > > They rely on finalizers (before Java 11), which create unnecessary GC load. > The alternatives, Files.newInputStream, are as easy to use and don't have > this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9821) Let dynamic properties overwrite configuration settings in TaskManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540847#comment-16540847
]
ASF GitHub Bot commented on FLINK-9821:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/6318
[FLINK-9821] Forward dynamic properties to configuration in
TaskManagerRunner
## What is the purpose of the change
With this commit we can use dynamic properties to overwrite configuration
values in the
`TaskManagerRunner`.
This PR is based on #6317
cc @GJL
## Verifying this change
- Tested manually
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink
forwardDynamicPropertiesTaskManager
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6318.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6318
commit b798af824433657ca215f8598112094a23819ee0
Author: Till Rohrmann
Date: 2018-07-11T15:30:53Z
[hotfix] Make PackagedProgram(Class, String...) constructor public
commit d373b6b01ec9d5b63513718a8e6b7db87629a477
Author: Till Rohrmann
Date: 2018-07-11T15:41:27Z
[FLINK-9818] Add cluster component command line parser
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922
Author: Till Rohrmann
Date: 2018-07-09T21:54:55Z
[FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint
The StandaloneJobClusterEntryPoint is the basic entry point for containers.
It is started with
the user code jar in its classpath and the classname of the user program.
The entrypoint will
then load this user program via the classname and execute its main method.
This will generate
a JobGraph which is then used to start the MiniDispatcher.
commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa
Author: Till Rohrmann
Date: 2018-07-10T09:24:26Z
[FLINK-9819] Add startup scripts for standalone job cluster entry point
commit 75fb8125e3ec270994628f46457b281cdb587874
Author: Till Rohrmann
Date: 2018-07-10T21:23:59Z
[FLINK-9820] Forward dynamic properties to Flink configuration in
ClusterEntrypoint
With this commit we can use dynamic properties to overwrite configuration
values in the
ClusterEntrypoint.
commit b38683205961c625e8c99eff1552ef5a8142ee89
Author: Till Rohrmann
Date: 2018-07-10T21:43:34Z
[FLINK-9821] Forward dynamic properties to configuration in
TaskManagerRunner
With this commit we can use dynamic properties to overwrite configuration
values in the
TaskManagerRunner.
> Let dynamic properties overwrite configuration settings in TaskManagerRunner
>
>
> Key: FLINK-9821
> URL: https://issues.apache.org/jira/browse/FLINK-9821
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> Similar to FLINK-9820 we should also allow dynamic properties to overwrite
> configuration values in the {{TaskManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6318: [FLINK-9821] Forward dynamic properties to configu...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6318 [FLINK-9821] Forward dynamic properties to configuration in TaskManagerRunner ## What is the purpose of the change With this commit we can use dynamic properties to overwrite configuration values in the `TaskManagerRunner`. This PR is based on #6317 cc @GJL ## Verifying this change - Tested manually ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink forwardDynamicPropertiesTaskManager Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6318.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6318 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa Author: Till Rohrmann Date: 2018-07-10T09:24:26Z [FLINK-9819] Add startup scripts for standalone job cluster entry point commit 75fb8125e3ec270994628f46457b281cdb587874 Author: Till Rohrmann Date: 2018-07-10T21:23:59Z [FLINK-9820] Forward dynamic properties to Flink configuration in ClusterEntrypoint With this commit we can use dynamic properties to overwrite configuration values in the ClusterEntrypoint. commit b38683205961c625e8c99eff1552ef5a8142ee89 Author: Till Rohrmann Date: 2018-07-10T21:43:34Z [FLINK-9821] Forward dynamic properties to configuration in TaskManagerRunner With this commit we can use dynamic properties to overwrite configuration values in the TaskManagerRunner. ---
[jira] [Updated] (FLINK-9821) Let dynamic properties overwrite configuration settings in TaskManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9821:
--
Labels: pull-request-available (was: )
> Let dynamic properties overwrite configuration settings in TaskManagerRunner
>
>
> Key: FLINK-9821
> URL: https://issues.apache.org/jira/browse/FLINK-9821
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination
>Affects Versions: 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> Similar to FLINK-9820 we should also allow dynamic properties to overwrite
> configuration values in the {{TaskManagerRunner}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9817) Version not correctly bumped to 5.0
[
https://issues.apache.org/jira/browse/FLINK-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540617#comment-16540617
]
ASF GitHub Bot commented on FLINK-9817:
---
NicoK opened a new pull request #45: [FLINK-9817] update version to 5.0, part 2
URL: https://github.com/apache/flink-shaded/pull/45
the previous commit, i.e. 49c9fa878cab53ab76f3e4302e4f48920566a2e2, did not
adapt the sub-modules' version strings
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Version not correctly bumped to 5.0
> ---
>
> Key: FLINK-9817
> URL: https://issues.apache.org/jira/browse/FLINK-9817
> Project: Flink
> Issue Type: Bug
> Components: flink-shaded.git
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
>
> Current {{master}} of {{flink-shaded}} only made half of the changes needed
> when updating version numbers: the suffix in each sub-module's version number
> also needs to be adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6312: [FLINK-9792] Added custom Description class for Co...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6312#discussion_r201838689
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/description/LineBreakElement.java
---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration.description;
+
+/**
+ * Represents a line break in the {@link Description}.
+ */
+public class LineBreakElement implements BlockElement {
+
+ /**
+* Creates a line break in the description.
+*/
+ public static LineBreakElement linebreak() {
+ return new LineBreakElement();
+ }
+
+ private LineBreakElement() {
+ }
+
+ @Override
+ public String format(Formatter formatter) {
+ return formatter.format(this);
--- End diff --
hmm..
Alright so this works but I still think it's kinda weird that the element
calls into the formatter; this effectively gives the element control over how
it is formatted, since it _could_ just return an arbitrary string.
Let me sketch out an alternative, the general idea being to generalize the
parent interface (DescriptionElement) to accommodate all sub-classes, and
introduce an enum for categorization.
With this the elements are just a container for data, and most (but not all
logic unfortunately unless we go for instanceof checks) formatting logic is in
the formatter.
```
public enum ElementType {
TEXT,
LINK,
LIST,
LINE_BREAK,
SEQUENCE // replaces your nested Text constructor
}
public interface Element {
String getValue()
List getChildren()
ElementType getType()
}
class HtmlFormatter implements Formatter {
format(Description description) {
description.getElements().stream()
.forEach(this::format)
.collect(Collectors.joining())
}
String format(Element element) {
switch (element.getType()) {
case TEXT:
return element.getValue()
case LIST:
StringBuilder sb = new StringBuilder()
for (Element item : element.getChildren()) {
sb.append()
sb.append(format(item))
sb.append()
}
return sb.toString()
case LINK:
return ""
case LINE_BREAK:
reutrn ""
case SEQUENCE:
StringBuilder sb = new StringBuilder()
for (Element item : element.getChildren()) {
sb.append(format(item))
}
return sb.toString()
}
}
}
class Link implements Element {
private final String value
Link(String text) {
this.value = text
}
getValue() {
return this.value
}
getChildren() {
return Collections.emptyList()
}
getType() {
return LINK
}
}
class Text implements Element {
Text(String text) {
this.value = text
this.children = Collections.emptyList()
}
getValue() {
return this.value
}
getChildren() {
return Collections.emptyList
}
getType() {
return TEXT
}
}
class List implements Element {
private final List
[jira] [Commented] (FLINK-9792) Cannot add html tags in options description
[
https://issues.apache.org/jira/browse/FLINK-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540662#comment-16540662
]
ASF GitHub Bot commented on FLINK-9792:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6312#discussion_r201838689
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/description/LineBreakElement.java
---
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration.description;
+
+/**
+ * Represents a line break in the {@link Description}.
+ */
+public class LineBreakElement implements BlockElement {
+
+ /**
+* Creates a line break in the description.
+*/
+ public static LineBreakElement linebreak() {
+ return new LineBreakElement();
+ }
+
+ private LineBreakElement() {
+ }
+
+ @Override
+ public String format(Formatter formatter) {
+ return formatter.format(this);
--- End diff --
hmm..
Alright so this works but I still think it's kinda weird that the element
calls into the formatter; this effectively gives the element control over how
it is formatted, since it _could_ just return an arbitrary string.
Let me sketch out an alternative, the general idea being to generalize the
parent interface (DescriptionElement) to accommodate all sub-classes, and
introduce an enum for categorization.
With this the elements are just a container for data, and most (but not all
logic unfortunately unless we go for instanceof checks) formatting logic is in
the formatter.
```
public enum ElementType {
TEXT,
LINK,
LIST,
LINE_BREAK,
SEQUENCE // replaces your nested Text constructor
}
public interface Element {
String getValue()
List getChildren()
ElementType getType()
}
class HtmlFormatter implements Formatter {
format(Description description) {
description.getElements().stream()
.forEach(this::format)
.collect(Collectors.joining())
}
String format(Element element) {
switch (element.getType()) {
case TEXT:
return element.getValue()
case LIST:
StringBuilder sb = new StringBuilder()
for (Element item : element.getChildren()) {
sb.append()
sb.append(format(item))
sb.append()
}
return sb.toString()
case LINK:
return ""
case LINE_BREAK:
reutrn ""
case SEQUENCE:
StringBuilder sb = new StringBuilder()
for (Element item : element.getChildren()) {
sb.append(format(item))
}
return sb.toString()
}
}
}
class Link implements Element {
private final String value
Link(String text) {
this.value = text
}
getValue() {
return this.value
}
getChildren() {
return Collections.emptyList()
}
getType() {
return LINK
}
}
class Text implements Element {
Text(String text) {
this.value = text
this.children = Collections.emptyList()
}
getValue() {
return
[GitHub] flink pull request #6320: [FLINK-9823] Add Kubernetes deployment ymls
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6320 [FLINK-9823] Add Kubernetes deployment ymls ## What is the purpose of the change The Kubernetes files contain a job-cluster service specification, a job specification for the StandaloneJobClusterEntryPoint and a deployment for TaskManagers. This PR is based on #6319. cc @GJL ## Verifying this change - Tested manually ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (README) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink containerEntrypoint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6320.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6320 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. commit 3b78e4099de1511bbe52c203fd2d05e5cfa03efa Author: Till Rohrmann Date: 2018-07-10T09:24:26Z [FLINK-9819] Add startup scripts for standalone job cluster entry point commit 75fb8125e3ec270994628f46457b281cdb587874 Author: Till Rohrmann Date: 2018-07-10T21:23:59Z [FLINK-9820] Forward dynamic properties to Flink configuration in ClusterEntrypoint With this commit we can use dynamic properties to overwrite configuration values in the ClusterEntrypoint. commit b38683205961c625e8c99eff1552ef5a8142ee89 Author: Till Rohrmann Date: 2018-07-10T21:43:34Z [FLINK-9821] Forward dynamic properties to configuration in TaskManagerRunner With this commit we can use dynamic properties to overwrite configuration values in the TaskManagerRunner. commit 339a24fb2508c7f3cd041bc2cf9b15fd62980fcf Author: Till Rohrmann Date: 2018-07-10T13:41:18Z [FLINK-9822] Add Dockerfile for StandaloneJobClusterEntryPoint image This commit adds a Dockerfile for a standalone job cluster image. The image contains the Flink distribution and a specified user code jar. The entrypoint will start the StandaloneJobClusterEntryPoint with the provided job classname. commit c0f8ce88a1e5ce877add31214d9b2674acfbc90f Author: Till Rohrmann Date: 2018-07-10T22:52:08Z [FLINK-9823] Add Kubernetes deployment ymls The Kubernetes files contain a job-cluster service specification, a job specification for the StandaloneJobClusterEntryPoint and a deployment for TaskManagers. commit b32a8f4149fecf953d24bd62af56f8620b360610 Author: Till Rohrmann Date: 2018-07-11T14:13:30Z [hotfix] Support building a job image from a Flink archive Extend the flink-container/docker/build.sh script to also accept a Flink archive to build the image from. This makes it easier to build an image from one of the convenience releases. ---
[jira] [Assigned] (FLINK-9806) Add a canonical link element to documentation HTML
[ https://issues.apache.org/jira/browse/FLINK-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned FLINK-9806: --- Assignee: vinoyang > Add a canonical link element to documentation HTML > -- > > Key: FLINK-9806 > URL: https://issues.apache.org/jira/browse/FLINK-9806 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.5.0 >Reporter: Patrick Lucas >Assignee: vinoyang >Priority: Major > > Flink has suffered for a while with non-optimal SEO for its documentation, > meaning a web search for a topic covered in the documentation often produces > results for many versions of Flink, even preferring older versions since > those pages have been around for longer. > Using a canonical link element (see references) may alleviate this by > informing search engines about where to find the latest documentation (i.e. > pages hosted under [https://ci.apache.org/projects/flink/flink-docs-master/).] > I think this is at least worth experimenting with, and if it doesn't cause > problems, even backporting it to the older release branches to eventually > clean up the Flink docs' SEO and converge on advertising only the latest docs > (unless a specific version is specified). > References: > * [https://moz.com/learn/seo/canonicalization] > * [https://yoast.com/rel-canonical/] > * [https://support.google.com/webmasters/answer/139066?hl=en] > * [https://en.wikipedia.org/wiki/Canonical_link_element] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9750) Create new StreamingFileSink that works on Flink's FileSystem abstraction
[ https://issues.apache.org/jira/browse/FLINK-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540654#comment-16540654 ] ASF GitHub Bot commented on FLINK-9750: --- Github user kl0u commented on the issue: https://github.com/apache/flink/pull/6281 Hi @aljoscha and @StephanEwen . I have updated the PR, please have a look. > Create new StreamingFileSink that works on Flink's FileSystem abstraction > - > > Key: FLINK-9750 > URL: https://issues.apache.org/jira/browse/FLINK-9750 > Project: Flink > Issue Type: Sub-task > Components: Streaming Connectors >Reporter: Stephan Ewen >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > > Using Flink's own file system abstraction means that we can add additional > streaming/checkpointing related behavior. > In addition, the new StreamingFileSink should only rely on internal > checkpointed state what files are possibly in progress or need to roll over, > never assume enumeration of files in the file system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9818) Add cluster component command line parser
Till Rohrmann created FLINK-9818:
Summary: Add cluster component command line parser
Key: FLINK-9818
URL: https://issues.apache.org/jira/browse/FLINK-9818
Project: Flink
Issue Type: Improvement
Components: Distributed Coordination
Affects Versions: 1.6.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.6.0
In order to parse command line options for the cluster components
({{TaskManagerRunner}}, {{ClusterEntrypoints}}), we should add a
{{CommandLineParser}} which supports the common command line options
({{--configDir}}, {{--webui-port}} and dynamic properties which can override
configuration values).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #6315: [FLINK-9488] Add container entry point StandaloneJ...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/6315 [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint ## What is the purpose of the change The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. This PR creates a new `flink-container` module. cc @GJL ## Verifying this change - Added `StandaloneJobClusterEntryPointTest` and `StandaloneClusterConfigurationParserFactoryTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink standaloneJobClusterEntryPoint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6315 commit b798af824433657ca215f8598112094a23819ee0 Author: Till Rohrmann Date: 2018-07-11T15:30:53Z [hotfix] Make PackagedProgram(Class, String...) constructor public commit d373b6b01ec9d5b63513718a8e6b7db87629a477 Author: Till Rohrmann Date: 2018-07-11T15:41:27Z [FLINK-9818] Add cluster component command line parser The cluster component command line parser is responsible for parsing the common command line arguments with which the cluster components are started. These include the configDir, webui-port and dynamic properties. commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922 Author: Till Rohrmann Date: 2018-07-09T21:54:55Z [FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint The StandaloneJobClusterEntryPoint is the basic entry point for containers. It is started with the user code jar in its classpath and the classname of the user program. The entrypoint will then load this user program via the classname and execute its main method. This will generate a JobGraph which is then used to start the MiniDispatcher. ---
[jira] [Updated] (FLINK-9488) Create common entry point for master and workers
[
https://issues.apache.org/jira/browse/FLINK-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9488:
--
Labels: pull-request-available (was: )
> Create common entry point for master and workers
>
>
> Key: FLINK-9488
> URL: https://issues.apache.org/jira/browse/FLINK-9488
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> To make the container setup easier, we should provide a single cluster entry
> point which uses leader election to become either the master or a worker
> which runs the {{TaskManager}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9488) Create common entry point for master and workers
[
https://issues.apache.org/jira/browse/FLINK-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540771#comment-16540771
]
ASF GitHub Bot commented on FLINK-9488:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/6315
[FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint
## What is the purpose of the change
The StandaloneJobClusterEntryPoint is the basic entry point for containers.
It is started with
the user code jar in its classpath and the classname of the user program.
The entrypoint will
then load this user program via the classname and execute its main method.
This will generate
a JobGraph which is then used to start the MiniDispatcher.
This PR creates a new `flink-container` module.
cc @GJL
## Verifying this change
- Added `StandaloneJobClusterEntryPointTest` and
`StandaloneClusterConfigurationParserFactoryTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink
standaloneJobClusterEntryPoint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6315.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6315
commit b798af824433657ca215f8598112094a23819ee0
Author: Till Rohrmann
Date: 2018-07-11T15:30:53Z
[hotfix] Make PackagedProgram(Class, String...) constructor public
commit d373b6b01ec9d5b63513718a8e6b7db87629a477
Author: Till Rohrmann
Date: 2018-07-11T15:41:27Z
[FLINK-9818] Add cluster component command line parser
The cluster component command line parser is responsible for parsing the
common command line
arguments with which the cluster components are started. These include the
configDir, webui-port
and dynamic properties.
commit 2526bcc69ff4eb3d196a4a7ceba2e59a7f455922
Author: Till Rohrmann
Date: 2018-07-09T21:54:55Z
[FLINK-9488] Add container entry point StandaloneJobClusterEntryPoint
The StandaloneJobClusterEntryPoint is the basic entry point for containers.
It is started with
the user code jar in its classpath and the classname of the user program.
The entrypoint will
then load this user program via the classname and execute its main method.
This will generate
a JobGraph which is then used to start the MiniDispatcher.
> Create common entry point for master and workers
>
>
> Key: FLINK-9488
> URL: https://issues.apache.org/jira/browse/FLINK-9488
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> To make the container setup easier, we should provide a single cluster entry
> point which uses leader election to become either the master or a worker
> which runs the {{TaskManager}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
