[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
[ https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376417#comment-15376417 ] Tzu-Li (Gordon) Tai commented on FLINK-4035: Hi [~elevy], do you have any feedback on the approach of forcing the 0.9 connector to use 0.10 jars? It'll be helpful to see how we should proceed with the PR :) > Bump Kafka producer in Kafka sink to Kafka 0.10.0.0 > --- > > Key: FLINK-4035 > URL: https://issues.apache.org/jira/browse/FLINK-4035 > Project: Flink > Issue Type: Bug > Components: Kafka Connector >Affects Versions: 1.0.3 >Reporter: Elias Levy >Priority: Minor > > Kafka 0.10.0.0 introduced protocol changes related to the producer. > Published messages now include timestamps and compressed messages now include > relative offsets. As it is now, brokers must decompress publisher compressed > messages, assign offset to them, and recompress them, which is wasteful and > makes it less likely that compression will be used at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
[ https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376410#comment-15376410 ] ASF GitHub Bot commented on FLINK-4035: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2231 Hi @radekg , thank you for opening a PR for this! From a first look it seems that there isn't much changes to the code of `flink-connector-kafka-0.9` and this PR. Also, from the original discussion / comments in the JIRA, the Kafka API doesn't seem to have changed between 0.9 and 0.10, so it might be possible to let the Kafka 0.9 connector use the 0.10 client by putting the Kafka 0.10 dependency into the user pom. May I ask whether you have tried this approach out already? Also, > At The Weather Company we bumped into a problem while trying to use Flink with Kafka 0.10.x. What was the problem? If you can describe, it'll be helpful for deciding how we can proceed with this :) There's another contributor who was trying this out, I'll also try to ask for his feedback on this in the JIRA. > Bump Kafka producer in Kafka sink to Kafka 0.10.0.0 > --- > > Key: FLINK-4035 > URL: https://issues.apache.org/jira/browse/FLINK-4035 > Project: Flink > Issue Type: Bug > Components: Kafka Connector >Affects Versions: 1.0.3 >Reporter: Elias Levy >Priority: Minor > > Kafka 0.10.0.0 introduced protocol changes related to the producer. > Published messages now include timestamps and compressed messages now include > relative offsets. As it is now, brokers must decompress publisher compressed > messages, assign offset to them, and recompress them, which is wasteful and > makes it less likely that compression will be used at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2231: [FLINK-4035] Bump Kafka producer in Kafka sink to Kafka 0...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2231 Hi @radekg , thank you for opening a PR for this! From a first look it seems that there isn't much changes to the code of `flink-connector-kafka-0.9` and this PR. Also, from the original discussion / comments in the JIRA, the Kafka API doesn't seem to have changed between 0.9 and 0.10, so it might be possible to let the Kafka 0.9 connector use the 0.10 client by putting the Kafka 0.10 dependency into the user pom. May I ask whether you have tried this approach out already? Also, > At The Weather Company we bumped into a problem while trying to use Flink with Kafka 0.10.x. What was the problem? If you can describe, it'll be helpful for deciding how we can proceed with this :) There's another contributor who was trying this out, I'll also try to ask for his feedback on this in the JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3034) Redis SInk Connector
[ https://issues.apache.org/jira/browse/FLINK-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376320#comment-15376320 ] ASF GitHub Bot commented on FLINK-3034: --- Github user subhankarb commented on the issue: https://github.com/apache/flink/pull/1813 Hi @kl0u, IMO that is the expected behavior. The sink would not know that if the Redis is down or not unless it tries to send the next data to the Redis. When ever a new message reaches the sink it tries to use the connection pool, then an then only it can throw exception that it can not send the data to Redis. You can build a heartbeat mechanism to check periodically that Redis serve is up or down, and can cancel the job if Redis is down. @mjsax please correct me if my understanding is wrong. > Redis SInk Connector > > > Key: FLINK-3034 > URL: https://issues.apache.org/jira/browse/FLINK-3034 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: Matthias J. Sax >Assignee: Subhankar Biswas >Priority: Minor > Fix For: 1.1.0 > > > Flink does not provide a sink connector for Redis. > See FLINK-3033 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #1813: [FLINK-3034] Redis Sink Connector
Github user subhankarb commented on the issue: https://github.com/apache/flink/pull/1813 Hi @kl0u, IMO that is the expected behavior. The sink would not know that if the Redis is down or not unless it tries to send the next data to the Redis. When ever a new message reaches the sink it tries to use the connection pool, then an then only it can throw exception that it can not send the data to Redis. You can build a heartbeat mechanism to check periodically that Redis serve is up or down, and can cancel the job if Redis is down. @mjsax please correct me if my understanding is wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-4214) JobExceptionsHandler will return all exceptions
[ https://issues.apache.org/jira/browse/FLINK-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla updated FLINK-4214: Priority: Minor (was: Major) > JobExceptionsHandler will return all exceptions > --- > > Key: FLINK-4214 > URL: https://issues.apache.org/jira/browse/FLINK-4214 > Project: Flink > Issue Type: Bug >Reporter: Sumit Chawla >Priority: Minor > > JobExceptionsHandler will return all exceptions and is not incrementing the > integer to track the exceptions being serialized -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4214) JobExceptionsHandler will return all exceptions
[ https://issues.apache.org/jira/browse/FLINK-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376269#comment-15376269 ] Sumit Chawla commented on FLINK-4214: - Change added in https://github.com/apache/flink/pull/2242 > JobExceptionsHandler will return all exceptions > --- > > Key: FLINK-4214 > URL: https://issues.apache.org/jira/browse/FLINK-4214 > Project: Flink > Issue Type: Bug >Reporter: Sumit Chawla > > JobExceptionsHandler will return all exceptions and is not incrementing the > integer to track the exceptions being serialized -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4214) JobExceptionsHandler will return all exceptions
Sumit Chawla created FLINK-4214: --- Summary: JobExceptionsHandler will return all exceptions Key: FLINK-4214 URL: https://issues.apache.org/jira/browse/FLINK-4214 Project: Flink Issue Type: Bug Reporter: Sumit Chawla JobExceptionsHandler will return all exceptions and is not incrementing the integer to track the exceptions being serialized -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4213) Provide CombineHint in Gelly algorithms
Greg Hogan created FLINK-4213: - Summary: Provide CombineHint in Gelly algorithms Key: FLINK-4213 URL: https://issues.apache.org/jira/browse/FLINK-4213 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 1.1.0 Reporter: Greg Hogan Assignee: Greg Hogan Many graph algorithms will see better {{reduce}} performance with the hash-combine compared with the still default sort-combine, e.g. HITS and LocalClusteringCoefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-3477) Add hash-based combine strategy for ReduceFunction
[ https://issues.apache.org/jira/browse/FLINK-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan closed FLINK-3477. - Resolution: Implemented Implemented in 52e191a5067322e82192314c16e70ae9e937ae2c > Add hash-based combine strategy for ReduceFunction > -- > > Key: FLINK-3477 > URL: https://issues.apache.org/jira/browse/FLINK-3477 > Project: Flink > Issue Type: Sub-task > Components: Local Runtime >Reporter: Fabian Hueske >Assignee: Gabor Gevay > > This issue is about adding a hash-based combine strategy for ReduceFunctions. > The interface of the {{reduce()}} method is as follows: > {code} > public T reduce(T v1, T v2) > {code} > Input type and output type are identical and the function returns only a > single value. A Reduce function is incrementally applied to compute a final > aggregated value. This allows to hold the preaggregated value in a hash-table > and update it with each function call. > The hash-based strategy requires special implementation of an in-memory hash > table. The hash table should support in place updates of elements (if the > updated value has the same size as the new value) but also appending updates > with invalidation of the old value (if the binary length of the new value > differs). The hash table needs to be able to evict and emit all elements if > it runs out-of-memory. > We should also add {{HASH}} and {{SORT}} compiler hints to > {{DataSet.reduce()}} and {{Grouping.reduce()}} to allow users to pick the > execution strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2246) Add chained combine driver strategy for ReduceFunction
[ https://issues.apache.org/jira/browse/FLINK-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan closed FLINK-2246. - Resolution: Implemented Implemented in 0db804b936efd8631f1a08db37753dad7f1f71ea > Add chained combine driver strategy for ReduceFunction > -- > > Key: FLINK-2246 > URL: https://issues.apache.org/jira/browse/FLINK-2246 > Project: Flink > Issue Type: Improvement > Components: Local Runtime >Affects Versions: 0.9, 0.10.0 >Reporter: Ufuk Celebi >Assignee: Gabor Gevay >Priority: Minor > > Running the WordCount example with a text file input/output results and a > manual reduce function (instead of the sum(1)) results in a combiner, which > is not chained. > Replace sum(1) with the following to reproduce and use a text file as input: > {code} > fileOutput = true; > textPath = "..."; > outputPath = "..."; > {code} > {code} > .reduce(new ReduceFunction>() { > @Override > public Tuple2 reduce(Tuple2 value1, > Tuple2 value2) throws Exception { > return new Tuple2(value1.f0, value1.f1 + value2.f1); > } > }); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3477) Add hash-based combine strategy for ReduceFunction
[ https://issues.apache.org/jira/browse/FLINK-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376126#comment-15376126 ] ASF GitHub Bot commented on FLINK-3477: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1517 > Add hash-based combine strategy for ReduceFunction > -- > > Key: FLINK-3477 > URL: https://issues.apache.org/jira/browse/FLINK-3477 > Project: Flink > Issue Type: Sub-task > Components: Local Runtime >Reporter: Fabian Hueske >Assignee: Gabor Gevay > > This issue is about adding a hash-based combine strategy for ReduceFunctions. > The interface of the {{reduce()}} method is as follows: > {code} > public T reduce(T v1, T v2) > {code} > Input type and output type are identical and the function returns only a > single value. A Reduce function is incrementally applied to compute a final > aggregated value. This allows to hold the preaggregated value in a hash-table > and update it with each function call. > The hash-based strategy requires special implementation of an in-memory hash > table. The hash table should support in place updates of elements (if the > updated value has the same size as the new value) but also appending updates > with invalidation of the old value (if the binary length of the new value > differs). The hash table needs to be able to evict and emit all elements if > it runs out-of-memory. > We should also add {{HASH}} and {{SORT}} compiler hints to > {{DataSet.reduce()}} and {{Grouping.reduce()}} to allow users to pick the > execution strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #1517: [FLINK-3477] [runtime] Add hash-based combine stra...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1517 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3874) Add a Kafka TableSink with JSON serialization
[ https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375822#comment-15375822 ] ASF GitHub Bot commented on FLINK-3874: --- Github user mushketyk commented on the issue: https://github.com/apache/flink/pull/2244 I tried to inherit TableSink trait in Java code but it seems that it is impossible to inherit traits with vars in Java, therefor I had to change class structures there somewhat. > Add a Kafka TableSink with JSON serialization > - > > Key: FLINK-3874 > URL: https://issues.apache.org/jira/browse/FLINK-3874 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Fabian Hueske >Priority: Minor > > Add a TableSink that writes JSON serialized data to Kafka. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2244: [FLINK-3874] Add a Kafka TableSink with JSON serializatio...
Github user mushketyk commented on the issue: https://github.com/apache/flink/pull/2244 I tried to inherit TableSink trait in Java code but it seems that it is impossible to inherit traits with vars in Java, therefor I had to change class structures there somewhat. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2244: Kafka json
GitHub user mushketyk opened a pull request: https://github.com/apache/flink/pull/2244 Kafka json Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/mushketyk/flink kafka-json Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2244.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2244 commit 9ff33698e44afc005360d8acb10fdbf2ccba814b Author: Ivan Mushketyk Date: 2016-07-05T21:00:18Z [FLINK-3874] Implement KafkaJsonTableSink commit 3eeb1dcd0f4febe37f92725bc94f3d3b13e3368f Author: Ivan Mushketyk Date: 2016-07-13T21:43:13Z [FLINK-3874] Implement tests for CsvTableSink --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3477) Add hash-based combine strategy for ReduceFunction
[ https://issues.apache.org/jira/browse/FLINK-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375658#comment-15375658 ] ASF GitHub Bot commented on FLINK-3477: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/1517 CI tests are passing. I've been testing Gelly algorithms with this without error. I will merge this ... > Add hash-based combine strategy for ReduceFunction > -- > > Key: FLINK-3477 > URL: https://issues.apache.org/jira/browse/FLINK-3477 > Project: Flink > Issue Type: Sub-task > Components: Local Runtime >Reporter: Fabian Hueske >Assignee: Gabor Gevay > > This issue is about adding a hash-based combine strategy for ReduceFunctions. > The interface of the {{reduce()}} method is as follows: > {code} > public T reduce(T v1, T v2) > {code} > Input type and output type are identical and the function returns only a > single value. A Reduce function is incrementally applied to compute a final > aggregated value. This allows to hold the preaggregated value in a hash-table > and update it with each function call. > The hash-based strategy requires special implementation of an in-memory hash > table. The hash table should support in place updates of elements (if the > updated value has the same size as the new value) but also appending updates > with invalidation of the old value (if the binary length of the new value > differs). The hash table needs to be able to evict and emit all elements if > it runs out-of-memory. > We should also add {{HASH}} and {{SORT}} compiler hints to > {{DataSet.reduce()}} and {{Grouping.reduce()}} to allow users to pick the > execution strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #1517: [FLINK-3477] [runtime] Add hash-based combine strategy fo...
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/1517 CI tests are passing. I've been testing Gelly algorithms with this without error. I will merge this ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4196) Remove "recoveryTimestamp"
[ https://issues.apache.org/jira/browse/FLINK-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375630#comment-15375630 ] ASF GitHub Bot commented on FLINK-4196: --- GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/2243 [FLINK-4196] [runtime] Remove recovery timestamp from checkpoint restores The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the master upon recovery. Because this timestamp cannot be relied upon in distributed setups, it is removed here. If we need something like this in the future, we should try and get a globally progress counter or logical timestamp instead. No code in the core Flink repository is affected by this change. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink remove_recovery_timestamp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2243.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2243 commit ae65ef4b8ce35aadd089be46b52e7fddb5a3ef85 Author: Stephan Ewen Date: 2016-07-05T08:18:38Z [hotfix] [kafka connector] Minor code cleanups in the Kafka Producer commit c738fcd9f6031becc405c17cc479b9c2340c2414 Author: Stephan Ewen Date: 2016-07-11T18:36:44Z [hotfix] [runtim] Minor code cleanups. commit 116321241923194e9fa6db556681b333197fceed Author: Stephan Ewen Date: 2016-07-13T15:31:35Z [FLINK-4196] [runtime] Remove the 'recoveryTimestamp' from checkpoint restores. The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the master upon recovery. This this timestamp cannot be relied upon in distributed setups, it is removed. > Remove "recoveryTimestamp" > -- > > Key: FLINK-4196 > URL: https://issues.apache.org/jira/browse/FLINK-4196 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.0.3 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > > I think we should remove the {{recoveryTimestamp}} that is attached on state > restore calls. > Given that this is a wall clock timestamp from a master node, which may > change when clocks are adjusted, and between different master nodes during > leader change, this is an unsafe concept. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2243: [FLINK-4196] [runtime] Remove recovery timestamp f...
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/2243 [FLINK-4196] [runtime] Remove recovery timestamp from checkpoint restores The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the master upon recovery. Because this timestamp cannot be relied upon in distributed setups, it is removed here. If we need something like this in the future, we should try and get a globally progress counter or logical timestamp instead. No code in the core Flink repository is affected by this change. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink remove_recovery_timestamp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2243.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2243 commit ae65ef4b8ce35aadd089be46b52e7fddb5a3ef85 Author: Stephan Ewen Date: 2016-07-05T08:18:38Z [hotfix] [kafka connector] Minor code cleanups in the Kafka Producer commit c738fcd9f6031becc405c17cc479b9c2340c2414 Author: Stephan Ewen Date: 2016-07-11T18:36:44Z [hotfix] [runtim] Minor code cleanups. commit 116321241923194e9fa6db556681b333197fceed Author: Stephan Ewen Date: 2016-07-13T15:31:35Z [FLINK-4196] [runtime] Remove the 'recoveryTimestamp' from checkpoint restores. The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the master upon recovery. This this timestamp cannot be relied upon in distributed setups, it is removed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2242: ExceptionHandler keep count of exceptions
GitHub user sumitchawla opened a pull request: https://github.com/apache/flink/pull/2242 ExceptionHandler keep count of exceptions This is just a bug identified while going through exception handler. There is no JIRA ticket for this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sumitchawla/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2242.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2242 commit 35a75cd672db8c40d591d2121f246fc1e642ae7c Author: Sumit Chawla Date: 2016-07-13T19:16:01Z increment the exceptions counter commit 2afb0b32f750c5d77ed7da8f86e7ec78da27f3d5 Author: Sumit Chawla Date: 2016-07-13T19:18:12Z Merge pull request #1 from sumitchawla/sumitchawla-exceptions-handler-fix increment the exceptions counter --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream
[ https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375530#comment-15375530 ] ASF GitHub Bot commented on FLINK-2125: --- Github user delding commented on the issue: https://github.com/apache/flink/pull/2233 Hi @zentol , thanks for your comments. I have updated this PR. Please let me know if there are further improvements that need to be done. > String delimiter for SocketTextStream > - > > Key: FLINK-2125 > URL: https://issues.apache.org/jira/browse/FLINK-2125 > Project: Flink > Issue Type: Improvement > Components: Streaming >Affects Versions: 0.9 >Reporter: Márton Balassi >Priority: Minor > Labels: starter > > The SocketTextStreamFunction uses a character delimiter, despite other parts > of the API using String delimiter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2233: [FLINK-2125][streaming] Delimiter change from char to str...
Github user delding commented on the issue: https://github.com/apache/flink/pull/2233 Hi @zentol , thanks for your comments. I have updated this PR. Please let me know if there are further improvements that need to be done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2237: [FLINK-4206][metrics] Remove alphanumeric name restrictio...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2237 Actually, I just wrote a more efficient version of a check that was there before. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4206) Metric names should allow special characters
[ https://issues.apache.org/jira/browse/FLINK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375505#comment-15375505 ] ASF GitHub Bot commented on FLINK-4206: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2237 Actually, I just wrote a more efficient version of a check that was there before. > Metric names should allow special characters > > > Key: FLINK-4206 > URL: https://issues.apache.org/jira/browse/FLINK-4206 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.1.0 > > > Currently, the name of the metric is restricted to alphanumeric characters. > This restriction was originally put in place to circumvent issues due to > systems not supporting certain characters. > However, this restriction does not make a lot of sense since for group names > we don't enforce such a restriction. > This also affects the integration of the Kafka metrics, so i suggest removing > the restriction. > From now on it will be the responsibility of the reporter to make sure that > the metric identifier is supported by the external system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2228: [FLINK-4170][kinesis-connector] Simplify Kinesis connecte...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2228 Rebased again on the new endpoint config merged in https://github.com/apache/flink/pull/2227. This PR should be ready for a final review now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4170) Remove `CONFIG_` prefix from KinesisConfigConstants variables
[ https://issues.apache.org/jira/browse/FLINK-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375464#comment-15375464 ] ASF GitHub Bot commented on FLINK-4170: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2228 Rebased again on the new endpoint config merged in https://github.com/apache/flink/pull/2227. This PR should be ready for a final review now. > Remove `CONFIG_` prefix from KinesisConfigConstants variables > - > > Key: FLINK-4170 > URL: https://issues.apache.org/jira/browse/FLINK-4170 > Project: Flink > Issue Type: Improvement > Components: Streaming Connectors >Reporter: Ufuk Celebi >Assignee: Tzu-Li (Gordon) Tai > Fix For: 1.1.0 > > > I find the static variable names verbose. I think it's clear from context > that they refer to the Kinesis configuration since they are all gathered in > that class. > Therefore would like to remove the {{CONFIG_}} prefix before the release, so > that we have > {code} > conf.put(KinesisConfigConstants.AWS_REGION, "") > {code} > instead of > {code} > conf.put(KinesisConfigConstants.CONFIG_AWS_REGION, "") > {code} > For longer variables it becomes even longer otherwise. > --- > Some basic variable names that might be accessed frequently are also very > long: > {code} > CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY > CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID > {code} > It might suffice to just have: > {code} > AWS_SECRET_KEY > AWS_ACCESS_KEY > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3466) Job might get stuck in restoreState() from HDFS due to interrupt
[ https://issues.apache.org/jira/browse/FLINK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375397#comment-15375397 ] Stephan Ewen commented on FLINK-3466: - Here is a Unit test that minimally reproduces getting stuck in interrupt sensitive state handles (like those reading from HDFS) {code} public class InterruptSensitiveRestoreTest { private static final OneShotLatch IN_RESTORE_LATCH = new OneShotLatch(); @Test public void testRestoreWithInterrupt() throws Exception { Configuration taskConfig = new Configuration(); StreamConfig cfg = new StreamConfig(taskConfig); cfg.setTimeCharacteristic(TimeCharacteristic.ProcessingTime); TaskDeploymentDescriptor tdd = createTaskDeploymentDescriptor(taskConfig, new InterruptLockingStateHandle()); Task task = createTask(tdd); // start the task and wait until it is in "restore" task.startTaskThread(); IN_RESTORE_LATCH.await(); // trigger cancellation and signal to continue task.cancelExecution(); task.getExecutingThread().join(3); if (task.getExecutionState() == ExecutionState.CANCELING) { fail("Task is stuck and not canceling"); } assertEquals(ExecutionState.CANCELED, task.getExecutionState()); assertNull(task.getFailureCause()); } // // Utilities // private static TaskDeploymentDescriptor createTaskDeploymentDescriptor( Configuration taskConfig, StateHandle state) throws IOException { return new TaskDeploymentDescriptor( new JobID(), "test job name", new JobVertexID(), new ExecutionAttemptID(), new SerializedValue<>(new ExecutionConfig()), "test task name", 0, 1, 0, new Configuration(), taskConfig, SourceStreamTask.class.getName(), Collections.emptyList(), Collections.emptyList(), Collections.emptyList(), Collections.emptyList(), 0, new SerializedValue>(state)); } private static Task createTask(TaskDeploymentDescriptor tdd) throws IOException { return new Task( tdd, mock(MemoryManager.class), mock(IOManager.class), mock(NetworkEnvironment.class), mock(BroadcastVariableManager.class), mock(ActorGateway.class), mock(ActorGateway.class), new FiniteDuration(10, TimeUnit.SECONDS), new FallbackLibraryCacheManager(), new FileCache(new Configuration()), new TaskManagerRuntimeInfo( "localhost", new Configuration(), EnvironmentInformation.getTemporaryFileDirectory()), mock(TaskMetricGroup.class)); } @SuppressWarnings("serial") private static class InterruptLockingStateHandle extends StreamTaskStateList { public InterruptLockingStateHandle() throws Exception { super(new StreamTaskState[0]); } @Override public StreamTaskState[] getState(ClassLoader userCodeClassLoader) { IN_RESTORE_LATCH.trigger(); // this mimics what happens in the HDFS client code. // an interrupt on a waiting object leads to an infinite loop try { synchronized (this) { wait(); } } catch (InterruptedException e) { while (true) { try { synchronized (this) {
[GitHub] flink issue #2237: [FLINK-4206][metrics] Remove alphanumeric name restrictio...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2237 I stumbled across this and I found the limitation quite annoying, so if there are no good reasons for the check, I agree to remove it. Since @StephanEwen wrote this, it might be good if he could confirm. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4206) Metric names should allow special characters
[ https://issues.apache.org/jira/browse/FLINK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375377#comment-15375377 ] ASF GitHub Bot commented on FLINK-4206: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2237 I stumbled across this and I found the limitation quite annoying, so if there are no good reasons for the check, I agree to remove it. Since @StephanEwen wrote this, it might be good if he could confirm. > Metric names should allow special characters > > > Key: FLINK-4206 > URL: https://issues.apache.org/jira/browse/FLINK-4206 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.1.0 > > > Currently, the name of the metric is restricted to alphanumeric characters. > This restriction was originally put in place to circumvent issues due to > systems not supporting certain characters. > However, this restriction does not make a lot of sense since for group names > we don't enforce such a restriction. > This also affects the integration of the Kafka metrics, so i suggest removing > the restriction. > From now on it will be the responsibility of the reporter to make sure that > the metric identifier is supported by the external system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2234: [hotfix][kinesis-connector] Remove duplicate info ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2234 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2234: [hotfix][kinesis-connector] Remove duplicate info in Kine...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2234 Merging ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-4197) Allow Kinesis Endpoint to be Overridden via Config
[ https://issues.apache.org/jira/browse/FLINK-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger resolved FLINK-4197. --- Resolution: Fixed Assignee: Scott Kidder Fix Version/s: (was: 1.0.4) 1.1.0 Resolved in http://git-wip-us.apache.org/repos/asf/flink/commit/bc3a96f5 Thank you for the contribution [~skidder]. > Allow Kinesis Endpoint to be Overridden via Config > -- > > Key: FLINK-4197 > URL: https://issues.apache.org/jira/browse/FLINK-4197 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector >Affects Versions: 1.0.3 >Reporter: Scott Kidder >Assignee: Scott Kidder >Priority: Minor > Labels: easyfix > Fix For: 1.1.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > I perform local testing of my application stack with Flink configured as a > consumer on a Kinesis stream provided by Kinesalite, an implementation of > Kinesis built on LevelDB. This requires me to override the AWS endpoint to > refer to my local Kinesalite server rather than reference the real AWS > endpoint. I'd like to add a configuration property to the Kinesis streaming > connector that allows the AWS endpoint to be specified explicitly. > This should be a fairly small change and provide a lot of flexibility to > people looking to integrate Flink with Kinesis in a non-production setup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2227 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4197) Allow Kinesis Endpoint to be Overridden via Config
[ https://issues.apache.org/jira/browse/FLINK-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375360#comment-15375360 ] ASF GitHub Bot commented on FLINK-4197: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2227 > Allow Kinesis Endpoint to be Overridden via Config > -- > > Key: FLINK-4197 > URL: https://issues.apache.org/jira/browse/FLINK-4197 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector >Affects Versions: 1.0.3 >Reporter: Scott Kidder >Priority: Minor > Labels: easyfix > Fix For: 1.0.4 > > Original Estimate: 1h > Remaining Estimate: 1h > > I perform local testing of my application stack with Flink configured as a > consumer on a Kinesis stream provided by Kinesalite, an implementation of > Kinesis built on LevelDB. This requires me to override the AWS endpoint to > refer to my local Kinesalite server rather than reference the real AWS > endpoint. I'd like to add a configuration property to the Kinesis streaming > connector that allows the AWS endpoint to be specified explicitly. > This should be a fairly small change and provide a lot of flexibility to > people looking to integrate Flink with Kinesis in a non-production setup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-4152) TaskManager registration exponential backoff doesn't work
[ https://issues.apache.org/jira/browse/FLINK-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375296#comment-15375296 ] Till Rohrmann edited comment on FLINK-4152 at 7/13/16 4:24 PM: --- The restarted registration attempts are the observable symptoms caused by a different problem. The actual problem is that the {{YarnFlinkRessourceManager}} forgets about the registered task managers if the job manager loses its leadership. Each task manager has a resource ID with which it registers at the resource manager. The {{YarnFlinkResourceManager}} has two states for allocated resources: {{containersInLaunch}} and {{registeredWorkers}}. A container can only go from {{containersInLaunch}} to {{registeredWorkers}}. This also works for the initial registration. However, when the job manager loses its leadership and the {{registeredWorkers}} list is cleared, there is no longer an container in launch associated with the respective resource ID. Consequently, when the old task manager is being re-registered by the new leader, the registration is rejected. This rejection is then sent to the task manager. Upon receiving a rejection, the task manager reschedules another registration attempt after waiting for some time. Here the problem is that the old registration attempts are not cancelled. Consequently, one will have multiple registration attempts taking place at the "same" time/concurrently. That's the reason why you observe many registration attempt messages in the log. I think the symptom can be fixed by cancelling all currently active registration attempts when you want to restart the registration. It is a bit unclear to me what the expected behaviour of the FlinkYarnResourceManager should be. In the {{jobManagerLostLeadership}} method where the {{registeredWorkers}} list is cleared, a comment says "all currently registered TaskManagers are put under "awaiting registration"". But there is no such state. Furthermore, I'm not sure whether registered TaskManagers have to re-register if only the job manager has failed. Thus, I see two solutions. Either not clearing {{registeredWorkers}} or introducing a new state "awaiting registration" which keeps all formerly registered task managers which can be re-registered. Maybe [~mxm] can give some input. was (Author: till.rohrmann): [~mxm]The restarted registration attempts are the observable symptoms caused by a different problem. The actual problem is that the {{YarnFlinkRessourceManager}} forgets about the registered task managers if the job manager loses its leadership. Each task manager has a resource ID with which it registers at the resource manager. The {{YarnFlinkResourceManager}} has two states for allocated resources: {{containersInLaunch}} and {{registeredWorkers}}. A container can only go from {{containersInLaunch}} to {{registeredWorkers}}. This also works for the initial registration. However, when the job manager loses its leadership and the {{registeredWorkers}} list is cleared, there is no longer an container in launch associated with the respective resource ID. Consequently, when the old task manager is being re-registered by the new leader, the registration is rejected. This rejection is then sent to the task manager. Upon receiving a rejection, the task manager reschedules another registration attempt after waiting for some time. Here the problem is that the old registration attempts are not cancelled. Consequently, one will have multiple registration attempts taking place at the "same" time/concurrently. That's the reason why you observe many registration attempt messages in the log. I think the symptom can be fixed by cancelling all currently active registration attempts when you want to restart the registration. It is a bit unclear to me what the expected behaviour of the FlinkYarnResourceManager should be. In the {{jobManagerLostLeadership}} method where the {{registeredWorkers}} list is cleared, a comment says "all currently registered TaskManagers are put under "awaiting registration"". But there is no such state. Furthermore, I'm not sure whether registered TaskManagers have to re-register if only the job manager has failed. Thus, I see two solutions. Either not clearing {{registeredWorkers}} or introducing a new state "awaiting registration" which keeps all formerly registered task managers which can be re-registered. Maybe [~mxm] can give some input. > TaskManager registration exponential backoff doesn't work > - > > Key: FLINK-4152 > URL: https://issues.apache.org/jira/browse/FLINK-4152 > Project: Flink > Issue Type: Bug > Components: Distributed Coordination, TaskManager, YARN Client >Reporter: Robert Metzger >Assignee: Till Rohrmann
[jira] [Commented] (FLINK-4152) TaskManager registration exponential backoff doesn't work
[ https://issues.apache.org/jira/browse/FLINK-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375296#comment-15375296 ] Till Rohrmann commented on FLINK-4152: -- [~mxm]The restarted registration attempts are the observable symptoms caused by a different problem. The actual problem is that the {{YarnFlinkRessourceManager}} forgets about the registered task managers if the job manager loses its leadership. Each task manager has a resource ID with which it registers at the resource manager. The {{YarnFlinkResourceManager}} has two states for allocated resources: {{containersInLaunch}} and {{registeredWorkers}}. A container can only go from {{containersInLaunch}} to {{registeredWorkers}}. This also works for the initial registration. However, when the job manager loses its leadership and the {{registeredWorkers}} list is cleared, there is no longer an container in launch associated with the respective resource ID. Consequently, when the old task manager is being re-registered by the new leader, the registration is rejected. This rejection is then sent to the task manager. Upon receiving a rejection, the task manager reschedules another registration attempt after waiting for some time. Here the problem is that the old registration attempts are not cancelled. Consequently, one will have multiple registration attempts taking place at the "same" time/concurrently. That's the reason why you observe many registration attempt messages in the log. I think the symptom can be fixed by cancelling all currently active registration attempts when you want to restart the registration. It is a bit unclear to me what the expected behaviour of the FlinkYarnResourceManager should be. In the {{jobManagerLostLeadership}} method where the {{registeredWorkers}} list is cleared, a comment says "all currently registered TaskManagers are put under "awaiting registration"". But there is no such state. Furthermore, I'm not sure whether registered TaskManagers have to re-register if only the job manager has failed. Thus, I see two solutions. Either not clearing {{registeredWorkers}} or introducing a new state "awaiting registration" which keeps all formerly registered task managers which can be re-registered. Maybe [~mxm] can give some input. > TaskManager registration exponential backoff doesn't work > - > > Key: FLINK-4152 > URL: https://issues.apache.org/jira/browse/FLINK-4152 > Project: Flink > Issue Type: Bug > Components: Distributed Coordination, TaskManager, YARN Client >Reporter: Robert Metzger >Assignee: Till Rohrmann > Attachments: logs.tgz > > > While testing Flink 1.1 I've found that the TaskManagers are logging many > messages when registering at the JobManager. > This is the log file: > https://gist.github.com/rmetzger/0cebe0419cdef4507b1e8a42e33ef294 > Its logging more than 3000 messages in less than a minute. I don't think that > this is the expected behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4212) Lock on pid file when starting daemons
Greg Hogan created FLINK-4212: - Summary: Lock on pid file when starting daemons Key: FLINK-4212 URL: https://issues.apache.org/jira/browse/FLINK-4212 Project: Flink Issue Type: Improvement Components: Startup Shell Scripts Affects Versions: 1.1.0 Reporter: Greg Hogan Assignee: Greg Hogan As noted on the mailing list (0), when multiple TaskManagers are started in parallel (using pdsh) there is a race condition on updating the pid: 1) the pid file is first read to parse the process' index, 2) the process is started, and 3) on success the daemon pid is appended to the pid file. We could use a tool such as {{flock}} to lock on the pid file while starting the Flink daemon. 0: http://mail-archives.apache.org/mod_mbox/flink-user/201607.mbox/%3CCA%2BssbKXw954Bz_sBRwP6db0FntWyGWzTyP7wJZ5nhOeQnof3kg%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 Hi, Yes, this is exactly the situation, in a previous pull request I was optimizing the flink docker image, however I found that the image used supervisord to catch and keep alive those daemons, so I wanted to remove this dependency (because it adds around 40MB to the image + python and some extra stuff). Can you give me some hints on the best way to address this ? Or how can I improve my current approach (notice that I took the start-foreground idea from zookeeper). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode
[ https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375216#comment-15375216 ] ASF GitHub Bot commented on FLINK-4208: --- Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 Hi, Yes, this is exactly the situation, in a previous pull request I was optimizing the flink docker image, however I found that the image used supervisord to catch and keep alive those daemons, so I wanted to remove this dependency (because it adds around 40MB to the image + python and some extra stuff). Can you give me some hints on the best way to address this ? Or how can I improve my current approach (notice that I took the start-foreground idea from zookeeper). > Support Running Flink processes in foreground mode > -- > > Key: FLINK-4208 > URL: https://issues.apache.org/jira/browse/FLINK-4208 > Project: Flink > Issue Type: Improvement >Reporter: Ismaël Mejía >Priority: Minor > > Flink clusters are started automatically in daemon mode, this is definitely > the default case, however if we want to start containers based on flinks, the > execution context gets lost. Running flink as foreground processes can fix > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/2239 Hi @iemejia, is the situation with Docker that if the Flink processes are started as daemons and the script returns that Docker assumes the process has terminated? Skipping the pid file might work fine for a container where one wouldn't start multiple TaskManagers but would cause logfile issues otherwise. We do need to add locking to the pid file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode
[ https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375204#comment-15375204 ] ASF GitHub Bot commented on FLINK-4208: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/2239 Hi @iemejia, is the situation with Docker that if the Flink processes are started as daemons and the script returns that Docker assumes the process has terminated? Skipping the pid file might work fine for a container where one wouldn't start multiple TaskManagers but would cause logfile issues otherwise. We do need to add locking to the pid file. > Support Running Flink processes in foreground mode > -- > > Key: FLINK-4208 > URL: https://issues.apache.org/jira/browse/FLINK-4208 > Project: Flink > Issue Type: Improvement >Reporter: Ismaël Mejía >Priority: Minor > > Flink clusters are started automatically in daemon mode, this is definitely > the default case, however if we want to start containers based on flinks, the > execution context gets lost. Running flink as foreground processes can fix > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4149) Fix Serialization of NFA in AbstractKeyedCEPPatternOperator
[ https://issues.apache.org/jira/browse/FLINK-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375195#comment-15375195 ] ASF GitHub Bot commented on FLINK-4149: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2202 Thanks for the review @tillrohrmann! The `commons.io` was just an IDE mess-up. The `equals` I will fix. > Fix Serialization of NFA in AbstractKeyedCEPPatternOperator > --- > > Key: FLINK-4149 > URL: https://issues.apache.org/jira/browse/FLINK-4149 > Project: Flink > Issue Type: Bug > Components: CEP >Affects Versions: 1.0.0, 1.1.0, 1.0.1, 1.0.2, 1.0.3 >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Fix For: 1.1.0 > > > A job that uses CEP fails upon restore with a {{NullPointerException}} in > {{NFA.process()}}. The reason seems to be that field {{computationStates}} is > {{null}}. This field is transient and read in a custom {{readObject()}} > method. > In {{AbstractKeyedCEPPatternOperator}} this snipped is used to construct a > {{StateDescriptor}} for an {{NFA}} state: > {code} > new ValueStateDescriptor>( > NFA_OPERATOR_STATE_NAME, > new KryoSerializer>((Class>) (Class) NFA.class, > getExecutionConfig()), > null) > {code} > It seems Kryo does not invoke {{readObject}}/{{writeObject}}. We probably > need a custom {{TypeSerializer}} for {{NFA}} to solve the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2202: [FLINK-4149] Fix Serialization of NFA in AbstractKeyedCEP...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2202 Thanks for the review @tillrohrmann! The `commons.io` was just an IDE mess-up. The `equals` I will fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-4152) TaskManager registration exponential backoff doesn't work
[ https://issues.apache.org/jira/browse/FLINK-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann reassigned FLINK-4152: Assignee: Till Rohrmann > TaskManager registration exponential backoff doesn't work > - > > Key: FLINK-4152 > URL: https://issues.apache.org/jira/browse/FLINK-4152 > Project: Flink > Issue Type: Bug > Components: Distributed Coordination, TaskManager, YARN Client >Reporter: Robert Metzger >Assignee: Till Rohrmann > Attachments: logs.tgz > > > While testing Flink 1.1 I've found that the TaskManagers are logging many > messages when registering at the JobManager. > This is the log file: > https://gist.github.com/rmetzger/0cebe0419cdef4507b1e8a42e33ef294 > Its logging more than 3000 messages in less than a minute. I don't think that > this is the expected behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375178#comment-15375178 ] ASF GitHub Bot commented on FLINK-4167: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2210 > TaskMetricGroup does not close IOMetricGroup > > > Key: FLINK-4167 > URL: https://issues.apache.org/jira/browse/FLINK-4167 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This > causes that metrics registered under the {{ioMetrics}} are not deregistered > after the termination of a job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2220: [FLINK-4184] [metrics] Replace invalid characters in Sche...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2220 I rebased on the latest master and introduced a `CharacterFilter` interface. The `CharacterFilter` allows to filter out invalid characters while generating the fully qualified metric name. In order to do this, the `AbstractMetricGroup#generateMetricName` takes a `CharacterFilter` as argument. The `AbstractReporter` and the `ScheduledDropwizardReporter` implement this interface to filter out reporter specific characters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4184) Ganglia and GraphiteReporter report metric names with invalid characters
[ https://issues.apache.org/jira/browse/FLINK-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375162#comment-15375162 ] ASF GitHub Bot commented on FLINK-4184: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2220 I rebased on the latest master and introduced a `CharacterFilter` interface. The `CharacterFilter` allows to filter out invalid characters while generating the fully qualified metric name. In order to do this, the `AbstractMetricGroup#generateMetricName` takes a `CharacterFilter` as argument. The `AbstractReporter` and the `ScheduledDropwizardReporter` implement this interface to filter out reporter specific characters. > Ganglia and GraphiteReporter report metric names with invalid characters > > > Key: FLINK-4184 > URL: https://issues.apache.org/jira/browse/FLINK-4184 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > Flink's {{GangliaReporter}} and {{GraphiteReporter}} report metrics with > names which contain invalid characters. For example, quotes are not filtered > out which can be problematic for Ganglia. Moreover, dots are not replaced > which causes Graphite to think that an IP address is actually a scoped metric > name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-4167. Resolution: Fixed Fixed via d17fe4f636b56f2200444c59a0dead9010dfaa5d > TaskMetricGroup does not close IOMetricGroup > > > Key: FLINK-4167 > URL: https://issues.apache.org/jira/browse/FLINK-4167 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This > causes that metrics registered under the {{ioMetrics}} are not deregistered > after the termination of a job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2210: [FLINK-4167] [metrics] Close IOMetricGroup in Task...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2210 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375173#comment-15375173 ] ASF GitHub Bot commented on FLINK-4167: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2210 Merging... > TaskMetricGroup does not close IOMetricGroup > > > Key: FLINK-4167 > URL: https://issues.apache.org/jira/browse/FLINK-4167 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This > causes that metrics registered under the {{ioMetrics}} are not deregistered > after the termination of a job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2210: [FLINK-4167] [metrics] Close IOMetricGroup in TaskMetricG...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2210 Merging... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-4211) Dynamic Properties not working for jobs submitted to Yarn session
Stefan Richter created FLINK-4211: - Summary: Dynamic Properties not working for jobs submitted to Yarn session Key: FLINK-4211 URL: https://issues.apache.org/jira/browse/FLINK-4211 Project: Flink Issue Type: Bug Components: YARN Client Reporter: Stefan Richter The command line argument for dynamic properties (-D) is not working when submitting jobs to a flink session. Example: {code} bin/flink run -p 4 myJob.jar -D recovery.zookeeper.path.root=/flink/xyz {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
[ https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375102#comment-15375102 ] Radoslaw Gruchalski commented on FLINK-4035: The tests are passing. > Bump Kafka producer in Kafka sink to Kafka 0.10.0.0 > --- > > Key: FLINK-4035 > URL: https://issues.apache.org/jira/browse/FLINK-4035 > Project: Flink > Issue Type: Bug > Components: Kafka Connector >Affects Versions: 1.0.3 >Reporter: Elias Levy >Priority: Minor > > Kafka 0.10.0.0 introduced protocol changes related to the producer. > Published messages now include timestamps and compressed messages now include > relative offsets. As it is now, brokers must decompress publisher compressed > messages, assign offset to them, and recompress them, which is wasteful and > makes it less likely that compression will be used at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-4196) Remove "recoveryTimestamp"
[ https://issues.apache.org/jira/browse/FLINK-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen reassigned FLINK-4196: --- Assignee: Stephan Ewen > Remove "recoveryTimestamp" > -- > > Key: FLINK-4196 > URL: https://issues.apache.org/jira/browse/FLINK-4196 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.0.3 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > > I think we should remove the {{recoveryTimestamp}} that is attached on state > restore calls. > Given that this is a wall clock timestamp from a master node, which may > change when clocks are adjusted, and between different master nodes during > leader change, this is an unsafe concept. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3666) Remove Nephele references
[ https://issues.apache.org/jira/browse/FLINK-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375078#comment-15375078 ] ASF GitHub Bot commented on FLINK-3666: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/2241 [FLINK-3666] Remove all remaining Nephele references You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3666_bye_bye_nephele Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2241.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2241 commit f1d22ed2bfb2b4ec62a1ee86c201a40a0fa97573 Author: zentol Date: 2016-07-13T14:15:28Z [FLINK-3666] Remove all remaining Nephele references > Remove Nephele references > - > > Key: FLINK-3666 > URL: https://issues.apache.org/jira/browse/FLINK-3666 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > There still exist a few references to nephele which should be removed: > {code} > flink\docs\setup\local_setup.md: >79 $ tail log/flink-*-jobmanager-*.log >80 INFO ... - Initializing memory manager with 409 megabytes of memory >81: INFO ... - Trying to load > org.apache.flinknephele.jobmanager.scheduler.local.LocalScheduler as scheduler >82 INFO ... - Setting up web info server, using web-root directory ... >83: INFO ... - Web info server will display information about nephele > job-manager on localhost, port 8081. >84 INFO ... - Starting web info server for JobManager on port 8081 >85 ~~~ >.. > 118 $ cd flink > 119 $ bin/start-local.sh > 120: Starting Nephele job manager > 121 ~~~ > {code} > {code} > flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\TaskContext.java: >70:AbstractInvokable getOwningNepheleTask(); > {code} > {code} > flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\BatchTask.java: > 1149 * @param message The main message for the log. > 1150 * @param taskName The name of the task. > 1151: * @param parent The nephele task that contains the code > producing the message. > 1152 * > 1153 * @return The string for logging. > > 1254 */ > 1255 @SuppressWarnings("unchecked") > 1256:public static Collector initOutputs(AbstractInvokable > nepheleTask, ClassLoader cl, TaskConfig config, > 1257 > List> chainedTasksTarget, > 1258 > List> eventualOutputs, > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-3666) Remove Nephele references
[ https://issues.apache.org/jira/browse/FLINK-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-3666: --- Assignee: Chesnay Schepler > Remove Nephele references > - > > Key: FLINK-3666 > URL: https://issues.apache.org/jira/browse/FLINK-3666 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > There still exist a few references to nephele which should be removed: > {code} > flink\docs\setup\local_setup.md: >79 $ tail log/flink-*-jobmanager-*.log >80 INFO ... - Initializing memory manager with 409 megabytes of memory >81: INFO ... - Trying to load > org.apache.flinknephele.jobmanager.scheduler.local.LocalScheduler as scheduler >82 INFO ... - Setting up web info server, using web-root directory ... >83: INFO ... - Web info server will display information about nephele > job-manager on localhost, port 8081. >84 INFO ... - Starting web info server for JobManager on port 8081 >85 ~~~ >.. > 118 $ cd flink > 119 $ bin/start-local.sh > 120: Starting Nephele job manager > 121 ~~~ > {code} > {code} > flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\TaskContext.java: >70:AbstractInvokable getOwningNepheleTask(); > {code} > {code} > flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\BatchTask.java: > 1149 * @param message The main message for the log. > 1150 * @param taskName The name of the task. > 1151: * @param parent The nephele task that contains the code > producing the message. > 1152 * > 1153 * @return The string for logging. > > 1254 */ > 1255 @SuppressWarnings("unchecked") > 1256:public static Collector initOutputs(AbstractInvokable > nepheleTask, ClassLoader cl, TaskConfig config, > 1257 > List> chainedTasksTarget, > 1258 > List> eventualOutputs, > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2241: [FLINK-3666] Remove all remaining Nephele referenc...
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/2241 [FLINK-3666] Remove all remaining Nephele references You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3666_bye_bye_nephele Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2241.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2241 commit f1d22ed2bfb2b4ec62a1ee86c201a40a0fa97573 Author: zentol Date: 2016-07-13T14:15:28Z [FLINK-3666] Remove all remaining Nephele references --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 I hope this gets into 1.1.0 so I can push the changes to the docker image once this is available in the official binary distribution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4207) WindowOperator becomes very slow with allowed lateness
[ https://issues.apache.org/jira/browse/FLINK-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375072#comment-15375072 ] Aljoscha Krettek commented on FLINK-4207: - It is somehow caused by the new cleanup timers. If the code in {{registerCleanupTimer}} is commented out then it performs as before. > WindowOperator becomes very slow with allowed lateness > -- > > Key: FLINK-4207 > URL: https://issues.apache.org/jira/browse/FLINK-4207 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.1.0 >Reporter: Aljoscha Krettek >Priority: Blocker > > In this simple example the throughput (as measured by the count the window > emits) becomes very low when an allowed lateness is set: > {code} > public class WindowWordCount { > public static void main(String[] args) throws Exception { > final StreamExecutionEnvironment env = > StreamExecutionEnvironment.getExecutionEnvironment(); > > env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime); > env.setParallelism(1); > env.addSource(new InfiniteTupleSource(100_000)) > .keyBy(0) > .timeWindow(Time.seconds(3)) > .allowedLateness(Time.seconds(1)) > .reduce(new ReduceFunction Integer>>() { > @Override > public Tuple2 > reduce(Tuple2 value1, > Tuple2 > value2) throws Exception { > return Tuple2.of(value1.f0, > value1.f1 + value2.f1); > } > }) > .filter(new FilterFunction Integer>>() { > private static final long > serialVersionUID = 1L; > @Override > public boolean filter(Tuple2 Integer> value) throws Exception { > return > value.f0.startsWith("Tuple 0"); > } > }) > .print(); > // execute program > env.execute("WindowWordCount"); > } > public static class InfiniteTupleSource implements > ParallelSourceFunction> { > private static final long serialVersionUID = 1L; > private int numGroups; > public InfiniteTupleSource(int numGroups) { > this.numGroups = numGroups; > } > @Override > public void run(SourceContext> out) > throws Exception { > long index = 0; > while (true) { > Tuple2 tuple = new > Tuple2<>("Tuple " + (index % numGroups), 1); > out.collect(tuple); > index++; > } > } > @Override > public void cancel() { > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4209) Docker image breaks with multiple NICs
[ https://issues.apache.org/jira/browse/FLINK-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375058#comment-15375058 ] ASF GitHub Bot commented on FLINK-4209: --- GitHub user iemejia opened a pull request: https://github.com/apache/flink/pull/2240 [FLINK-4209] Docker image breaks with multiple NICs Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/iemejia/flink FLINK-4209 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2240.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2240 commit 1af7c871a1aa9d446436df96f4b1e2d9115aee34 Author: Ismaël Mejía Date: 2016-07-05T02:16:07Z [FLINK-4209] Change hostname resolution from IP to name This solves issues when a host has multiple IPs commit 385cb7f2ec2d629912b2a7af03f1ba1c76f49483 Author: Ismaël Mejía Date: 2016-07-05T02:17:09Z [FLINK-4209] Separate build dependencies in the docker image and remove them once it is ready commit 88102e54fde4c967831ae1cc3e4025543e5fed57 Author: Ismaël Mejía Date: 2016-07-05T07:23:46Z [FLINK-4209] Add debug information of the build steps commit b365fbd47db025dcf9d8d6d797ed7bdd85093b3f Author: Ismaël Mejía Date: 2016-07-11T15:23:17Z [FLINK-4209] Simplify docker-compose script (volumes are now local) > Docker image breaks with multiple NICs > -- > > Key: FLINK-4209 > URL: https://issues.apache.org/jira/browse/FLINK-4209 > Project: Flink > Issue Type: Improvement > Components: flink-contrib >Reporter: Ismaël Mejía >Priority: Minor > > The resolution of the host is done by IP today in the docker image scripts, > this is an issue when the system has multiple network cards, if the hostname > resolution is done by name, this is fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode
[ https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375063#comment-15375063 ] ASF GitHub Bot commented on FLINK-4208: --- Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 @aljoscha As discussed in FLINK-4118 I am doing this to support this use case and eventually remove the dependency on python and supervisord from the docker image. > Support Running Flink processes in foreground mode > -- > > Key: FLINK-4208 > URL: https://issues.apache.org/jira/browse/FLINK-4208 > Project: Flink > Issue Type: Improvement >Reporter: Ismaël Mejía >Priority: Minor > > Flink clusters are started automatically in daemon mode, this is definitely > the default case, however if we want to start containers based on flinks, the > execution context gets lost. Running flink as foreground processes can fix > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode
[ https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375068#comment-15375068 ] ASF GitHub Bot commented on FLINK-4208: --- Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 I hope this gets into 1.1.0 so I can push the changes to the docker image once this is available in the official binary distribution. > Support Running Flink processes in foreground mode > -- > > Key: FLINK-4208 > URL: https://issues.apache.org/jira/browse/FLINK-4208 > Project: Flink > Issue Type: Improvement >Reporter: Ismaël Mejía >Priority: Minor > > Flink clusters are started automatically in daemon mode, this is definitely > the default case, however if we want to start containers based on flinks, the > execution context gets lost. Running flink as foreground processes can fix > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 @aljoscha As discussed in FLINK-4118 I am doing this to support this use case and eventually remove the dependency on python and supervisord from the docker image. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4209) Docker image breaks with multiple NICs
[ https://issues.apache.org/jira/browse/FLINK-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375062#comment-15375062 ] ASF GitHub Bot commented on FLINK-4209: --- Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 I added some additional fixes for example to exclude build dependencies from the final image, and to make the docker-compose script simpler. @aljoscha This is a set of small fixes that I missed at FLINK-4118. > Docker image breaks with multiple NICs > -- > > Key: FLINK-4209 > URL: https://issues.apache.org/jira/browse/FLINK-4209 > Project: Flink > Issue Type: Improvement > Components: flink-contrib >Reporter: Ismaël Mejía >Priority: Minor > > The resolution of the host is done by IP today in the docker image scripts, > this is an issue when the system has multiple network cards, if the hostname > resolution is done by name, this is fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2240: [FLINK-4209] Docker image breaks with multiple NICs
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 I added some additional fixes for example to exclude build dependencies from the final image, and to make the docker-compose script simpler. @aljoscha This is a set of small fixes that I missed at FLINK-4118. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2240: [FLINK-4209] Docker image breaks with multiple NIC...
GitHub user iemejia opened a pull request: https://github.com/apache/flink/pull/2240 [FLINK-4209] Docker image breaks with multiple NICs Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/iemejia/flink FLINK-4209 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2240.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2240 commit 1af7c871a1aa9d446436df96f4b1e2d9115aee34 Author: Ismaël MejÃa Date: 2016-07-05T02:16:07Z [FLINK-4209] Change hostname resolution from IP to name This solves issues when a host has multiple IPs commit 385cb7f2ec2d629912b2a7af03f1ba1c76f49483 Author: Ismaël MejÃa Date: 2016-07-05T02:17:09Z [FLINK-4209] Separate build dependencies in the docker image and remove them once it is ready commit 88102e54fde4c967831ae1cc3e4025543e5fed57 Author: Ismaël MejÃa Date: 2016-07-05T07:23:46Z [FLINK-4209] Add debug information of the build steps commit b365fbd47db025dcf9d8d6d797ed7bdd85093b3f Author: Ismaël MejÃa Date: 2016-07-11T15:23:17Z [FLINK-4209] Simplify docker-compose script (volumes are now local) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode
[ https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375056#comment-15375056 ] ASF GitHub Bot commented on FLINK-4208: --- GitHub user iemejia opened a pull request: https://github.com/apache/flink/pull/2239 [FLINK-4208] Support Running Flink processes in foreground mode Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/iemejia/flink FLINK-4208 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2239.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2239 commit af9ac6cdb3f6601d6248abe82df4fd44de4453e5 Author: Ismaël Mejía Date: 2016-07-13T13:55:13Z [FLINK-4208] Add support to run flink as a foreground process commit 5a27c3ce0bfaf7cf862aba3929f7712a3897bc19 Author: Ismaël Mejía Date: 2016-07-13T13:56:13Z [FLINK-4208] Make the flink daemon a real daemon nohup ignores the HUP terminal signals, so the process is still alive even after the terminal ends > Support Running Flink processes in foreground mode > -- > > Key: FLINK-4208 > URL: https://issues.apache.org/jira/browse/FLINK-4208 > Project: Flink > Issue Type: Improvement >Reporter: Ismaël Mejía >Priority: Minor > > Flink clusters are started automatically in daemon mode, this is definitely > the default case, however if we want to start containers based on flinks, the > execution context gets lost. Running flink as foreground processes can fix > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2239: [FLINK-4208] Support Running Flink processes in fo...
GitHub user iemejia opened a pull request: https://github.com/apache/flink/pull/2239 [FLINK-4208] Support Running Flink processes in foreground mode Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/iemejia/flink FLINK-4208 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2239.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2239 commit af9ac6cdb3f6601d6248abe82df4fd44de4453e5 Author: Ismaël MejÃa Date: 2016-07-13T13:55:13Z [FLINK-4208] Add support to run flink as a foreground process commit 5a27c3ce0bfaf7cf862aba3929f7712a3897bc19 Author: Ismaël MejÃa Date: 2016-07-13T13:56:13Z [FLINK-4208] Make the flink daemon a real daemon nohup ignores the HUP terminal signals, so the process is still alive even after the terminal ends --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream
[ https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375020#comment-15375020 ] ASF GitHub Bot commented on FLINK-2125: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/2233 Only had 2 small comments, otherwise this looks good to me. > String delimiter for SocketTextStream > - > > Key: FLINK-2125 > URL: https://issues.apache.org/jira/browse/FLINK-2125 > Project: Flink > Issue Type: Improvement > Components: Streaming >Affects Versions: 0.9 >Reporter: Márton Balassi >Priority: Minor > Labels: starter > > The SocketTextStreamFunction uses a character delimiter, despite other parts > of the API using String delimiter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2233: [FLINK-2125][streaming] Delimiter change from char to str...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2233 Only had 2 small comments, otherwise this looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-4210) Move close()/isClosed() out of MetricGroup interface
Chesnay Schepler created FLINK-4210: --- Summary: Move close()/isClosed() out of MetricGroup interface Key: FLINK-4210 URL: https://issues.apache.org/jira/browse/FLINK-4210 Project: Flink Issue Type: Improvement Components: Metrics Affects Versions: 1.1.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Priority: Minor Fix For: 1.1.0 The (user-facing) MetricGroup interface currently exposes a close() and isClosed() method which generally users shouldn't need to call. They are an internal thing, and thus should be moved into the AbstractMetricGroup class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4208) Support Running Flink processes in foreground mode
Ismaël Mejía created FLINK-4208: --- Summary: Support Running Flink processes in foreground mode Key: FLINK-4208 URL: https://issues.apache.org/jira/browse/FLINK-4208 Project: Flink Issue Type: Improvement Reporter: Ismaël Mejía Priority: Minor Flink clusters are started automatically in daemon mode, this is definitely the default case, however if we want to start containers based on flinks, the execution context gets lost. Running flink as foreground processes can fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4209) Docker image breaks with multiple NICs
Ismaël Mejía created FLINK-4209: --- Summary: Docker image breaks with multiple NICs Key: FLINK-4209 URL: https://issues.apache.org/jira/browse/FLINK-4209 Project: Flink Issue Type: Improvement Components: flink-contrib Reporter: Ismaël Mejía Priority: Minor The resolution of the host is done by IP today in the docker image scripts, this is an issue when the system has multiple network cards, if the hostname resolution is done by name, this is fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2226: [FLINK-4192] - Move Metrics API to separate module
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2226 Alright, although I am curious why this wasn't an issue when we merged the metric system initially. Since you mentioned that we should/could add additional helper methods, please provide a full list of methods that you think are necessary. Otherwise this will turn into longer back-and-forth than necessary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4192) Move Metrics API to separate module
[ https://issues.apache.org/jira/browse/FLINK-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374919#comment-15374919 ] ASF GitHub Bot commented on FLINK-4192: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/2226 Alright, although I am curious why this wasn't an issue when we merged the metric system initially. Since you mentioned that we should/could add additional helper methods, please provide a full list of methods that you think are necessary. Otherwise this will turn into longer back-and-forth than necessary. > Move Metrics API to separate module > --- > > Key: FLINK-4192 > URL: https://issues.apache.org/jira/browse/FLINK-4192 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.1.0 > > > All metrics code currently resides in flink-core. If a user implements a > reporter and wants a fat jar it will now have to include the entire > flink-core module. > Instead, we could move several interfaces into a separate module. > These interfaces to move include: > * Counter, Gauge, Histogram(Statistics) > * MetricGroup > * MetricReporter, Scheduled, AbstractReporter > In addition a new MetricRegistry interface will be required as well as a > replacement for the Configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream
[ https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374911#comment-15374911 ] ASF GitHub Bot commented on FLINK-2125: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/2233#discussion_r70614809 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java --- @@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws Exception { socket.connect(new InetSocketAddress(hostname, port), CONNECTION_TIMEOUT_TIME); BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream())); - int data; - while (isRunning && (data = reader.read()) != -1) { - // check if the string is complete - if (data != delimiter) { - buffer.append((char) data); - } - else { + char[] cbuf = new char[8192]; + int byteRead; --- End diff -- This variable should be named `bytesRead` . > String delimiter for SocketTextStream > - > > Key: FLINK-2125 > URL: https://issues.apache.org/jira/browse/FLINK-2125 > Project: Flink > Issue Type: Improvement > Components: Streaming >Affects Versions: 0.9 >Reporter: Márton Balassi >Priority: Minor > Labels: starter > > The SocketTextStreamFunction uses a character delimiter, despite other parts > of the API using String delimiter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2233: [FLINK-2125][streaming] Delimiter change from char...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/2233#discussion_r70614809 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java --- @@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws Exception { socket.connect(new InetSocketAddress(hostname, port), CONNECTION_TIMEOUT_TIME); BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream())); - int data; - while (isRunning && (data = reader.read()) != -1) { - // check if the string is complete - if (data != delimiter) { - buffer.append((char) data); - } - else { + char[] cbuf = new char[8192]; + int byteRead; --- End diff -- This variable should be named `bytesRead` . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4192) Move Metrics API to separate module
[ https://issues.apache.org/jira/browse/FLINK-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374910#comment-15374910 ] ASF GitHub Bot commented on FLINK-4192: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2226 The point of using `Properties` is that users of the API can use a class they are familiar with and maybe even reuse configurations and tools they internally that work with `Properties` types. Adding yet another config type means users may have to integrate yet another config type with their own tools and utils. > Move Metrics API to separate module > --- > > Key: FLINK-4192 > URL: https://issues.apache.org/jira/browse/FLINK-4192 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.1.0 > > > All metrics code currently resides in flink-core. If a user implements a > reporter and wants a fat jar it will now have to include the entire > flink-core module. > Instead, we could move several interfaces into a separate module. > These interfaces to move include: > * Counter, Gauge, Histogram(Statistics) > * MetricGroup > * MetricReporter, Scheduled, AbstractReporter > In addition a new MetricRegistry interface will be required as well as a > replacement for the Configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374907#comment-15374907 ] ASF GitHub Bot commented on FLINK-4167: --- Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/2210#discussion_r70614548 --- Diff: flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.metrics.groups; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Gauge; +import org.apache.flink.metrics.Histogram; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.util.Preconditions; + +/** + * Metric group which forwards all registration calls to its parent metric group. + * + * @param Type of the parent metric group + */ +@Internal +public class ProxyMetricGroup implements MetricGroup { + private final P parentMetricGroup; + + public ProxyMetricGroup(P parentMetricGroup) { + this.parentMetricGroup = Preconditions.checkNotNull(parentMetricGroup); + } + + @Override + public final void close() { + parentMetricGroup.close(); --- End diff -- Hmm you're right, it is a bit tricky. You don't want to close the complete parent metric group. But I guess you would want to unregister the metrics registered by the `IOMetricGroup` if you call `IOMetricGroup#close`. However, this is not really possible at the moment. I will remove the `parent.close` call then. I agree that `close` and `isClosed` should only be used internally and not being exposed to the user. This could be a follow-up task. > TaskMetricGroup does not close IOMetricGroup > > > Key: FLINK-4167 > URL: https://issues.apache.org/jira/browse/FLINK-4167 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This > causes that metrics registered under the {{ioMetrics}} are not deregistered > after the termination of a job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2226: [FLINK-4192] - Move Metrics API to separate module
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2226 The point of using `Properties` is that users of the API can use a class they are familiar with and maybe even reuse configurations and tools they internally that work with `Properties` types. Adding yet another config type means users may have to integrate yet another config type with their own tools and utils. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2210: [FLINK-4167] [metrics] Close IOMetricGroup in Task...
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/2210#discussion_r70614548 --- Diff: flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.metrics.groups; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Gauge; +import org.apache.flink.metrics.Histogram; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.util.Preconditions; + +/** + * Metric group which forwards all registration calls to its parent metric group. + * + * @param Type of the parent metric group + */ +@Internal +public class ProxyMetricGroup implements MetricGroup { + private final P parentMetricGroup; + + public ProxyMetricGroup(P parentMetricGroup) { + this.parentMetricGroup = Preconditions.checkNotNull(parentMetricGroup); + } + + @Override + public final void close() { + parentMetricGroup.close(); --- End diff -- Hmm you're right, it is a bit tricky. You don't want to close the complete parent metric group. But I guess you would want to unregister the metrics registered by the `IOMetricGroup` if you call `IOMetricGroup#close`. However, this is not really possible at the moment. I will remove the `parent.close` call then. I agree that `close` and `isClosed` should only be used internally and not being exposed to the user. This could be a follow-up task. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2233: [FLINK-2125][streaming] Delimiter change from char...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/2233#discussion_r70614465 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java --- @@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws Exception { socket.connect(new InetSocketAddress(hostname, port), CONNECTION_TIMEOUT_TIME); BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream())); - int data; - while (isRunning && (data = reader.read()) != -1) { - // check if the string is complete - if (data != delimiter) { - buffer.append((char) data); - } - else { + char[] cbuf = new char[8192]; + int byteRead; + while (isRunning && (byteRead = reader.read(cbuf)) != -1) { + buffer.append(cbuf, 0, byteRead); + int delimPos; + while (buffer.length() >= delimiter.length() && (delimPos = buffer.indexOf(delimiter)) != -1) { + String token = buffer.substring(0, delimPos); // truncate trailing carriage return - if (delimiter == '\n' && buffer.length() > 0 && buffer.charAt(buffer.length() - 1) == '\r') { - buffer.setLength(buffer.length() - 1); + if (delimiter.equals("\n") && token.endsWith("\r")) { + token = token.substring(0, token.length() - 1); + } + if (!token.isEmpty()) { + ctx.collect(token); --- End diff -- This changes existing behavior. Empty strings are filtered out, which wasn't done before. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream
[ https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374902#comment-15374902 ] ASF GitHub Bot commented on FLINK-2125: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/2233#discussion_r70614465 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java --- @@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws Exception { socket.connect(new InetSocketAddress(hostname, port), CONNECTION_TIMEOUT_TIME); BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream())); - int data; - while (isRunning && (data = reader.read()) != -1) { - // check if the string is complete - if (data != delimiter) { - buffer.append((char) data); - } - else { + char[] cbuf = new char[8192]; + int byteRead; + while (isRunning && (byteRead = reader.read(cbuf)) != -1) { + buffer.append(cbuf, 0, byteRead); + int delimPos; + while (buffer.length() >= delimiter.length() && (delimPos = buffer.indexOf(delimiter)) != -1) { + String token = buffer.substring(0, delimPos); // truncate trailing carriage return - if (delimiter == '\n' && buffer.length() > 0 && buffer.charAt(buffer.length() - 1) == '\r') { - buffer.setLength(buffer.length() - 1); + if (delimiter.equals("\n") && token.endsWith("\r")) { + token = token.substring(0, token.length() - 1); + } + if (!token.isEmpty()) { + ctx.collect(token); --- End diff -- This changes existing behavior. Empty strings are filtered out, which wasn't done before. > String delimiter for SocketTextStream > - > > Key: FLINK-2125 > URL: https://issues.apache.org/jira/browse/FLINK-2125 > Project: Flink > Issue Type: Improvement > Components: Streaming >Affects Versions: 0.9 >Reporter: Márton Balassi >Priority: Minor > Labels: starter > > The SocketTextStreamFunction uses a character delimiter, despite other parts > of the API using String delimiter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-4207) WindowOperator becomes very slow with allowed lateness
[ https://issues.apache.org/jira/browse/FLINK-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374897#comment-15374897 ] Stephan Ewen edited comment on FLINK-4207 at 7/13/16 12:20 PM: --- Is the overhead in the - "onElement()" processing? - the window triggering? - the window cleanups? was (Author: stephanewen): What is happening different now than before? > WindowOperator becomes very slow with allowed lateness > -- > > Key: FLINK-4207 > URL: https://issues.apache.org/jira/browse/FLINK-4207 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.1.0 >Reporter: Aljoscha Krettek >Priority: Blocker > > In this simple example the throughput (as measured by the count the window > emits) becomes very low when an allowed lateness is set: > {code} > public class WindowWordCount { > public static void main(String[] args) throws Exception { > final StreamExecutionEnvironment env = > StreamExecutionEnvironment.getExecutionEnvironment(); > > env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime); > env.setParallelism(1); > env.addSource(new InfiniteTupleSource(100_000)) > .keyBy(0) > .timeWindow(Time.seconds(3)) > .allowedLateness(Time.seconds(1)) > .reduce(new ReduceFunction Integer>>() { > @Override > public Tuple2 > reduce(Tuple2 value1, > Tuple2 > value2) throws Exception { > return Tuple2.of(value1.f0, > value1.f1 + value2.f1); > } > }) > .filter(new FilterFunction Integer>>() { > private static final long > serialVersionUID = 1L; > @Override > public boolean filter(Tuple2 Integer> value) throws Exception { > return > value.f0.startsWith("Tuple 0"); > } > }) > .print(); > // execute program > env.execute("WindowWordCount"); > } > public static class InfiniteTupleSource implements > ParallelSourceFunction> { > private static final long serialVersionUID = 1L; > private int numGroups; > public InfiniteTupleSource(int numGroups) { > this.numGroups = numGroups; > } > @Override > public void run(SourceContext> out) > throws Exception { > long index = 0; > while (true) { > Tuple2 tuple = new > Tuple2<>("Tuple " + (index % numGroups), 1); > out.collect(tuple); > index++; > } > } > @Override > public void cancel() { > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4207) WindowOperator becomes very slow with allowed lateness
[ https://issues.apache.org/jira/browse/FLINK-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374897#comment-15374897 ] Stephan Ewen commented on FLINK-4207: - What is happening different now than before? > WindowOperator becomes very slow with allowed lateness > -- > > Key: FLINK-4207 > URL: https://issues.apache.org/jira/browse/FLINK-4207 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.1.0 >Reporter: Aljoscha Krettek >Priority: Blocker > > In this simple example the throughput (as measured by the count the window > emits) becomes very low when an allowed lateness is set: > {code} > public class WindowWordCount { > public static void main(String[] args) throws Exception { > final StreamExecutionEnvironment env = > StreamExecutionEnvironment.getExecutionEnvironment(); > > env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime); > env.setParallelism(1); > env.addSource(new InfiniteTupleSource(100_000)) > .keyBy(0) > .timeWindow(Time.seconds(3)) > .allowedLateness(Time.seconds(1)) > .reduce(new ReduceFunction Integer>>() { > @Override > public Tuple2 > reduce(Tuple2 value1, > Tuple2 > value2) throws Exception { > return Tuple2.of(value1.f0, > value1.f1 + value2.f1); > } > }) > .filter(new FilterFunction Integer>>() { > private static final long > serialVersionUID = 1L; > @Override > public boolean filter(Tuple2 Integer> value) throws Exception { > return > value.f0.startsWith("Tuple 0"); > } > }) > .print(); > // execute program > env.execute("WindowWordCount"); > } > public static class InfiniteTupleSource implements > ParallelSourceFunction> { > private static final long serialVersionUID = 1L; > private int numGroups; > public InfiniteTupleSource(int numGroups) { > this.numGroups = numGroups; > } > @Override > public void run(SourceContext> out) > throws Exception { > long index = 0; > while (true) { > Tuple2 tuple = new > Tuple2<>("Tuple " + (index % numGroups), 1); > out.collect(tuple); > index++; > } > } > @Override > public void cancel() { > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2235: [hotfix] removed duplicated code
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2235 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3729) Several SQL tests fail on Windows OS
[ https://issues.apache.org/jira/browse/FLINK-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374889#comment-15374889 ] ASF GitHub Bot commented on FLINK-3729: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/2238 [FLINK-3729] Fix Several SQL tests on Windows OS - fixed ExplainTest by replacing all occurrences of `\r\n` with `\n` before comparing the output - replaced usages of `\n` in BatchTableEnvironment#explain() with System.lineSeparator - fixed TableSinkITCase by generating the qualidifed path using `File#toUri()` and `URI#toString()` - removed a lot of unused imports in TableSinkITCase You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3729_sql_tests_windows Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2238 commit 0694fde595289b4ae145544fb39598fee63e8bf5 Author: zentol Date: 2016-07-13T11:51:05Z [FLINK-3729][table] Fix ExplainTest on Windows OS commit d79866a78b30d6794c6c96431ac53d6a5119890d Author: zentol Date: 2016-07-13T11:51:34Z [FLINK-3729][table] Fix TableSinkITCase on Windows OS commit 31c121c25cae70262c0187e43b41a144a326ab33 Author: zentol Date: 2016-07-13T11:51:53Z [hotfix][table] Remove unused imports in TableSinkITCase > Several SQL tests fail on Windows OS > > > Key: FLINK-3729 > URL: https://issues.apache.org/jira/browse/FLINK-3729 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.0.1 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > > The Table API SqlExplain(Test/ITCase) fail categorically on Windows due to > different line-endings. These tests generate an string representation of an > abstract syntax tree; problem is there is a difference in line-endings. > The expected ones contain LF, the actual one CRLF. > The tests should be either changed to either > * include CRLF line-endings in the expected string when run on windows > * always use LF line-endings regardless of OS > * use a compare method that is aware of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2235: [hotfix] removed duplicated code
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2235 merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2238: [FLINK-3729] Fix Several SQL tests on Windows OS
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/2238 [FLINK-3729] Fix Several SQL tests on Windows OS - fixed ExplainTest by replacing all occurrences of `\r\n` with `\n` before comparing the output - replaced usages of `\n` in BatchTableEnvironment#explain() with System.lineSeparator - fixed TableSinkITCase by generating the qualidifed path using `File#toUri()` and `URI#toString()` - removed a lot of unused imports in TableSinkITCase You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3729_sql_tests_windows Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2238 commit 0694fde595289b4ae145544fb39598fee63e8bf5 Author: zentol Date: 2016-07-13T11:51:05Z [FLINK-3729][table] Fix ExplainTest on Windows OS commit d79866a78b30d6794c6c96431ac53d6a5119890d Author: zentol Date: 2016-07-13T11:51:34Z [FLINK-3729][table] Fix TableSinkITCase on Windows OS commit 31c121c25cae70262c0187e43b41a144a326ab33 Author: zentol Date: 2016-07-13T11:51:53Z [hotfix][table] Remove unused imports in TableSinkITCase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4204) Clean up gelly-examples
[ https://issues.apache.org/jira/browse/FLINK-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374877#comment-15374877 ] Greg Hogan commented on FLINK-4204: --- We should also add a {{Main-Class}} to {{flink-gelly-examples}} to print usage for running the drivers and example programs. Currently the only means to discover the available classes are to read the source or list classes in the jar. > Clean up gelly-examples > --- > > Key: FLINK-4204 > URL: https://issues.apache.org/jira/browse/FLINK-4204 > Project: Flink > Issue Type: Improvement > Components: Gelly >Affects Versions: 1.1.0 >Reporter: Vasia Kalavri > > The gelly-examples has grown quite big (14 examples) and contains several > examples that illustrate the same functionality. Examples should help users > understand how to use the API and ideally show how to use 1-2 features. > Also, it is helpful to state the purpose of each example in the comments. > We should keep the example set small and move everything that does not fit > there to the library. > I propose to remove the following: > - ClusteringCoefficient: the functionality already exists as a library method. > - HITS: the functionality already exists as a library method. > - JaccardIndex: the functionality already exists as a library method. > - SingleSourceShortestPaths: the example shows how to use scatter-gather > iterations. HITSAlgorithm shows the same feature plus the use of aggregators. > I propose we keep this one instead. > - TriangleListing: the functionality already exists as a library method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-3729) Several SQL tests fail on Windows OS
[ https://issues.apache.org/jira/browse/FLINK-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-3729: --- Assignee: Chesnay Schepler > Several SQL tests fail on Windows OS > > > Key: FLINK-3729 > URL: https://issues.apache.org/jira/browse/FLINK-3729 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.0.1 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > > The Table API SqlExplain(Test/ITCase) fail categorically on Windows due to > different line-endings. These tests generate an string representation of an > abstract syntax tree; problem is there is a difference in line-endings. > The expected ones contain LF, the actual one CRLF. > The tests should be either changed to either > * include CRLF line-endings in the expected string when run on windows > * always use LF line-endings regardless of OS > * use a compare method that is aware of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374776#comment-15374776 ] ASF GitHub Bot commented on FLINK-4167: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/2210 Only had a small comment, otherwise +1. > TaskMetricGroup does not close IOMetricGroup > > > Key: FLINK-4167 > URL: https://issues.apache.org/jira/browse/FLINK-4167 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This > causes that metrics registered under the {{ioMetrics}} are not deregistered > after the termination of a job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2210: [FLINK-4167] [metrics] Close IOMetricGroup in TaskMetricG...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2210 Only had a small comment, otherwise +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup
[ https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374764#comment-15374764 ] ASF GitHub Bot commented on FLINK-4167: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/2210#discussion_r70597751 --- Diff: flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.metrics.groups; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Gauge; +import org.apache.flink.metrics.Histogram; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.util.Preconditions; + +/** + * Metric group which forwards all registration calls to its parent metric group. + * + * @param Type of the parent metric group + */ +@Internal +public class ProxyMetricGroup implements MetricGroup { + private final P parentMetricGroup; + + public ProxyMetricGroup(P parentMetricGroup) { + this.parentMetricGroup = Preconditions.checkNotNull(parentMetricGroup); + } + + @Override + public final void close() { + parentMetricGroup.close(); --- End diff -- It is safer to not do anything here. `close()` should only be called by the Task on the `TaskMetricGroup`, this would open us to the possibility of components closing the TaskMG as well. There's also the looming StackOverflow when someone puts `ioMetrics.close()` into the `TaskMetricGroup#close()`. Now that i think about it i believe `close()` (and by extension, `isClosed()`) has no business being in the MetricGroup interface in the first place, as users actually don't need to call it. > TaskMetricGroup does not close IOMetricGroup > > > Key: FLINK-4167 > URL: https://issues.apache.org/jira/browse/FLINK-4167 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This > causes that metrics registered under the {{ioMetrics}} are not deregistered > after the termination of a job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2210: [FLINK-4167] [metrics] Close IOMetricGroup in Task...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/2210#discussion_r70597751 --- Diff: flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.metrics.groups; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Gauge; +import org.apache.flink.metrics.Histogram; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.util.Preconditions; + +/** + * Metric group which forwards all registration calls to its parent metric group. + * + * @param Type of the parent metric group + */ +@Internal +public class ProxyMetricGroup implements MetricGroup { + private final P parentMetricGroup; + + public ProxyMetricGroup(P parentMetricGroup) { + this.parentMetricGroup = Preconditions.checkNotNull(parentMetricGroup); + } + + @Override + public final void close() { + parentMetricGroup.close(); --- End diff -- It is safer to not do anything here. `close()` should only be called by the Task on the `TaskMetricGroup`, this would open us to the possibility of components closing the TaskMG as well. There's also the looming StackOverflow when someone puts `ioMetrics.close()` into the `TaskMetricGroup#close()`. Now that i think about it i believe `close()` (and by extension, `isClosed()`) has no business being in the MetricGroup interface in the first place, as users actually don't need to call it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4184) Ganglia and GraphiteReporter report metric names with invalid characters
[ https://issues.apache.org/jira/browse/FLINK-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374763#comment-15374763 ] ASF GitHub Bot commented on FLINK-4184: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2220 True, it conflicts with your proposed changes for the definable metric group delimiter. I will rebase and adapt this PR wrt #2219. > Ganglia and GraphiteReporter report metric names with invalid characters > > > Key: FLINK-4184 > URL: https://issues.apache.org/jira/browse/FLINK-4184 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann > Fix For: 1.1.0 > > > Flink's {{GangliaReporter}} and {{GraphiteReporter}} report metrics with > names which contain invalid characters. For example, quotes are not filtered > out which can be problematic for Ganglia. Moreover, dots are not replaced > which causes Graphite to think that an IP address is actually a scoped metric > name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2220: [FLINK-4184] [metrics] Replace invalid characters in Sche...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2220 True, it conflicts with your proposed changes for the definable metric group delimiter. I will rebase and adapt this PR wrt #2219. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4143) Configurable delimiter for metric identifier
[ https://issues.apache.org/jira/browse/FLINK-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374733#comment-15374733 ] ASF GitHub Bot commented on FLINK-4143: --- Github user zentol closed the pull request at: https://github.com/apache/flink/pull/2219 > Configurable delimiter for metric identifier > > > Key: FLINK-4143 > URL: https://issues.apache.org/jira/browse/FLINK-4143 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > > The metric identifier is currently hard-coded to separate components with a > dot. > We should make this configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4143) Configurable delimiter for metric identifier
[ https://issues.apache.org/jira/browse/FLINK-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-4143. --- Resolution: Fixed Implemented in 790a654c5e08e0e54f3e02499be4dd8c4006227a > Configurable delimiter for metric identifier > > > Key: FLINK-4143 > URL: https://issues.apache.org/jira/browse/FLINK-4143 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > > The metric identifier is currently hard-coded to separate components with a > dot. > We should make this configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)