[GitHub] flink pull request #4883: [FLINK-4809] Operators should tolerate checkpoint ...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4883#discussion_r182155763 --- Diff: docs/dev/stream/state/checkpointing.md --- @@ -118,6 +120,9 @@ env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500) // checkpoints have to complete within one minute, or are discarded env.getCheckpointConfig.setCheckpointTimeout(6) +// prevent the tasks from failing if an error happens in their checkpointing, the checkpoint will just be declined. +env.getCheckpointConfig.setFailTasksOnCheckpointingErrors(false) --- End diff -- This line is missing from the Java tab. ---
[GitHub] flink pull request #5293: [hotfix][docs] Mention maven dependency for RocksD...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/5293 [hotfix][docs] Mention maven dependency for RocksDB state backend This hotfix adds a note to the RocksDB state backend page, mentioning that it requires a maven dependency to be added to the users' project. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink docs_rocks_maven Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5293 commit e20ac587de96e6602b564077663500a42d6f4fec Author: Robert Metzger <rmetzger@...> Date: 2018-01-13T11:06:58Z [hotfix][docs] Mention maven dependency for RocksDB state backend ---
[GitHub] flink issue #4616: [FLINK-7552] [FLINK-7553] Enhance SinkInterface / Use in ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4616 No, you can define fine-grained exclusions. Check the excludes in the example: http://siom79.github.io/japicmp/MavenPlugin.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4616: [FLINK-7552] [FLINK-7553] Enhance SinkInterface / Use in ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4616 I see. It is a similar problem we've discussed with the SourceContext. Users only implement it, but don't call it. I believe in this case, it makes sense to add an exception to Japicmp to allow this change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4522: [FLINK-7366][kinesis connector] Upgrade kinesis producer ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4522 I've pushed the PR to CI again. If it passes, I'll merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4566: [FLINK-7477] [FLINK-7480] Various improvements to Flink s...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4566 +1 to merge (assuming `command` is available on all operating systems / or is a bash command?) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4566: [FLINK-7477] [FLINK-7480] Various improvements to ...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/4566#discussion_r133985865
--- Diff: flink-dist/src/main/flink-bin/bin/config.sh ---
@@ -351,8 +351,20 @@ if [ -z "$HADOOP_CONF_DIR" ]; then
fi
fi
+# try and set HADOOP_CONF_DIR to some common default if it's not set
+if [ -z "$HADOOP_CONF_DIR" ]; then
+if [ -d "/etc/hadoop/conf" ]; then
+HADOOP_CONF_DIR="/etc/hadoop/conf"
+fi
+fi
+
INTERNAL_HADOOP_CLASSPATHS="${HADOOP_CLASSPATH}:${HADOOP_CONF_DIR}:${YARN_CONF_DIR}"
+# check if the "hadoop" binary is available, if yes, use that to augment
the CLASSPATH
+if command -v hadoop >/dev/null 2>&1; then
+
INTERNAL_HADOOP_CLASSPATHS="${HADOOP_CLASSPATH}:${HADOOP_CONF_DIR}:${YARN_CONF_DIR}:`hadoop
classpath`"
--- End diff --
I would actually append `INTERNAL_HADOOP_CLASSPATHS` instead of overwriting
it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4566: [FLINK-7477] [FLINK-7480] Various improvements to ...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4566#discussion_r133985612 --- Diff: flink-dist/src/main/flink-bin/bin/config.sh --- @@ -351,8 +351,20 @@ if [ -z "$HADOOP_CONF_DIR" ]; then fi fi +# try and set HADOOP_CONF_DIR to some common default if it's not set +if [ -z "$HADOOP_CONF_DIR" ]; then +if [ -d "/etc/hadoop/conf" ]; then +HADOOP_CONF_DIR="/etc/hadoop/conf" --- End diff -- I would suggest to print a message to the user that we are using this HADOOP_CONF_DIR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4553: [FLINK-7642] [docs] Add very obvious warning about outdat...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4553 Afaik the problem is that the doc builders are independent of Flink version. So we use the same toolchain for building 0.8 docs and 1.4 docs. The `flink.conf` file in https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects contains the builder config. I've introduced @twalthr into the ugly details of it a few weeks ago, maybe he can look into it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4452: [FLINK-7013] [shading] Introduce flink-shaded-netty-4
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4452 +1 to merge: - Checked with a project depending on Flink. It doesn't fail anymore after using your PR - mvn build passes - the dependency-red-pom of flink-runtime looks good (no netty deps) - start-local.sh works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4452: [FLINK-7013] [shading] Introduce flink-shaded-nett...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/4452#discussion_r130890710
--- Diff: tools/travis_mvn_watchdog.sh ---
@@ -290,6 +290,15 @@ check_shaded_artifacts() {
echo
"=="
return 1
fi
+
+NETTY=`cat allClasses | grep '^io/netty' | wc -1`
--- End diff --
really minor, but: Tab vs space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4452: [FLINK-7013] [shading] Introduce flink-shaded-nett...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4452#discussion_r130668606 --- Diff: flink-dist/pom.xml --- @@ -204,6 +204,11 @@ under the License. + + + org.apache.flink + flink-shaded-netty + --- End diff -- So we are going to keep it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4452: [FLINK-7013] [shading] Introduce flink-shaded-nett...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4452#discussion_r130615205 --- Diff: flink-dist/pom.xml --- @@ -204,6 +204,11 @@ under the License. + + + org.apache.flink + flink-shaded-netty + --- End diff -- Why does this need to be added explicitly to flink-dist? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4233 I think this looks good. It's basically a change now restricted to the build system only. +1 to give it a try on master and see how things go with it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4233#discussion_r126991245 --- Diff: pom.xml --- @@ -858,32 +856,6 @@ under the License. - - - flink-fast-tests-a - - - %regex[.*/[J-Z].*] - - - - flink-fast-tests-b - - - %regex[.*/[A-IS-Z].*] - - - - flink-fast-tests-c - - - %regex[.*/[A-R].*] - - --- End diff -- I'm super happy to see this being removed :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4233#discussion_r126990790 --- Diff: .travis.yml --- @@ -17,32 +17,45 @@ matrix: include: # Always run test groups A and B together --- End diff -- I think this comment is invalid now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4233#discussion_r125493654 --- Diff: flink-connectors/pom.xml --- @@ -87,6 +87,39 @@ under the License. + + run-tests + + + flink.test.con --- End diff -- I think `connectors` instead of `con` would be more intuitive --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/4233#discussion_r125493890
--- Diff: flink-fs-tests/pom.xml ---
@@ -96,6 +96,40 @@ under the License.
${hadoop.version}
+
+
+
+ run-tests
+
+
+ flink.test.tst
--- End diff --
test ~ tst?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4233#discussion_r125493851 --- Diff: flink-examples/pom.xml --- @@ -70,6 +70,40 @@ under the License. + + + + run-tests + + + flink.test.msc --- End diff -- misc instead of msc? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4233 So executing `mvn clean verify` won't run the tests, right? Have you considered doing this, basically without touching the pom files at all? Basically by having a script that says "cd flink-core; mvn install cd flink-runtime; mvn install cd flink-tests; mvn verify " I think there's also a command to tell maven "build all your dependencies" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3911: [FLINK-6539] Add end-to-end tests
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3911 Okay --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3911: [FLINK-6539] Add end-to-end tests
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3911 I think the change is okay to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3270 This pull request actually updates the Kafka connector examples, so I think there's no need to do anything here: https://github.com/apache/flink/pull/3911/files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3270 Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4211: [FLINK-6674] [FLINK-6680] [docs] Update migration ...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/4211#discussion_r124589466
--- Diff: docs/dev/migration.md ---
@@ -25,6 +25,62 @@ under the License.
* This will be replaced by the TOC
{:toc}
+## Migrating from Flink 1.2 to Flink 1.3
+
+There are a few APIs that have been changed since Flink 1.2. Most of the
changes are documented in their
+specific documentations. The following is a consolidated list of API
changes and links to details for migration when
+upgrading to Flink 1.3.
+
+### `TypeSerializer` interface changes
+
+This would be relevant mostly for users implementing custom
`TypeSerializer`s for their state.
+
+Since Flink 1.3, two additional methods have been added that are related
to serializer compatibility
+across savepoint restores. Please see
+[Handling serializer upgrades and
compatibility](../stream/state.html#handling-serializer-upgrades-and-compatibility)
+for further details on how to implement these methods.
+
+### `ProcessFunction` is always a `RichFunction`
+
+In Flink 1.2, `ProcessFunction` and its rich variant `RichProcessFunction`
was introduced.
+Since Flink 1.3, `RichProcessFunction` was removed and `ProcessFunction`
is now always a `RichFunction` with access to
+the lifecycle methods and runtime context.
+
+### Flink CEP library API changes
+
+The CEP library in Flink 1.3 ships with a number of new features which
have led to some changes in the API.
+Please visit the [CEP Migration
docs](../libs/cep.html#migrating-from-an-older-flink-version) for details.
+
+### Table API Changes
+
+*TBA*
+
+### Queryable State client construction changes
+
+*TBA*
--- End diff --
I don't think these TBA sections are a good idea in the documentation.
Maybe we should comment them out before committing them, so that they don't
show up in the page
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4208: [FLINK-7004] Switch to Travis Trusty image
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4208 +1 to merge once travis has passed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4207: [FLINK-7004] Switch to Travis Trusty image
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4207 +1 to merge once travis has passed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4182: [FLINK-7004] Switch to Travis Trusty image
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4182#discussion_r124588442 --- Diff: pom.xml --- @@ -1021,6 +1021,7 @@ under the License. tools/artifacts/** tools/flink*/** + apache-maven-3.2.5/** --- End diff -- Maybe add a comment why this exclusion is needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4182: [FLINK-7004] Switch to Travis Trusty image
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4182 +1 to merge once my concerns are addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4182: [FLINK-7004] Switch to Travis Trusty image
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/4182#discussion_r124588048 --- Diff: .travis.yml --- @@ -1,7 +1,8 @@ # s3 deployment based on http://about.travis-ci.org/blog/2012-12-18-travis-artifacts/ # send to container based infrastructure: http://docs.travis-ci.com/user/workers/container-based-infrastructure/ --- End diff -- This comment is actually invalid now. Its the "SUDO-ENABLED" infra. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3270#discussion_r124046763 --- Diff: flink-examples/flink-examples-streaming/pom.xml --- @@ -482,54 +502,19 @@ under the License. + - + org.apache.maven.plugins maven-shade-plugin - fat-jar-kafka-example - package - - shade - - - false - false - false - - - org.apache.flink.streaming.examples.kafka.ReadFromKafka - - - Kafka - - - - * - - org/apache/flink/streaming/examples/kafka/** - org/apache/flink/streaming/** - org/apache/kafka/** - org/apache/curator/** - org/apache/zookeeper/** - org/apache/jute/** - org/I0Itec/** - jline/** - com/yammer/** - kafka/** - - - - - - --- End diff -- What's the reasoning for deleting this section? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3270#discussion_r124046687 --- Diff: flink-examples/flink-examples-streaming/pom.xml --- @@ -360,6 +360,26 @@ under the License. + + + Kafka09 + package + + jar + + + Kafka09 + + + org.apache.flink.streaming.examples.kafka.ReadFromKafka09 + + + + org/apache/flink/streaming/examples/kafka/ReadFromKafka09.class + org/apache/flink/streaming/examples/kafka/WriteToKafka09.class + + + --- End diff -- I don't think you can use the maven jar plugin for creating the kafka09 example. This won't include the needed dependencies into the jar. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4175: [FLINK-6994] [docs] Wrong base url in master docs
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4175 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4017: [FLINK-6774][build] set missing build-helper-maven-plugin...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4017 That would be nice, yes. Thx Sent from my iPhone > On 23. Jun 2017, at 11:52, zentol <notificati...@github.com> wrote: > > I could include this in my merge PR #4124 if you want @rmetzger > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread. > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4017: [FLINK-6774][build] set missing build-helper-maven-plugin...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4017 With the current travis situation, I can not validate this branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4017: [FLINK-6774][build] set missing build-helper-maven-plugin...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4017 Change looks good. I'll merge it once travis gives green light on my fork. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4111: [FLINK-6896][table] Fix generate PojoType input result ex...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4111 Please see my message in the 1.3.1 thread on the dev@ list :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4102: [FLINK-6886][table]Fix Timestamp field can not be selecte...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4102 Please see my message in the 1.3.1 thread on the dev@ list :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2487: [FLINK-4520][flink-siddhi] Integrate Siddhi as a light-we...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2487 @asdf2014 I think there was never a PR for siddhi at the Bahir project. But if you are interested, you could work on contributing it to bahir. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4083: [FLINK-6742] Improve savepoint migration failure error me...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4083 Change looks good to merge! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4048: [FLINK-6812] Enforce Java8 when creating a release
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4048 I've tested the change with the release scripts, and the elasticsearch5 connector was showing up. I'll merge the change now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4039: [FLINK-6783] Changed passing index of type argument while...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4039 Cool, thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4039: [FLINK-6783] Changed passing index of type argument while...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4039 @dawidwys What's your schedule to address the comments? This is one of the real blockers for the 1.3.1 release, and I would like to put the first RC this week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4048: [FLINK-6812] Enforce Java8 when creating a release
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/4048 [FLINK-6812] Enforce Java8 when creating a release You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink6812 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4048.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4048 commit 207eda0ecbe0b7f9c8bf2f637ea5633caf55520a Author: Robert Metzger <rmetz...@apache.org> Date: 2017-06-02T10:12:49Z [FLINK-6812] Enforce Java8 when creating a release --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4041: [FLINK-6198] [cep] Update CEP documentation.
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4041 I'll update the release announcement blog post, once the CEP doc has been updated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3916: [FLINK-6590] Integrate automatic docs generation
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3916 How about the following: - We commit the generated files to git - We try in build-docs.sh to update the generated files, but a failure won't break the doc build process This way, we have an "eventual consistency" model for the config parameters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4041: [FLINK-6198] [cep] Update CEP documentation.
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4041 Cool, thx --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4041: [FLINK-6198] [cep] Update CEP documentation.
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4041 @dawidwys Do you think we can merge the updated documentation today already? I would like to put out the 1.3 release ann with a link to the CEP docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3916: [FLINK-6590] Integrate automatic docs generation
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3916#discussion_r119607441 --- Diff: docs/_includes/generated/environment_options_configuration.html --- @@ -0,0 +1 @@ +KeyDefault ValueDescriptionenv.java.opts(none)Set custom JVM options. This value is respected by Flink's start scripts, both JobManager and TaskManager, and Flink's YARN client. This can be used to set different garbage collectors or to include remote debuggers into the JVMs running Flink's services. Enclosing options in double quotes delays parameter substitution allowing access to variables from Flink's startup scripts.pUse `env.java.opts.jobmanager` and `env.java.opts.taskmanager` for JobManager or TaskManager-specific options, respectively. /penv.java.opts.jobmanager(none)JobManager-specific JVM options. These are used in addition to the regular `env.java.opts`. env.java.opts.taskmanager(none)TaskManager-specific JVM options. These are used in addition to the regular `env.java.opts`. --- End diff -- Looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3833: [FLINK-6221] Add PrometheusReporter
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3833#discussion_r119595452 --- Diff: flink-metrics/flink-metrics-prometheus/pom.xml --- @@ -0,0 +1,129 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd;> + + 4.0.0 + + + org.apache.flink + flink-metrics + 1.3-SNAPSHOT --- End diff -- I guess that's outdated by now. Sorry :( --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3916: [FLINK-6590] Integrate automatic docs generation
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3916 +1 for integration into the build-docs.sh, we just have to make sure that the docs are still properly build on the buildbot for the flink website. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3916: [FLINK-6590] Integrate automatic docs generation
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3916#discussion_r119594668 --- Diff: docs/_includes/generated/environment_options_configuration.html --- @@ -0,0 +1 @@ +KeyDefault ValueDescriptionenv.java.opts(none)Set custom JVM options. This value is respected by Flink's start scripts, both JobManager and TaskManager, and Flink's YARN client. This can be used to set different garbage collectors or to include remote debuggers into the JVMs running Flink's services. Enclosing options in double quotes delays parameter substitution allowing access to variables from Flink's startup scripts.pUse `env.java.opts.jobmanager` and `env.java.opts.taskmanager` for JobManager or TaskManager-specific options, respectively. /penv.java.opts.jobmanager(none)JobManager-specific JVM options. These are used in addition to the regular `env.java.opts`. env.java.opts.taskmanager(none)TaskManager-specific JVM options. These are used in addition to the regular `env.java.opts`. --- End diff -- Maybe we can improve human readability by using newlines after each table row? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4011: [FLINK-6766] Update documentation about async backends an...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4011 I'll merge the change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3979: [FLINK-6704][yarn] Fix user-jars not being possible to ex...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3979 +1 to merge (assuming this change has been tested on a cluster) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3987: [FLINK-6714] [runtime] Use user classloader for operator ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3987 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3984: [FLINK-6710] Remove Twitter-InputFormat
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3984 +1 to remove it (however, please merge this only after at least one other PMC member also +1ed and after some time passed. I don't want to cause any discussions afterwards) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3960: [FLINK-6654][build] let 'flink-dist' properly depend on '...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3960 I'm not sure if the proposed fix is correct, because it will add the shaded hadoop uber project's dependencies into Flink dist. IIRC, this was the whole point of Stephan's recent refactoring of the dependencies. I did a quick check, just based on the flink-dist file size and content, and with the current rel 1.3 branch, we have the following file sizes: ``` -rw-r--r-- 1 robert robert 68M May 24 09:47 flink-dist_2.10-1.3-SNAPSHOT.jar -rw-r--r-- 1 robert robert 99K May 24 09:40 flink-python_2.10-1.3-SNAPSHOT.jar -rw-r--r-- 1 robert robert 23M May 24 09:47 flink-shaded-hadoop2-uber-1.3-SNAPSHOT.jar -rw-r--r-- 1 robert robert 479K Nov 17 2015 log4j-1.2.17.jar -rw-r--r-- 1 robert robert 8.7K Nov 17 2015 slf4j-log4j12-1.7.7.jar ``` after the change, the flink dist file grows significantly: ``` -rw-r--r-- 1 robert robert 87M May 24 09:51 flink-dist_2.10-1.3-SNAPSHOT.jar -rw-r--r-- 1 robert robert 99K May 24 09:40 flink-python_2.10-1.3-SNAPSHOT.jar -rw-r--r-- 1 robert robert 23M May 24 09:47 flink-shaded-hadoop2-uber-1.3-SNAPSHOT.jar -rw-r--r-- 1 robert robert 479K Nov 17 2015 log4j-1.2.17.jar -rw-r--r-- 1 robert robert 8.7K Nov 17 2015 slf4j-log4j12-1.7.7.jar ``` I quickly opened the file and I found all `org.apache.hadoop` classes in the dist again. I'm against doing a "quick fix" before the release here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3970: [FLINK-6675] Activate strict checkstyle for flink-annotat...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3970 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3960: [FLINK-6654][build] let 'flink-dist' properly depend on '...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3960 Okay, I was able to reproduce the issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3960: [FLINK-6654][build] let 'flink-dist' properly depend on '...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3960 Not sure about the initial issue. For me the command worked (without yoru changes on the 1.3 branch) ``` [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 10:12 min [INFO] Finished at: 2017-05-23T16:50:31+02:00 [INFO] Final Memory: 131M/441M [INFO] mvn clean install -pl flink-dist -am -DskipTests ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3961: [FLINK-6656] [cep] Change element PriorityQueue to MapSta...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3961 @kl0u asked me to merge this PR to the release-1.3 branch. It seems the change is good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3965: [FLINK-6328] [chkPts] Don't add savepoints to CompletedCh...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3965 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3937: [FLINK-6482] [core] Add nested serializers to config snap...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3937 Thanks a lot for the quick review and merge!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3937: [FLINK-6482] [core] Add nested serializers to config snap...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3937 Thank you for your response. This PR is the last item for the next release candidate. Would be good to get this in in the next few hours so that I can finally start the vote for the 1.3 release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3947: [FLINK-6610][web] Allow uploadDir to be null in WebFronte...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3947 +1 to merge (tested the change locally) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3949: [FLINK-6629] Use HAServices to find connecting address fo...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3949 +1 to merge this change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3937: [FLINK-6482] [core] Add nested serializers to config snap...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3937 Hey, what's the status of this PR? I see the JIRA is marked as a blocker. When do you think is this PR ready to be merged? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3950: [FLINK-5636][metrics] Measure numRecordsIn in StreamTwoIn...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3950 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3911: [FLINK-6539] Add end-to-end tests
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3911 I tried it locally, but got some weird results: ``` Found error in log files. Found exception in log files. Found non-empty file build-target/log/flink-robert-jobmanager-0-robert-da.out Found non-empty file build-target/log/flink-robert-taskmanager-0-robert-da.out One or more tests FAILED. ``` It seems that the script deleted the log files. Is that intended? Also it seemed that WC was running correctly, so I wonder why it failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3911: [FLINK-6539] Add end-to-end tests
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3911#discussion_r117445440 --- Diff: tools/travis_mvn_watchdog.sh --- @@ -227,5 +227,15 @@ upload_artifacts_s3 # we are going back to cd ../../ +# run end-to-end tests + +echo "Running automated end-to-end tests" + +.test-infa/end-to-end-test/test_batch_wordcount.sh build-target cluster +E2E_WC_EXIT_CODE=$? +EXIT_CODE=$(($EXIT_CODE+$?)) +.test-infa/end-to-end-test/test_streaming_kafka010.sh build-target cluster +EXIT_CODE=$(($EXIT_CODE+$?)) --- End diff -- I like the travis integration a lot --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3856: [FLINK-5998] Fix shaded Hadoop Jar
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3856 Afaik this PR can be closed because it has been superseded by Stephan's PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3910: [FLINK-6582] [docs] Project from maven archetype is not b...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3910 Thanks a lot for fixing the issue. I've tested it again, and it works. Merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3910: [FLINK-6582] [docs] Project from maven archetype is not b...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3910 Thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3933: [FLINK-6606] Create checkpoint hook with user classloader
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3933 Lets see what Till says. In my opinion, both approaches are fine. What you are proposing makes things on Flink's side a bit more complicated, but the behavior is consistent with other places where user code is executed. Till's proposal (so get the CL via the class) keeps our code straightforward, but users have to know it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3933: [FLINK-6606] Create checkpoint hook with user classloader
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3933 Till and I had an offline discussion about this. What you can do in your user code to set the TCCL properly is the following: ``` Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader()); ``` `this` points in user code to a class loaded through the user code classlaoder. So `this.getClass().getClassLoader()` is the CL you need to set for the thread. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3910: [FLINK-6582] [docs] Project from maven archetype is not b...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3910 Not sure what's happening, but when I'm trying to generate a project from the archetype, I'm getting: ``` Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.4:generate (default-cli) on project standalone-pom: The defined artifact is not an archetype ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3910: [FLINK-6582] [docs] Project from maven archetype is not b...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3910 the change *looks* good to merge. I'm currently building Flink to quickly try it out --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3906: [FLINK-6416] Fix divide-by-zero in InputGateMetrics
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3906 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3917: [FLINK-6440][metrics] Downgrade fetching failure logging ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3917 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3910: [FLINK-6582] [docs] Project from maven archetype is not b...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3910 The script is deploying scala 2.10 for the quickstarts, yes. I think the reason for this is that scala 2.10 is still considered "default" in Flink. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3932: [FLINK-6616] [docs] Clarify provenance of official Docker...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3932 +1 Very good, thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3933: [FLINK-6606] Create checkpoint hook with user classloader
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3933 I haven't worked on this part of the code before, but I believe this code is +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3931: [FLINK-6031][yarn] Add config parameter for user-jar incl...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3931 I'll merge the change now ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3931: [FLINK-6031][yarn] Add config parameter for user-jar incl...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3931 Thank you for addressing my comments so quickly. The change is good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3931: [FLINK-6031][yarn] Add config parameter for user-jar incl...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3931 (for both 1.3 and 1.4) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3931: [FLINK-6031][yarn] Add config parameter for user-j...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3931#discussion_r117204403 --- Diff: docs/setup/yarn_setup.md --- @@ -245,6 +245,18 @@ Note: You can use a different configuration directory per job by setting the env Note: It is possible to combine `-m yarn-cluster` with a detached YARN submission (`-yd`) to "fire and forget" a Flink job to the YARN cluster. In this case, your application will not get any accumulator results or exceptions from the ExecutionEnvironment.execute() call! +### User jars & Classpath + +By default Flink will include the user jars into the system classpath when running a single job. This behavior can be controlled with the `yarn.per-job-cluster.include-job-jar` parameter. --- End diff -- The configuration parameter is actually called `yarn.per-job-cluster.include-user-jar`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3931: [FLINK-6031][yarn] Add config parameter for user-j...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3931#discussion_r117199176
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -665,58 +685,52 @@ public ApplicationReport startAppMaster(JobGraph
jobGraph, YarnClient yarnClient
1));
}
+ String configuredUserJarInclusion =
flinkConfiguration.getString(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR);
+ YarnConfigOptions.UserJarInclusion userJarInclusion;
+ try {
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(configuredUserJarInclusion.toUpperCase());
+ } catch (IllegalArgumentException e) {
+ LOG.warn("Configuration parameter {} was configured
with an invalid value {}. Falling back to default ({}).",
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.key(),
+ configuredUserJarInclusion,
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
--- End diff --
This also seems very similar to the other duplicates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3931: [FLINK-6031][yarn] Add config parameter for user-j...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3931#discussion_r117198446
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -200,6 +214,17 @@ public void setTaskManagerMemory(int memoryMb) {
public void
setFlinkConfiguration(org.apache.flink.configuration.Configuration conf) {
this.flinkConfiguration = conf;
+
+ String configuredUserJarInclusion =
flinkConfiguration.getString(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR);
+ try {
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(configuredUserJarInclusion.toUpperCase());
+ } catch (IllegalArgumentException e) {
+ LOG.warn("Configuration parameter {} was configured
with an invalid value {}. Falling back to default ({}).",
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.key(),
+ configuredUserJarInclusion,
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ }
--- End diff --
This code and the code above looks like exactly the same.
If this is duplicate code, it should be extracted into one method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3931: [FLINK-6031][yarn] Add config parameter for user-j...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3931#discussion_r117199375 --- Diff: docs/setup/yarn_setup.md --- @@ -245,6 +245,18 @@ Note: You can use a different configuration directory per job by setting the env Note: It is possible to combine `-m yarn-cluster` with a detached YARN submission (`-yd`) to "fire and forget" a Flink job to the YARN cluster. In this case, your application will not get any accumulator results or exceptions from the ExecutionEnvironment.execute() call! +### User jars & Classpath + +By default Flink will include the user jars into the system classpath when running a single job. This behavior can be controlled with the `yarn.per-job-cluster.include-job-jar` parameter. + +When setting this to `DISABLED` Flink will include the jar in the user classpath instead. + +The user-jars position in the class path can be controlled by setting the parameter to one of the following: + +- `ORDER`: (default) Adds the jar to the system class path based on the lexicographic order. +- `FIRST`: Adds the jar to the beginning of the system class path. +- `LAST`: Adds the jar to the end of the system class path. + --- End diff -- Ideally, you put the configuration property also into the `setup/config.md` page, where all config options are listed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3877: [backport] [FLINK-6514] [build] Create a proper separate ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3877 @StephanEwen can you manually close the PR? Thx. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3926: [FLINK-6570] QueryableStateClient docs with matching cons...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3926 Sadly, our tests are a little bit unstable. The issue is not related to your change. @tillrohrmann could you talk a quick look here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3903: [FLINK-6581] [cli] Correct dynamic property parsing for Y...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3903 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3900: [FLINK-6580] Sync default heap sizes from code with confi...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3900 @zentol do you agree to merge this now. Its the last thing I would like to get into RC1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3900: [FLINK-6580] Sync default heap sizes from code wit...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3900#discussion_r116492253
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
---
@@ -62,7 +62,7 @@
*/
public static final ConfigOption JOB_MANAGER_HEAP_MEMORY =
key("jobmanager.heap.mb")
- .defaultValue(1024);
+ .defaultValue(768);
--- End diff --
Yeah, I'll go back to 1024.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3877: [backport] [FLINK-6514] [build] Create a proper separate ...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3877 Merging this change ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3876: [FLINK-6514] [build] Create a proper separate Hadoop uber...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3876 I'm merging this change now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3900: [FLINK-6580] Sync default heap sizes from code wit...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/3900 [FLINK-6580] Sync default heap sizes from code with config file Flink didn't start on YARN anymore without explicit configuration of JM and TM heap because the default heap sizes in the yaml file were set too low. This PR increases the default heap sizes according to the `TaskManagerOptions` and `JobManagerOptions` classes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink FLINK-6580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3900.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3900 commit d115416613313f5d01a47e92c33e87b3075008e8 Author: Robert Metzger <rmetz...@apache.org> Date: 2017-05-15T09:15:48Z [FLINK-6580] Sync default heap sizes from code with config file --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3876: [FLINK-6514] [build] Create a proper separate Hadoop uber...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3876 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3890: [FLINK-6569] flink-table KafkaJsonTableSource example doe...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3890 Thank you for addressing this so quickly. +1 from my side. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3870: [Flink 6537] Fixes and improvements for incremental check...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3870 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
