[jira] [Created] (FLINK-1560) Add ITCases for streaming examples
Márton Balassi created FLINK-1560: - Summary: Add ITCases for streaming examples Key: FLINK-1560 URL: https://issues.apache.org/jira/browse/FLINK-1560 Project: Flink Issue Type: Test Components: Streaming Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Márton Balassi Currently there are no tests for consistency of the streaming example programs. This might be a real show stopper for users who encounter an issue there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-947) Add support for "Named Datasets"
[
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323372#comment-14323372
]
ASF GitHub Bot commented on FLINK-947:
--
Github user mbalassi commented on the pull request:
https://github.com/apache/flink/pull/405#issuecomment-74579541
Great additions! Really looking to trying it out. :)
> Add support for "Named Datasets"
>
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
> Issue Type: New Feature
> Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
> .project("a", "b", "c")
> .map( (UserTypeIn in) -> return new UserTypeOut(...) )
> .print();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-947) Add support for "Named Datasets"
[
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323370#comment-14323370
]
ASF GitHub Bot commented on FLINK-947:
--
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/405#discussion_r24778932
--- Diff: flink-staging/flink-linq/pom.xml ---
@@ -0,0 +1,227 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-staging
+ 0.9-SNAPSHOT
+ ..
+
+
+ flink-linq
+ flink-linq
+
+ jar
+
+
+
+
+ org.apache.flink
+ flink-scala
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-streaming-scala
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-scala-examples
+ ${project.version}
+
+
+
+ org.scala-lang
+ scala-reflect
+
+
+
+ org.scala-lang
+ scala-library
+
+
+
+ org.scala-lang
+ scala-compiler
+
+
--- End diff --
You transitively depend on the scala stuff through flink-scala, so you
could omit these.
> Add support for "Named Datasets"
>
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
> Issue Type: New Feature
> Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
> .project("a", "b", "c")
> .map( (UserTypeIn in) -> return new UserTypeOut(...) )
> .print();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-947] Add a declarative expression API
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/405#discussion_r24778932
--- Diff: flink-staging/flink-linq/pom.xml ---
@@ -0,0 +1,227 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-staging
+ 0.9-SNAPSHOT
+ ..
+
+
+ flink-linq
+ flink-linq
+
+ jar
+
+
+
+
+ org.apache.flink
+ flink-scala
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-streaming-scala
+ ${project.version}
+
+
+
+ org.apache.flink
+ flink-scala-examples
+ ${project.version}
+
+
+
+ org.scala-lang
+ scala-reflect
+
+
+
+ org.scala-lang
+ scala-library
+
+
+
+ org.scala-lang
+ scala-compiler
+
+
--- End diff --
You transitively depend on the scala stuff through flink-scala, so you
could omit these.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-947] Add a declarative expression API
Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/405#issuecomment-74579541 Great additions! Really looking to trying it out. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1466] Add HCatInputFormats to read from...
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/411 [FLINK-1466] Add HCatInputFormats to read from HCatalog tables. Right now the Flink-tuple mode is restricted to primitive types (no ARRAY, STRUCT, and MAP types) and the max Java/Scala tuple width. The default HCatRecord mode works for all types and any tuple width. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink hcatalog Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #411 commit 61d8c3e2c7948768f60bbb20755176aa8ef14fcd Author: Fabian Hueske Date: 2015-01-29T09:34:29Z [FLINK-1466] Adds HCatInputFormats to read from HCatalog tables. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1466) Add InputFormat to read HCatalog tables
[ https://issues.apache.org/jira/browse/FLINK-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323344#comment-14323344 ] ASF GitHub Bot commented on FLINK-1466: --- GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/411 [FLINK-1466] Add HCatInputFormats to read from HCatalog tables. Right now the Flink-tuple mode is restricted to primitive types (no ARRAY, STRUCT, and MAP types) and the max Java/Scala tuple width. The default HCatRecord mode works for all types and any tuple width. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink hcatalog Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #411 commit 61d8c3e2c7948768f60bbb20755176aa8ef14fcd Author: Fabian Hueske Date: 2015-01-29T09:34:29Z [FLINK-1466] Adds HCatInputFormats to read from HCatalog tables. > Add InputFormat to read HCatalog tables > --- > > Key: FLINK-1466 > URL: https://issues.apache.org/jira/browse/FLINK-1466 > Project: Flink > Issue Type: New Feature > Components: Java API, Scala API >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Minor > > HCatalog is a metadata repository and InputFormat to make Hive tables > accessible to other frameworks such as Pig. > Adding support for HCatalog would give access to Hive managed data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Add auto-parallelism to Jobs (0.8 branch)
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/410 Add auto-parallelism to Jobs (0.8 branch) You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink autopar Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/410.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #410 commit 923d1b4309c10a86cfa8ea3c385ff751c59e29a4 Author: Stephan Ewen Date: 2015-02-16T20:40:06Z Add autoparallelism to jobs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-785) Add Chained operators for AllReduce and AllGroupReduce
[ https://issues.apache.org/jira/browse/FLINK-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323225#comment-14323225 ] ASF GitHub Bot commented on FLINK-785: -- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/370#issuecomment-74565386 I think we should use the chained reducer from this pull request and drop the chained combiner change. > Add Chained operators for AllReduce and AllGroupReduce > -- > > Key: FLINK-785 > URL: https://issues.apache.org/jira/browse/FLINK-785 > Project: Flink > Issue Type: Improvement >Reporter: GitHub Import >Assignee: Chesnay Schepler > Labels: github-import > Fix For: pre-apache > > > Because the operators `AllReduce` and `AllGroupReduce` are used both for the > pre-reduce (combiner side) and the final reduce, they would greatly benefit > from a chained version. > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/785 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: runtime, > Milestone: Release 0.6 (unplanned) > Created at: Sun May 11 17:41:12 CEST 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1521] Chained operators respect reuse
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/392#issuecomment-74565272 At this point, we also have to worry about efficiency and performance. This is also something that the user can easily mitigate in the UDF, at a fraction of the cost in most cases. I vote postpone the fix until we get some tangible user feedback concerning this issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-785] Chained AllReduce / AllGroupReduce...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/370#issuecomment-74565386 I think we should use the chained reducer from this pull request and drop the chained combiner change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag
[ https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323223#comment-14323223 ] ASF GitHub Bot commented on FLINK-1521: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/392#issuecomment-74565272 At this point, we also have to worry about efficiency and performance. This is also something that the user can easily mitigate in the UDF, at a fraction of the cost in most cases. I vote postpone the fix until we get some tangible user feedback concerning this issue. > Some Chained Drivers do not respect object-reuse/non-reuse flag > --- > > Key: FLINK-1521 > URL: https://issues.apache.org/jira/browse/FLINK-1521 > Project: Flink > Issue Type: Bug >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1556] Corrects faulty JobClient behavio...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/406#issuecomment-74565029 Looks good to me! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1557] Move JobManager web frontend serv...
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/409 [FLINK-1557] Move JobManager web frontend server out of JobManager actor You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink webserver Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit a24cbec30aa3ae5c715b3d41fc11a120537b802b Author: Stephan Ewen Date: 2015-02-16T15:34:13Z [FLINK-1557] Move JobManager web frontend server out of JobManager actor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception
[
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323219#comment-14323219
]
ASF GitHub Bot commented on FLINK-1556:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/406#issuecomment-74565029
Looks good to me!
> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
> Issue Type: Bug
>Reporter: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}}
> terminates itself and returns the error to the {{Client}}. This indicates to
> the user that the job has been completely failed which is not necessarily
> true.
> If the user directly after such a failure submits another job, then it might
> be the case that not all slots of the formerly failed job are returned. This
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in
> the {{JobClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1557) Move WebFrontend out of JobManager Actor
[ https://issues.apache.org/jira/browse/FLINK-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323218#comment-14323218 ] ASF GitHub Bot commented on FLINK-1557: --- GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/409 [FLINK-1557] Move JobManager web frontend server out of JobManager actor You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink webserver Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit a24cbec30aa3ae5c715b3d41fc11a120537b802b Author: Stephan Ewen Date: 2015-02-16T15:34:13Z [FLINK-1557] Move JobManager web frontend server out of JobManager actor > Move WebFrontend out of JobManager Actor > > > Key: FLINK-1557 > URL: https://issues.apache.org/jira/browse/FLINK-1557 > Project: Flink > Issue Type: Improvement > Components: JobManager >Affects Versions: 0.9 >Reporter: Stephan Ewen >Assignee: Stephan Ewen > Fix For: 0.9 > > > Currently, the Webfrontent is started inside the JobManager actor. > I propose to move it out of the actor, since it is a heavy component that is > independent of the actor and whose failure cannot be controlled by the job > manager. > The change is fairly straightforward, since the web frontend only > communicates with the jobmanager through actor messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1542) Test case at BlobUtilsTest should not assume user could not create new item in root directory
[ https://issues.apache.org/jira/browse/FLINK-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1542. - Resolution: Fixed Fix Version/s: 0.9 Fixed via 5ab1f1f197b4ab86382e7e20b4964e991663b62c > Test case at BlobUtilsTest should not assume user could not create new item > in root directory > - > > Key: FLINK-1542 > URL: https://issues.apache.org/jira/browse/FLINK-1542 > Project: Flink > Issue Type: Bug >Reporter: Henry Saputra >Assignee: Henry Saputra > Fix For: 0.9 > > > The test at BlobUtilsTest #testExceptionOnCreateStorageDirectoryFailure > assume that current user cannot create new directory at root. > In my MacOSX, my user has admin group so the test will always fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1542) Test case at BlobUtilsTest should not assume user could not create new item in root directory
[ https://issues.apache.org/jira/browse/FLINK-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323208#comment-14323208 ] ASF GitHub Bot commented on FLINK-1542: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/396 > Test case at BlobUtilsTest should not assume user could not create new item > in root directory > - > > Key: FLINK-1542 > URL: https://issues.apache.org/jira/browse/FLINK-1542 > Project: Flink > Issue Type: Bug >Reporter: Henry Saputra >Assignee: Henry Saputra > Fix For: 0.9 > > > The test at BlobUtilsTest #testExceptionOnCreateStorageDirectoryFailure > assume that current user cannot create new directory at root. > In my MacOSX, my user has admin group so the test will always fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1542] Test case at BlobUtilsTest should...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/396 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1454) CliFrontend blocks for 100 seconds when submitting to a non-existent JobManager
[
https://issues.apache.org/jira/browse/FLINK-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323205#comment-14323205
]
ASF GitHub Bot commented on FLINK-1454:
---
Github user StephanEwen closed the pull request at:
https://github.com/apache/flink/pull/404
> CliFrontend blocks for 100 seconds when submitting to a non-existent
> JobManager
> ---
>
> Key: FLINK-1454
> URL: https://issues.apache.org/jira/browse/FLINK-1454
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> When a user tries to submit a job to a job manager which doesn't exist at
> all, the CliFrontend blocks for 100 seconds.
> Ideally, Akka would fail because it can not connect to the given
> hostname:port.
>
> {code}
> ./bin/flink run ./examples/flink-java-examples-0.9-SNAPSHOT-WordCount.jar -c
> foo.Baz
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:449)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:350)
> at org.apache.flink.client.program.Client.run(Client.java:242)
> at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:389)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:362)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1078)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1102)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [100 seconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265)
> at
> org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:169)
> at
> org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
> at org.apache.flink.client.program.Client.run(Client.java:314)
> at org.apache.flink.client.program.Client.run(Client.java:296)
> at org.apache.flink.client.program.Client.run(Client.java:290)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:434)
> ... 6 more
> The exception above occurred while trying to run your command.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1454) CliFrontend blocks for 100 seconds when submitting to a non-existent JobManager
[
https://issues.apache.org/jira/browse/FLINK-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323204#comment-14323204
]
ASF GitHub Bot commented on FLINK-1454:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/404#issuecomment-74563335
Manually merged in b941cf2d091bb38b9a45e1f2412136acae2b0f3f
> CliFrontend blocks for 100 seconds when submitting to a non-existent
> JobManager
> ---
>
> Key: FLINK-1454
> URL: https://issues.apache.org/jira/browse/FLINK-1454
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> When a user tries to submit a job to a job manager which doesn't exist at
> all, the CliFrontend blocks for 100 seconds.
> Ideally, Akka would fail because it can not connect to the given
> hostname:port.
>
> {code}
> ./bin/flink run ./examples/flink-java-examples-0.9-SNAPSHOT-WordCount.jar -c
> foo.Baz
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:449)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:350)
> at org.apache.flink.client.program.Client.run(Client.java:242)
> at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:389)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:362)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1078)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1102)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [100 seconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265)
> at
> org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:169)
> at
> org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
> at org.apache.flink.client.program.Client.run(Client.java:314)
> at org.apache.flink.client.program.Client.run(Client.java:296)
> at org.apache.flink.client.program.Client.run(Client.java:290)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:434)
> ... 6 more
> The exception above occurred while trying to run your command.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1454] [job client] Improve error handli...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/404#issuecomment-74563335 Manually merged in b941cf2d091bb38b9a45e1f2412136acae2b0f3f --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1454] [job client] Improve error handli...
Github user StephanEwen closed the pull request at: https://github.com/apache/flink/pull/404 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-1559) Normalize all Akka URLs to use IP addresses, not hostnames
[
https://issues.apache.org/jira/browse/FLINK-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1559.
-
Resolution: Fixed
Fixed via 2dcff4c122f37ca58a485e886fec9295fef0c832
> Normalize all Akka URLs to use IP addresses, not hostnames
> --
>
> Key: FLINK-1559
> URL: https://issues.apache.org/jira/browse/FLINK-1559
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Akka has the nasty property that it does not use IP addresses and hostnames
> interchangeably.
> Example: If the JobManager is started at
> {{akka:tcp//flink@localhost:1234/user/JOBMANAGER}} and a job client actor
> tries to send messages to {{akka:tcp//flink@127.0.0.1:1234/user/JOBMANAGER}},
> the job manager actor system will receive the messages, but throw them away,
> because they are not destined for the actor system.
> I am not sure whether this is a bug or a feature in akka, but we need to deal
> with it. To be on the safe side, I suggest to build all akka URLs with ip
> addresses. We can still use host names in our logic (connection infos, local
> assignments, etc)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-1454) CliFrontend blocks for 100 seconds when submitting to a non-existent JobManager
[
https://issues.apache.org/jira/browse/FLINK-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1454.
-
Resolution: Fixed
Fix Version/s: 0.9
Assignee: Stephan Ewen
Fixed via b941cf2d091bb38b9a45e1f2412136acae2b0f3f
> CliFrontend blocks for 100 seconds when submitting to a non-existent
> JobManager
> ---
>
> Key: FLINK-1454
> URL: https://issues.apache.org/jira/browse/FLINK-1454
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> When a user tries to submit a job to a job manager which doesn't exist at
> all, the CliFrontend blocks for 100 seconds.
> Ideally, Akka would fail because it can not connect to the given
> hostname:port.
>
> {code}
> ./bin/flink run ./examples/flink-java-examples-0.9-SNAPSHOT-WordCount.jar -c
> foo.Baz
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:449)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:350)
> at org.apache.flink.client.program.Client.run(Client.java:242)
> at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:389)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:362)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1078)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1102)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [100 seconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265)
> at
> org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:169)
> at
> org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
> at org.apache.flink.client.program.Client.run(Client.java:314)
> at org.apache.flink.client.program.Client.run(Client.java:296)
> at org.apache.flink.client.program.Client.run(Client.java:290)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:434)
> ... 6 more
> The exception above occurred while trying to run your command.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1542] Test case at BlobUtilsTest should...
Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74558015 Thanks for the review Ufuk and Stephan. Also thx for Stephan for merging it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1542) Test case at BlobUtilsTest should not assume user could not create new item in root directory
[ https://issues.apache.org/jira/browse/FLINK-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323150#comment-14323150 ] ASF GitHub Bot commented on FLINK-1542: --- Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74558015 Thanks for the review Ufuk and Stephan. Also thx for Stephan for merging it. > Test case at BlobUtilsTest should not assume user could not create new item > in root directory > - > > Key: FLINK-1542 > URL: https://issues.apache.org/jira/browse/FLINK-1542 > Project: Flink > Issue Type: Bug >Reporter: Henry Saputra >Assignee: Henry Saputra > > The test at BlobUtilsTest #testExceptionOnCreateStorageDirectoryFailure > assume that current user cannot create new directory at root. > In my MacOSX, my user has admin group so the test will always fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1542) Test case at BlobUtilsTest should not assume user could not create new item in root directory
[ https://issues.apache.org/jira/browse/FLINK-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323147#comment-14323147 ] ASF GitHub Bot commented on FLINK-1542: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74556700 Looks good to me as well. Will merge it with the next batch... > Test case at BlobUtilsTest should not assume user could not create new item > in root directory > - > > Key: FLINK-1542 > URL: https://issues.apache.org/jira/browse/FLINK-1542 > Project: Flink > Issue Type: Bug >Reporter: Henry Saputra >Assignee: Henry Saputra > > The test at BlobUtilsTest #testExceptionOnCreateStorageDirectoryFailure > assume that current user cannot create new directory at root. > In my MacOSX, my user has admin group so the test will always fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1542] Test case at BlobUtilsTest should...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74556700 Looks good to me as well. Will merge it with the next batch... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1559) Normalize all Akka URLs to use IP addresses, not hostnames
Stephan Ewen created FLINK-1559:
---
Summary: Normalize all Akka URLs to use IP addresses, not hostnames
Key: FLINK-1559
URL: https://issues.apache.org/jira/browse/FLINK-1559
Project: Flink
Issue Type: Bug
Components: Distributed Runtime
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Akka has the nasty property that it does not use IP addresses and hostnames
interchangeably.
Example: If the JobManager is started at
{{akka:tcp//flink@localhost:1234/user/JOBMANAGER}} and a job client actor tries
to send messages to {{akka:tcp//flink@127.0.0.1:1234/user/JOBMANAGER}}, the job
manager actor system will receive the messages, but throw them away, because
they are not destined for the actor system.
I am not sure whether this is a bug or a feature in akka, but we need to deal
with it. To be on the safe side, I suggest to build all akka URLs with ip
addresses. We can still use host names in our logic (connection infos, local
assignments, etc)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1514) [Gelly] Add a Gather-Sum-Apply iteration method
[
https://issues.apache.org/jira/browse/FLINK-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323032#comment-14323032
]
ASF GitHub Bot commented on FLINK-1514:
---
Github user balidani commented on a diff in the pull request:
https://github.com/apache/flink/pull/408#discussion_r24763748
--- Diff:
flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/GatherSumApplyITCase.java
---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.graph.test;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.graph.Edge;
+import org.apache.flink.graph.Graph;
+import org.apache.flink.graph.Vertex;
+import org.apache.flink.graph.gsa.ApplyFunction;
+import org.apache.flink.graph.gsa.GatherFunction;
+import org.apache.flink.graph.gsa.SumFunction;
+import org.apache.flink.test.util.MultipleProgramsTestBase;
+import org.apache.flink.util.Collector;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+
+@RunWith(Parameterized.class)
+public class GatherSumApplyITCase extends MultipleProgramsTestBase {
+
+ public GatherSumApplyITCase(MultipleProgramsTestBase.ExecutionMode
mode){
+ super(mode);
+ }
+
+ private String resultPath;
+ private String expectedResult;
+
+ @Rule
+ public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ @Before
+ public void before() throws Exception{
+ resultPath = tempFolder.newFile().toURI().toString();
+ }
+
+ @After
+ public void after() throws Exception{
+ compareResultsByLinesInMemory(expectedResult, resultPath);
+ }
+
+ @Test
+ public void testGreedyGraphColoring() throws Exception {
+ /*
+* This test runs the Greedy Graph Coloring algorithm using a
GSA iteration
+* The expected result is that the lowest vertex value (1) gets
propagated to all vertices
+*/
+ final ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
+
+ Graph graph =
Graph.fromDataSet(TestGraphUtils.getLongLongVertexData(env),
+ TestGraphUtils.getLongLongEdgeData(env), env);
+
+ // Gather the target vertices into a one-element set
+ GatherFunction>>
gather =
+ new GatherFunction>>() {
+
+ @Override
+ public Tuple2>> gather(Tuple3,
+ Edge,
Vertex> triplet) {
+
+ HashSet>
result = new HashSet>();
+ result.add(triplet.f2);
+
+ return new Tuple2>>(triplet.f0.getId(), result);
+ }
+ };
+
+ // Merge the sets between neighbors
+ SumFunction>> sum =
+ new SumFunction>>() {
+
+ @Override
+ public Tuple2>> sum(
+ Tuple2>> arg0,
+ Tuple2>> arg1) {
+
+ HashSet>
result = new HashSet>();
+
[GitHub] flink pull request: [FLINK-1514][Gelly] Add a Gather-Sum-Apply ite...
Github user balidani commented on a diff in the pull request:
https://github.com/apache/flink/pull/408#discussion_r24763748
--- Diff:
flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/GatherSumApplyITCase.java
---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.graph.test;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.graph.Edge;
+import org.apache.flink.graph.Graph;
+import org.apache.flink.graph.Vertex;
+import org.apache.flink.graph.gsa.ApplyFunction;
+import org.apache.flink.graph.gsa.GatherFunction;
+import org.apache.flink.graph.gsa.SumFunction;
+import org.apache.flink.test.util.MultipleProgramsTestBase;
+import org.apache.flink.util.Collector;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+
+@RunWith(Parameterized.class)
+public class GatherSumApplyITCase extends MultipleProgramsTestBase {
+
+ public GatherSumApplyITCase(MultipleProgramsTestBase.ExecutionMode
mode){
+ super(mode);
+ }
+
+ private String resultPath;
+ private String expectedResult;
+
+ @Rule
+ public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ @Before
+ public void before() throws Exception{
+ resultPath = tempFolder.newFile().toURI().toString();
+ }
+
+ @After
+ public void after() throws Exception{
+ compareResultsByLinesInMemory(expectedResult, resultPath);
+ }
+
+ @Test
+ public void testGreedyGraphColoring() throws Exception {
+ /*
+* This test runs the Greedy Graph Coloring algorithm using a
GSA iteration
+* The expected result is that the lowest vertex value (1) gets
propagated to all vertices
+*/
+ final ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
+
+ Graph graph =
Graph.fromDataSet(TestGraphUtils.getLongLongVertexData(env),
+ TestGraphUtils.getLongLongEdgeData(env), env);
+
+ // Gather the target vertices into a one-element set
+ GatherFunction>>
gather =
+ new GatherFunction>>() {
+
+ @Override
+ public Tuple2>> gather(Tuple3,
+ Edge,
Vertex> triplet) {
+
+ HashSet>
result = new HashSet>();
+ result.add(triplet.f2);
+
+ return new Tuple2>>(triplet.f0.getId(), result);
+ }
+ };
+
+ // Merge the sets between neighbors
+ SumFunction>> sum =
+ new SumFunction>>() {
+
+ @Override
+ public Tuple2>> sum(
+ Tuple2>> arg0,
+ Tuple2>> arg1) {
+
+ HashSet>
result = new HashSet>();
+ result.addAll(arg0.f1);
+ result.addAll(arg1.f1);
+
+ return new Tuple2>>(arg0.f0, result);
+ }
+
[jira] [Commented] (FLINK-1514) [Gelly] Add a Gather-Sum-Apply iteration method
[ https://issues.apache.org/jira/browse/FLINK-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323029#comment-14323029 ] ASF GitHub Bot commented on FLINK-1514: --- GitHub user balidani opened a pull request: https://github.com/apache/flink/pull/408 [FLINK-1514][Gelly] Add a Gather-Sum-Apply iteration method You can merge this pull request into a Git repository by running: $ git pull https://github.com/balidani/flink gelly-gsa Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit ae96bfefdd09b2d26fae644f071ea882f69a2375 Author: Dániel Bali Date: 2015-02-16T14:06:05Z [FLINK-1514][Gelly] Add a Gather-Sum-Apply iteration method > [Gelly] Add a Gather-Sum-Apply iteration method > --- > > Key: FLINK-1514 > URL: https://issues.apache.org/jira/browse/FLINK-1514 > Project: Flink > Issue Type: New Feature > Components: Gelly >Affects Versions: 0.9 >Reporter: Vasia Kalavri >Assignee: Daniel Bali > > This will be a method that implements the GAS computation model, but without > the "scatter" step. The phases can be mapped into the following steps inside > a delta iteration: > gather: a map on each < srcVertex, edge, trgVertex > that produces a partial > value > sum: a reduce that combines the partial values > apply: join with vertex set to update the vertex values using the results of > sum and the previous state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1514][Gelly] Add a Gather-Sum-Apply ite...
GitHub user balidani opened a pull request: https://github.com/apache/flink/pull/408 [FLINK-1514][Gelly] Add a Gather-Sum-Apply iteration method You can merge this pull request into a Git repository by running: $ git pull https://github.com/balidani/flink gelly-gsa Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit ae96bfefdd09b2d26fae644f071ea882f69a2375 Author: Dániel Bali Date: 2015-02-16T14:06:05Z [FLINK-1514][Gelly] Add a Gather-Sum-Apply iteration method --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [builds] Print Java process stack traces of st...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/389#issuecomment-74535643 Subsumed by #407. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [builds] Print Java process stack traces of st...
Github user uce closed the pull request at: https://github.com/apache/flink/pull/389 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/407
[builds] Allow to upload build artifacts to S3
This PR includes #389.
Currently, logging of Travis builds is disabled as the log files are
discarded after a build and stdout logging is limited in size. With this PR, we
can log to a file and have them uploaded automatically.
1. **Console logging**: For console logging, we can use
`MavenForkNumberPrefixLayout` to prefix the log output with the fork number,
which makes it easier to group log output together.
We might consider moving that layout to a different package, though.
```
7 > 17:12:23,220 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying test
vertex (5/7) (attempt #0) to host3
7 > 17:12:23,220 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (6/7) - execution #0, sharingUnit=null,
locationConstraint=null}
7 > 17:12:23,220 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained
assignment: test vertex (6/7) --> dbc442af2610e30d3309d96e14fa9d8a @
akka://TestingActorSystem/user/$$a - 5 slots - 1067599825
7 > 17:12:23,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying test
vertex (6/7) (attempt #0) to host1
7 > 17:12:23,221 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (7/7) - execution #0, sharingUnit=null,
locationConstraint=null}
7 > 17:12:23,222 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained
assignment: test vertex (7/7) --> 8ce93c214951df5a8ad32c912ba0e456 @
akka://TestingActorSystem/user/$$a - 5 slots - 749927456
7 > 17:12:23,222 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying test
vertex (7/7) (attempt #0) to host2
7 > 17:12:23,223 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Attaching 1
topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
7 > 17:12:23,223 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Connecting
ExecutionJobVertex 126e16c348d4e5aa129104702700bf9e (test vertex) to 0
predecessors.
10 > 17:12:23,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying v1
(2/2) (attempt #0) to host3
6 > 17:12:23,236 DEBUG akka.event.EventStream
- shutting down: StandardOutLogger started
1 > 17:12:23,239 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Sending out
cancel request, to remove task execution from TaskManager.
1 > 17:12:23,244 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Concurrent
canceling/failing of TestVertex (1/1) - execution #0 while deployment was in
progress.
10 > 17:12:23,266 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Attaching 1
topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.61 sec -
in org.apache.flink.runtime.executiongraph.ExecutionVertexCancelTest
Running org.apache.flink.runtime.io.network.api.reader.UnionBufferReaderTest
10 > 17:12:23,269 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Connecting
ExecutionJobVertex e8869426e7765e498ad219e3af731729 (test vertex) to 0
predecessors.
10 > 17:12:23,269 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (1/2) - execution #0, sharingUnit=null,
locationConstraint=null}
10 > 17:12:23,270 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local
assignment: test vertex (1/2) --> ee9357ffc16d51c4f7522bc479a757ac @
akka://TestingActorSystem/user/$$a - 1 slots - 2073640037
10 > 17:12:23,270 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying test
vertex (1/2) (attempt #0) to host1
10 > 17:12:23,270 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (2/2) - execution #0, sharingUnit=null,
locationConstraint=null}
10 > 17:12:23,271 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local
assignment: test vertex (2/2) --> 6620d9f45b7567e2b147f5761d8060e8 @
akka://TestingActorSystem/user/$$a - 1 slots - 1864116663
10 > 17:12:23,271 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying test
vertex (2/2) (attempt #0) to host3
1 > 17:12:23,277 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Deploying
TestVertex (1/1) (attempt #0) to localhost
10 > 17:12:23,277 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph- Attaching 1
topologically sorted
[jira] [Created] (FLINK-1558) Spurious failure in PipelinedPartitionQueueTest
Stephan Ewen created FLINK-1558:
---
Summary: Spurious failure in PipelinedPartitionQueueTest
Key: FLINK-1558
URL: https://issues.apache.org/jira/browse/FLINK-1558
Project: Flink
Issue Type: Improvement
Components: Distributed Runtime
Affects Versions: 0.9
Reporter: Stephan Ewen
Fix For: 0.9
The failure is reported as
{code}
java.lang.AssertionError: Unexpected failure during test: null. Producer error:
null, consumer error: null
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.runtime.io.network.partition.queue.PipelinedPartitionQueueTest.doTestConcurrentProduceConsume(PipelinedPartitionQueueTest.java:214)
at
org.apache.flink.runtime.io.network.partition.queue.PipelinedPartitionQueueTest.doTestConcurrentProduceConsume(PipelinedPartitionQueueTest.java:171)
at
org.apache.flink.runtime.io.network.partition.queue.PipelinedPartitionQueueTest.testConcurrentProduceConsume(PipelinedPartitionQueueTest.java:142)
{code}
The test shows the following stack trace
{code}
java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at
org.apache.flink.runtime.io.network.partition.queue.PipelinedPartitionQueueTest.doTestConcurrentProduceConsume(PipelinedPartitionQueueTest.java:198)
at
org.apache.flink.runtime.io.network.partition.queue.PipelinedPartitionQueueTest.doTestConcurrentProduceConsume(PipelinedPartitionQueueTest.java:171)
at
org.apache.flink.runtime.io.network.partition.queue.PipelinedPartitionQueueTest.testConcurrentProduceConsume(PipelinedPartitionQueueTest.java:142)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[
https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322906#comment-14322906
]
ASF GitHub Bot commented on FLINK-1528:
---
Github user vasia commented on the pull request:
https://github.com/apache/flink/pull/400#issuecomment-74527767
Right, so let me copy:
The problem is here:
```
DataSet>> neighborhoods = input.reduceOnEdges(new
NeighborhoodEdgesFunction(), EdgeDirection.OUT);
```
and we try to get the return type `Tuple2>` like this:
```
public TypeInformation getProducedType() {
return TypeExtractor.createTypeInfo(EdgesFunction.class,
function.getClass(), 2, null, null);
}
```
Anyone have an idea? Is it because of the nested type parameter in
`Tuple2>` ?
> Add local clustering coefficient library method and example
> ---
>
> Key: FLINK-1528
> URL: https://issues.apache.org/jira/browse/FLINK-1528
> Project: Flink
> Issue Type: Task
> Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Daniel Bali
>
> Add a gelly library method and example to compute the local clustering
> coefficient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-1557) Move WebFrontend out of JobManager Actor
Stephan Ewen created FLINK-1557: --- Summary: Move WebFrontend out of JobManager Actor Key: FLINK-1557 URL: https://issues.apache.org/jira/browse/FLINK-1557 Project: Flink Issue Type: Improvement Components: JobManager Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 Currently, the Webfrontent is started inside the JobManager actor. I propose to move it out of the actor, since it is a heavy component that is independent of the actor and whose failure cannot be controlled by the job manager. The change is fairly straightforward, since the web frontend only communicates with the jobmanager through actor messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1528] Added local clustering coefficien...
Github user vasia commented on the pull request:
https://github.com/apache/flink/pull/400#issuecomment-74527767
Right, so let me copy:
The problem is here:
```
DataSet>> neighborhoods = input.reduceOnEdges(new
NeighborhoodEdgesFunction(), EdgeDirection.OUT);
```
and we try to get the return type `Tuple2>` like this:
```
public TypeInformation getProducedType() {
return TypeExtractor.createTypeInfo(EdgesFunction.class,
function.getClass(), 2, null, null);
}
```
Anyone have an idea? Is it because of the nested type parameter in
`Tuple2>` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1515) [Gelly] Enable access to aggregators and broadcast sets in vertex-centric iteration
[
https://issues.apache.org/jira/browse/FLINK-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322886#comment-14322886
]
ASF GitHub Bot commented on FLINK-1515:
---
Github user vasia commented on the pull request:
https://github.com/apache/flink/pull/402#issuecomment-74525590
Hi @martinkiefer! Thanks a lot for this pull request!
The change looks good :)
I was only thinking that it would be nice to have an example that
demonstrates how to use broadcast sets and / or aggregators now. It could
simply be a slight modification of an existing example, e.g. SSSP with a
counter of how many nodes have distance less than some value or similar. Could
you make this addition?
> [Gelly] Enable access to aggregators and broadcast sets in vertex-centric
> iteration
> ---
>
> Key: FLINK-1515
> URL: https://issues.apache.org/jira/browse/FLINK-1515
> Project: Flink
> Issue Type: Improvement
> Components: Gelly
>Reporter: Vasia Kalavri
>Assignee: Martin Kiefer
>
> Currently, aggregators and broadcast sets cannot be accessed through Gelly's
> {{runVertexCentricIteration}} method. The functionality is already present in
> the {{VertexCentricIteration}} and we just need to expose it.
> This could be done like this: We create a method
> {{createVertexCentricIteration}}, which will return a
> {{VertexCentricIteration}} object and we change {{runVertexCentricIteration}}
> to accept this as a parameter (and return the graph after running this
> iteration).
> The user can configure the {{VertexCentricIteration}} by directly calling the
> public methods {{registerAggregator}}, {{setName}}, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1515]Splitted runVertexCentricIteration...
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/402#issuecomment-74525590 Hi @martinkiefer! Thanks a lot for this pull request! The change looks good :) I was only thinking that it would be nice to have an example that demonstrates how to use broadcast sets and / or aggregators now. It could simply be a slight modification of an existing example, e.g. SSSP with a counter of how many nodes have distance less than some value or similar. Could you make this addition? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1556] Corrects faulty JobClient behavio...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/406 [FLINK-1556] Corrects faulty JobClient behaviour in case of a submission failure If an error occurred during job submission, a ```SubmissionFailure``` is sent to the ```JobClient```. As a reaction, the ```JobClient``` terminated itself and sent the failure to the ```Client```. However, this does not necessarily mean that the job has reached a terminal state, because the failing procedure is executed asynchronously. The ```JobClient``` now waits until it receives a ```JobResult``` message indicating that the job has completed and all resources are properly returned. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink minorFixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/406.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #406 commit 2f32e9c6b87b8e295f792c04306d78fbb858f80d Author: Till Rohrmann Date: 2015-02-16T09:17:21Z [FLINK-1556] [runtime] Corrects faulty JobClient behaviour in case of a submission failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception
[
https://issues.apache.org/jira/browse/FLINK-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322884#comment-14322884
]
ASF GitHub Bot commented on FLINK-1556:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/406
[FLINK-1556] Corrects faulty JobClient behaviour in case of a submission
failure
If an error occurred during job submission, a ```SubmissionFailure``` is
sent to the ```JobClient```. As a reaction, the ```JobClient``` terminated
itself and sent the failure to the ```Client```. However, this does not
necessarily mean that the job has reached a terminal state, because the failing
procedure is executed asynchronously.
The ```JobClient``` now waits until it receives a ```JobResult``` message
indicating that the job has completed and all resources are properly returned.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink minorFixes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/406.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #406
commit 2f32e9c6b87b8e295f792c04306d78fbb858f80d
Author: Till Rohrmann
Date: 2015-02-16T09:17:21Z
[FLINK-1556] [runtime] Corrects faulty JobClient behaviour in case of a
submission failure
> JobClient does not wait until a job failed completely if submission exception
> -
>
> Key: FLINK-1556
> URL: https://issues.apache.org/jira/browse/FLINK-1556
> Project: Flink
> Issue Type: Bug
>Reporter: Till Rohrmann
>
> If an exception occurs during job submission the {{JobClient}} received a
> {{SubmissionFailure}}. Upon receiving this message, the {{JobClient}}
> terminates itself and returns the error to the {{Client}}. This indicates to
> the user that the job has been completely failed which is not necessarily
> true.
> If the user directly after such a failure submits another job, then it might
> be the case that not all slots of the formerly failed job are returned. This
> can lead to a {{NoRessourceAvailableException}}.
> We can solve this problem by waiting for the completion of the job failure in
> the {{JobClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-1556) JobClient does not wait until a job failed completely if submission exception
Till Rohrmann created FLINK-1556:
Summary: JobClient does not wait until a job failed completely if
submission exception
Key: FLINK-1556
URL: https://issues.apache.org/jira/browse/FLINK-1556
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
If an exception occurs during job submission the {{JobClient}} received a
{{SubmissionFailure}}. Upon receiving this message, the {{JobClient}}
terminates itself and returns the error to the {{Client}}. This indicates to
the user that the job has been completely failed which is not necessarily true.
If the user directly after such a failure submits another job, then it might be
the case that not all slots of the formerly failed job are returned. This can
lead to a {{NoRessourceAvailableException}}.
We can solve this problem by waiting for the completion of the job failure in
the {{JobClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1554) LocalFlinkMiniCluster cannot start multiple local TaskManagers
[
https://issues.apache.org/jira/browse/FLINK-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322860#comment-14322860
]
ASF GitHub Bot commented on FLINK-1554:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/403#issuecomment-74523049
Perfect. I've justed the code and its working now.
> LocalFlinkMiniCluster cannot start multiple local TaskManagers
> --
>
> Key: FLINK-1554
> URL: https://issues.apache.org/jira/browse/FLINK-1554
> Project: Flink
> Issue Type: Bug
>Reporter: Till Rohrmann
>
> The {{LocalFlinkMiniCluster}} cannot currently start multiple
> {{TaskManager}}s in the same {{ActorSystem}}, because it assigns them the
> same actor name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1554] Allows the LocalFlinkMiniCluster ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/403#issuecomment-74523049 Perfect. I've justed the code and its working now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1555) Add utility to log the serializers of composite types
Robert Metzger created FLINK-1555:
-
Summary: Add utility to log the serializers of composite types
Key: FLINK-1555
URL: https://issues.apache.org/jira/browse/FLINK-1555
Project: Flink
Issue Type: Improvement
Reporter: Robert Metzger
Priority: Minor
Users affected by poor performance might want to understand how Flink is
serializing their data.
Therefore, it would be cool to have a tool utility which logs the serializers
like this:
{{SerializerUtils.getSerializers(TypeInformation t);}}
to get
{code}
PojoSerializer
TupleSerializer
IntSer
DateSer
GenericTypeSer(java.sql.Date)
PojoSerializer
GenericTypeSer(HashMap)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-947) Add support for "Named Datasets"
[
https://issues.apache.org/jira/browse/FLINK-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322816#comment-14322816
]
ASF GitHub Bot commented on FLINK-947:
--
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/405
[FLINK-947] Add a declarative expression API
This one is quite big. So you should check out the documentation, skaldic,
examples and test cases to see how the API works.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink linq
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/405.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #405
commit 147525ced43db6690a64fbae1395dbd258b8901d
Author: Aljoscha Krettek
Date: 2014-10-03T16:25:15Z
Change translateToDataflow to return Operator
Before, translateToDataflow of SingleInputOperator could only return
a single input operator of the lower layer, same for TwoInputOperator.
This change allows translateToDataflow to return more kinds of
operators.
commit 58b5b9ec6e65855bfd71287deb6352dfc4498451
Author: Aljoscha Krettek
Date: 2014-10-23T16:09:38Z
Add methods to CompositeType for querying field types and names
commit ac29ee3ad36a72d7c41549f38da1a00e66d85041
Author: Aljoscha Krettek
Date: 2014-10-01T11:12:18Z
[FLINK-947] Add a declarative expression API
> Add support for "Named Datasets"
>
>
> Key: FLINK-947
> URL: https://issues.apache.org/jira/browse/FLINK-947
> Project: Flink
> Issue Type: New Feature
> Components: Java API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> This would create an API that is a mix between SQL like declarativity and the
> power of user defined functions. Example user code could look like this:
> {code:Java}
> NamedDataSet one = ...
> NamedDataSet two = ...
> NamedDataSet result = one.join(two).where("key").equalTo("otherKey")
> .project("a", "b", "c")
> .map( (UserTypeIn in) -> return new UserTypeOut(...) )
> .print();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-947] Add a declarative expression API
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/405 [FLINK-947] Add a declarative expression API This one is quite big. So you should check out the documentation, skaldic, examples and test cases to see how the API works. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink linq Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/405.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #405 commit 147525ced43db6690a64fbae1395dbd258b8901d Author: Aljoscha Krettek Date: 2014-10-03T16:25:15Z Change translateToDataflow to return Operator Before, translateToDataflow of SingleInputOperator could only return a single input operator of the lower layer, same for TwoInputOperator. This change allows translateToDataflow to return more kinds of operators. commit 58b5b9ec6e65855bfd71287deb6352dfc4498451 Author: Aljoscha Krettek Date: 2014-10-23T16:09:38Z Add methods to CompositeType for querying field types and names commit ac29ee3ad36a72d7c41549f38da1a00e66d85041 Author: Aljoscha Krettek Date: 2014-10-01T11:12:18Z [FLINK-947] Add a declarative expression API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-1539) Runtime context not initialized when running streaming PojoExample
[ https://issues.apache.org/jira/browse/FLINK-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi closed FLINK-1539. - Resolution: Fixed Fixed via 4470207 > Runtime context not initialized when running streaming PojoExample > -- > > Key: FLINK-1539 > URL: https://issues.apache.org/jira/browse/FLINK-1539 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 0.9 >Reporter: Márton Balassi >Assignee: Gyula Fora > > When running streaming PojoExample received the following exception: > Exception in thread "main" java.lang.IllegalStateException: The runtime > context has not been initialized. > at > org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49) > at > org.apache.flink.streaming.api.function.aggregation.SumAggregator$PojoSumAggregator.(SumAggregator.java:149) > at > org.apache.flink.streaming.api.function.aggregation.SumAggregator.getSumFunction(SumAggregator.java:52) > at > org.apache.flink.streaming.api.datastream.DataStream.sum(DataStream.java:632) > at > org.apache.flink.streaming.examples.wordcount.PojoExample.main(PojoExample.java:65) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1176) Add operator for global reduce/aggregations
[ https://issues.apache.org/jira/browse/FLINK-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi closed FLINK-1176. - Resolution: Implemented Implemented via 1146f64 and c560d76. > Add operator for global reduce/aggregations > --- > > Key: FLINK-1176 > URL: https://issues.apache.org/jira/browse/FLINK-1176 > Project: Flink > Issue Type: Improvement > Components: Streaming >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Minor > > Currently streaming reduce operators (like .reduce or .window(..).reduce, > sum, etc.) only allow local reduces/aggregations. While one can achieve > global aggregation by setting the parallelism of these operators to 1 it can > cause a serious bottleneck in the program. > For window/batch reduces and aggregations a global reducer can be added, with > parallelism 1, to further reduce partial results. This global reduce option > could be introduced as an optional boolean parameter for the intended > operators. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1231) Add test for streaming remote execution
[ https://issues.apache.org/jira/browse/FLINK-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi closed FLINK-1231. - Resolution: Fixed Implemented via 26ae979 > Add test for streaming remote execution > --- > > Key: FLINK-1231 > URL: https://issues.apache.org/jira/browse/FLINK-1231 > Project: Flink > Issue Type: Test > Components: Streaming >Affects Versions: 0.7.1-incubating >Reporter: Márton Balassi > > Add a test case that mocks the remote job execution. > PackagedProgramEndToEndITCase serves as an example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1231) Add test for streaming remote execution
[ https://issues.apache.org/jira/browse/FLINK-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322808#comment-14322808 ] ASF GitHub Bot commented on FLINK-1231: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/303 > Add test for streaming remote execution > --- > > Key: FLINK-1231 > URL: https://issues.apache.org/jira/browse/FLINK-1231 > Project: Flink > Issue Type: Test > Components: Streaming >Affects Versions: 0.7.1-incubating >Reporter: Márton Balassi > > Add a test case that mocks the remote job execution. > PackagedProgramEndToEndITCase serves as an example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: StreamWindow abstraction + modular window comp...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/395 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1231] Add test for streaming remote exe...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1454) CliFrontend blocks for 100 seconds when submitting to a non-existent JobManager
[
https://issues.apache.org/jira/browse/FLINK-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322803#comment-14322803
]
ASF GitHub Bot commented on FLINK-1454:
---
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/404
[FLINK-1454] [job client] Improve error handling for failed connections
between JobClient and JobManager
- Decreases time until connection failure is discovered.
- Improves error messages
- adds / extends tests
- Removes implicits in runtime, makes actor system and timeout always an
explicit parameter
- Removes old redundant config constants
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink client_connect
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/404.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #404
commit 9365ba8cb3859d7a04a301d3417e418914ce813c
Author: Stephan Ewen
Date: 2015-02-16T12:37:22Z
[FLINK-1454] [job client] Improve error handling for failed connections
between JobClient and JobManager
> CliFrontend blocks for 100 seconds when submitting to a non-existent
> JobManager
> ---
>
> Key: FLINK-1454
> URL: https://issues.apache.org/jira/browse/FLINK-1454
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 0.9
>Reporter: Robert Metzger
>
> When a user tries to submit a job to a job manager which doesn't exist at
> all, the CliFrontend blocks for 100 seconds.
> Ideally, Akka would fail because it can not connect to the given
> hostname:port.
>
> {code}
> ./bin/flink run ./examples/flink-java-examples-0.9-SNAPSHOT-WordCount.jar -c
> foo.Baz
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:449)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:350)
> at org.apache.flink.client.program.Client.run(Client.java:242)
> at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:389)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:362)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1078)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1102)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [100 seconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265)
> at
> org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:169)
> at
> org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
> at org.apache.flink.client.program.Client.run(Client.java:314)
> at org.apache.flink.client.program.Client.run(Client.java:296)
> at org.apache.flink.client.program.Client.run(Client.java:290)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:434)
> ... 6 more
> The exception above occurred while trying to run your command.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1454] [job client] Improve error handli...
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/404 [FLINK-1454] [job client] Improve error handling for failed connections between JobClient and JobManager - Decreases time until connection failure is discovered. - Improves error messages - adds / extends tests - Removes implicits in runtime, makes actor system and timeout always an explicit parameter - Removes old redundant config constants You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink client_connect Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/404.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #404 commit 9365ba8cb3859d7a04a301d3417e418914ce813c Author: Stephan Ewen Date: 2015-02-16T12:37:22Z [FLINK-1454] [job client] Improve error handling for failed connections between JobClient and JobManager --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1554) LocalFlinkMiniCluster cannot start multiple local TaskManagers
[
https://issues.apache.org/jira/browse/FLINK-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322797#comment-14322797
]
ASF GitHub Bot commented on FLINK-1554:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/403
[FLINK-1554] Allows the LocalFlinkMiniCluster to start multiple TaskManager
in the same ActorSystem
Allows the ```LocalFlinkMiniCluster``` to start multiple ```TaskManager```
in the same ```ActorSystem``` by assigning them unique actor names.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink fixLocalFlinkMiniCluster
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/403.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #403
commit cecb45d45f05c0aa86f4ae780e8b49830f3ea6ac
Author: Till Rohrmann
Date: 2015-02-16T14:04:42Z
[FLINK-1554] [runtime] Allows the LocalFlinkMiniCluster to start multiple
TaskManager in the same ActorSystem.
> LocalFlinkMiniCluster cannot start multiple local TaskManagers
> --
>
> Key: FLINK-1554
> URL: https://issues.apache.org/jira/browse/FLINK-1554
> Project: Flink
> Issue Type: Bug
>Reporter: Till Rohrmann
>
> The {{LocalFlinkMiniCluster}} cannot currently start multiple
> {{TaskManager}}s in the same {{ActorSystem}}, because it assigns them the
> same actor name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1554] Allows the LocalFlinkMiniCluster ...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/403 [FLINK-1554] Allows the LocalFlinkMiniCluster to start multiple TaskManager in the same ActorSystem Allows the ```LocalFlinkMiniCluster``` to start multiple ```TaskManager``` in the same ```ActorSystem``` by assigning them unique actor names. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink fixLocalFlinkMiniCluster Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/403.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #403 commit cecb45d45f05c0aa86f4ae780e8b49830f3ea6ac Author: Till Rohrmann Date: 2015-02-16T14:04:42Z [FLINK-1554] [runtime] Allows the LocalFlinkMiniCluster to start multiple TaskManager in the same ActorSystem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1554) LocalFlinkMiniCluster cannot start multiple local TaskManagers
Till Rohrmann created FLINK-1554:
Summary: LocalFlinkMiniCluster cannot start multiple local
TaskManagers
Key: FLINK-1554
URL: https://issues.apache.org/jira/browse/FLINK-1554
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
The {{LocalFlinkMiniCluster}} cannot currently start multiple {{TaskManager}}s
in the same {{ActorSystem}}, because it assigns them the same actor name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1542) Test case at BlobUtilsTest should not assume user could not create new item in root directory
[ https://issues.apache.org/jira/browse/FLINK-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322791#comment-14322791 ] ASF GitHub Bot commented on FLINK-1542: --- Github user uce commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74511789 Thanks, Henry. The change looks good to me. :+1: > Test case at BlobUtilsTest should not assume user could not create new item > in root directory > - > > Key: FLINK-1542 > URL: https://issues.apache.org/jira/browse/FLINK-1542 > Project: Flink > Issue Type: Bug >Reporter: Henry Saputra >Assignee: Henry Saputra > > The test at BlobUtilsTest #testExceptionOnCreateStorageDirectoryFailure > assume that current user cannot create new directory at root. > In my MacOSX, my user has admin group so the test will always fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1542] Test case at BlobUtilsTest should...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74511789 Thanks, Henry. The change looks good to me. :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1549) Add proper exception handling for YarnJobManager
[ https://issues.apache.org/jira/browse/FLINK-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322787#comment-14322787 ] ASF GitHub Bot commented on FLINK-1549: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/397#issuecomment-74511630 Looks good. Very much needed for our YARN deployment. > Add proper exception handling for YarnJobManager > > > Key: FLINK-1549 > URL: https://issues.apache.org/jira/browse/FLINK-1549 > Project: Flink > Issue Type: Bug >Reporter: Till Rohrmann > > The YarnJobManager actor thread calls methods which can throw an exception. > These exceptions should be caught and properly handled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1549] Adds proper exception handling to...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/397#issuecomment-74511630 Looks good. Very much needed for our YARN deployment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1552) Allow secondary sorts in Vertex Centric Iteration
[ https://issues.apache.org/jira/browse/FLINK-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322766#comment-14322766 ] Vasia Kalavri commented on FLINK-1552: -- This would be very useful to have! Would you like to work on it [~martinkiefer]? > Allow secondary sorts in Vertex Centric Iteration > - > > Key: FLINK-1552 > URL: https://issues.apache.org/jira/browse/FLINK-1552 > Project: Flink > Issue Type: Wish > Components: Gelly >Reporter: Martin Kiefer >Priority: Minor > > The `VertexCentricIteration` class holds the logic to transform a > `VertexUpdateFunction` and a `MessagingFunction` into an iteration with two > CoGroup operators working on the set of messages and edges. Graph algorithms > can profit from implying an order on the edges or messages based on their > value and/or the vertex ID. This can be implemented easily making use of > secondary sorts. I would suggest extending the `VertexCentricIteration` to > allow to specify these kind of orderings optionally. > For example, this comes handy when it is necessary to find the edges with the > minimum or maximum value or the algorithm requires to pick the edge with > lower vertex ID for edges with equal value. Similar use cases might be found > for orders on the messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1417) Automatically register nested types at Kryo
[
https://issues.apache.org/jira/browse/FLINK-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322743#comment-14322743
]
ASF GitHub Bot commented on FLINK-1417:
---
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/393#issuecomment-74505787
Looks good except for the one remark I had.
Did you run any performance tests on this? What is the overhead of using an
Avro POJO vs. a plain POJO vs. using the Kryo Serializer?
> Automatically register nested types at Kryo
> ---
>
> Key: FLINK-1417
> URL: https://issues.apache.org/jira/browse/FLINK-1417
> Project: Flink
> Issue Type: Improvement
> Components: Java API, Scala API
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
> Fix For: 0.9
>
>
> Currently, the {{GenericTypeInfo}} registers the class of the type at Kryo.
> In order to get the best performance, it should recursively walk the classes
> and make sure that it registered all contained subtypes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322741#comment-14322741 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani commented on the pull request: https://github.com/apache/flink/pull/400#issuecomment-74505722 Hi! Sorry for the messed up commit history, hopefully my next PR will be better. I have the following exception when running `LocalClusteringCoefficientExample` Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class flink.graphs.library.LocalClusteringCoefficient $NeighborhoodEdgesFunction' could not be determined. This is most likely a type erasure problem. The type extraction currently supports types with generic variables only in cases where all variables in the return type can be deduced from the input type(s). I have no idea how to fix this though. For anyone else reading this, here's @vasia's reply from the old `flink-graph` thread: https://github.com/project-flink/flink-graph/pull/48#issuecomment-70472784 > Add local clustering coefficient library method and example > --- > > Key: FLINK-1528 > URL: https://issues.apache.org/jira/browse/FLINK-1528 > Project: Flink > Issue Type: Task > Components: Gelly >Reporter: Vasia Kalavri >Assignee: Daniel Bali > > Add a gelly library method and example to compute the local clustering > coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1417] Automatically register types with...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/393#issuecomment-74505787 Looks good except for the one remark I had. Did you run any performance tests on this? What is the overhead of using an Avro POJO vs. a plain POJO vs. using the Kryo Serializer? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1528] Added local clustering coefficien...
Github user balidani commented on the pull request: https://github.com/apache/flink/pull/400#issuecomment-74505722 Hi! Sorry for the messed up commit history, hopefully my next PR will be better. I have the following exception when running `LocalClusteringCoefficientExample` Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class flink.graphs.library.LocalClusteringCoefficient $NeighborhoodEdgesFunction' could not be determined. This is most likely a type erasure problem. The type extraction currently supports types with generic variables only in cases where all variables in the return type can be deduced from the input type(s). I have no idea how to fix this though. For anyone else reading this, here's @vasia's reply from the old `flink-graph` thread: https://github.com/project-flink/flink-graph/pull/48#issuecomment-70472784 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1417) Automatically register nested types at Kryo
[
https://issues.apache.org/jira/browse/FLINK-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322736#comment-14322736
]
ASF GitHub Bot commented on FLINK-1417:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/393#discussion_r24749284
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -292,7 +360,76 @@ public void registerKryoType(Class type) {
/**
* Returns the registered POJO types.
*/
- public Set> getRegisteredPojoTypes() {
+ public List> getRegisteredPojoTypes() {
return registeredPojoTypes;
}
+
+
+ public boolean isDisableAutoTypeRegistration() {
+ return disableAutoTypeRegistration;
+ }
+
+ /**
+* Control whether Flink is automatically registering all types in the
user programs with
+* Kryo.
+*
+* @param disableAutoTypeRegistration
+*/
+ public void setDisableAutoTypeRegistration(boolean
disableAutoTypeRegistration) {
--- End diff --
I would prefer "disableAutoTypeRegistration" here. And then
"isAutoTypeRegistrationDisabled", above.
> Automatically register nested types at Kryo
> ---
>
> Key: FLINK-1417
> URL: https://issues.apache.org/jira/browse/FLINK-1417
> Project: Flink
> Issue Type: Improvement
> Components: Java API, Scala API
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
> Fix For: 0.9
>
>
> Currently, the {{GenericTypeInfo}} registers the class of the type at Kryo.
> In order to get the best performance, it should recursively walk the classes
> and make sure that it registered all contained subtypes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1417] Automatically register types with...
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/393#discussion_r24749284
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java ---
@@ -292,7 +360,76 @@ public void registerKryoType(Class type) {
/**
* Returns the registered POJO types.
*/
- public Set> getRegisteredPojoTypes() {
+ public List> getRegisteredPojoTypes() {
return registeredPojoTypes;
}
+
+
+ public boolean isDisableAutoTypeRegistration() {
+ return disableAutoTypeRegistration;
+ }
+
+ /**
+* Control whether Flink is automatically registering all types in the
user programs with
+* Kryo.
+*
+* @param disableAutoTypeRegistration
+*/
+ public void setDisableAutoTypeRegistration(boolean
disableAutoTypeRegistration) {
--- End diff --
I would prefer "disableAutoTypeRegistration" here. And then
"isAutoTypeRegistrationDisabled", above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Fix standalone plan visualizer for 0.8 branch
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/401 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-1362) Spurious errors in execution state messaging between TaskManager and JobManager
[
https://issues.apache.org/jira/browse/FLINK-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-1362.
Resolution: Fixed
Fixed via 4046819b380b8dfa57d52c6d314f389546a159a3
> Spurious errors in execution state messaging between TaskManager and
> JobManager
> ---
>
> Key: FLINK-1362
> URL: https://issues.apache.org/jira/browse/FLINK-1362
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>
> In various tasks, it is possible that a notification from the TaskManager to
> the JobManager fails with the following exception:
> {code}
> Exception in thread "" java.lang.NullPointerException
> at
> org.apache.flink.runtime.taskmanager.TaskManager.notifyExecutionStateChange(TaskManager.scala:281)
> at org.apache.flink.runtime.taskmanager.Task.markFailed(Task.java:199)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:307)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1542) Test case at BlobUtilsTest should not assume user could not create new item in root directory
[ https://issues.apache.org/jira/browse/FLINK-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322699#comment-14322699 ] ASF GitHub Bot commented on FLINK-1542: --- Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74500203 Ping for review? > Test case at BlobUtilsTest should not assume user could not create new item > in root directory > - > > Key: FLINK-1542 > URL: https://issues.apache.org/jira/browse/FLINK-1542 > Project: Flink > Issue Type: Bug >Reporter: Henry Saputra >Assignee: Henry Saputra > > The test at BlobUtilsTest #testExceptionOnCreateStorageDirectoryFailure > assume that current user cannot create new directory at root. > In my MacOSX, my user has admin group so the test will always fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1542] Test case at BlobUtilsTest should...
Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/396#issuecomment-74500203 Ping for review? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322649#comment-14322649 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/400#issuecomment-74493011 Hi @balidani! Thanks for the PR. The commit history is a bit messed up, but we'll fix that. It's just the last commit that's relevant, right? If I recall correctly, you are having some type issues. Could you please give us some details on what is the error you are getting and what seems to be causing it? Thanks! > Add local clustering coefficient library method and example > --- > > Key: FLINK-1528 > URL: https://issues.apache.org/jira/browse/FLINK-1528 > Project: Flink > Issue Type: Task > Components: Gelly >Reporter: Vasia Kalavri >Assignee: Daniel Bali > > Add a gelly library method and example to compute the local clustering > coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1528] Added local clustering coefficien...
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/400#issuecomment-74493011 Hi @balidani! Thanks for the PR. The commit history is a bit messed up, but we'll fix that. It's just the last commit that's relevant, right? If I recall correctly, you are having some type issues. Could you please give us some details on what is the error you are getting and what seems to be causing it? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1231] Add test for streaming remote exe...
Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/303#issuecomment-74480487 Sorry for the late reply. Thanks for the great PR, I wanted to merge it for quite some time. It turns out that this test is redundant given the 6bec228 I've pushed last week. As a result of that I would like to close this PR without merging your commit. Sorry, it is on me: I've just recently realized that the difference between submitting through the `Client` and the `PackagedProgram` have very little differences. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1231) Add test for streaming remote execution
[ https://issues.apache.org/jira/browse/FLINK-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322565#comment-14322565 ] ASF GitHub Bot commented on FLINK-1231: --- Github user mbalassi commented on the pull request: https://github.com/apache/flink/pull/303#issuecomment-74480487 Sorry for the late reply. Thanks for the great PR, I wanted to merge it for quite some time. It turns out that this test is redundant given the 6bec228 I've pushed last week. As a result of that I would like to close this PR without merging your commit. Sorry, it is on me: I've just recently realized that the difference between submitting through the `Client` and the `PackagedProgram` have very little differences. > Add test for streaming remote execution > --- > > Key: FLINK-1231 > URL: https://issues.apache.org/jira/browse/FLINK-1231 > Project: Flink > Issue Type: Test > Components: Streaming >Affects Versions: 0.7.1-incubating >Reporter: Márton Balassi > > Add a test case that mocks the remote job execution. > PackagedProgramEndToEndITCase serves as an example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1551) Fix History view of JobManager web interface
[
https://issues.apache.org/jira/browse/FLINK-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-1551.
Resolution: Fixed
Fixed via 055f6dc3eb8705dbbfb04e44965a85cbfc0b10db
> Fix History view of JobManager web interface
>
>
> Key: FLINK-1551
> URL: https://issues.apache.org/jira/browse/FLINK-1551
> Project: Flink
> Issue Type: Bug
> Components: JobManager
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Priority: Critical
>
> The JobManager web history view is not working.
> {{scala.concurrent.impl.Promise$DefaultPromise cannot be cast to
> org.apache.flink.runtime.messages.JobManagerMessages$JobResponse}}
> According to git blame, the code has been touched in FLINK-1508.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: StreamWindow abstraction + modular window comp...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/395#discussion_r24737554
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/StreamWindowTypeInfo.java
---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUStreamRecordWARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.windowing;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+
+public class StreamWindowTypeInfo extends
TypeInformation> {
+
+ private static final long serialVersionUID = 1L;
+ TypeInformation innerType;
+
+ public StreamWindowTypeInfo(TypeInformation innerType) {
+ this.innerType = innerType;
+ }
+
+ public TypeInformation getInnerType() {
+ return innerType;
+ }
+
+ @Override
+ public boolean isBasicType() {
+ return innerType.isBasicType();
+ }
+
+ @Override
+ public boolean isTupleType() {
+ return innerType.isTupleType();
+ }
+
+ @Override
+ public int getArity() {
+ return innerType.getArity();
+ }
+
+ @Override
+ public Class> getTypeClass() {
+ // TODO Auto-generated method stub
--- End diff --
hehe
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: StreamWindow abstraction + modular window comp...
Github user mbalassi commented on a diff in the pull request: https://github.com/apache/flink/pull/395#discussion_r24737330 --- Diff: docs/streaming_guide.md --- @@ -857,7 +895,7 @@ After installing Docker an image can be pulled for each connector. Containers ca For the easiest set up create a jar with all the dependencies of the *flink-streaming-connectors* project. ~~~batch -cd /PATH/TO/GIT/flink/flink-staging/flink-streaming-connectors +cd /PATH/TO/GIT/incubator-flink/flink-addons/flink-streaming-connectors mvn assembly:assembly --- End diff -- An amend here would be nice. :) I'll do it while merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: StreamWindow abstraction + modular window comp...
Github user mbalassi commented on a diff in the pull request: https://github.com/apache/flink/pull/395#discussion_r24737233 --- Diff: docs/streaming_guide.md --- @@ -383,9 +389,40 @@ dataStream.groupBy(groupingField).window(Count.of(100)).every(â¦).max(field); This will create separate windows for different keys and apply the trigger and eviction policies on a per group basis. + Applying multiple transformations on a window +Using the `WindowedDataStream` abstraction we can apply several transformations one after another on the discretised streams without having to re-discretise it: + +~~~java +dataStream.window(Count.of(1000)).groupBy(firstKey).mapWindow(â¦).groupBy(secondKey).reduceWindow(â¦).flatten() +~~~ + +The above call would create global windows of 1000 elements group it by the first key and then apply a mapWindow transformation. The resulting windowed stream will then be grouped by the second key and further reduced. The results of the reduce transformation are then flattened. --- End diff -- Nice example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1553) Kafka connectors rework
Gábor Hermann created FLINK-1553: Summary: Kafka connectors rework Key: FLINK-1553 URL: https://issues.apache.org/jira/browse/FLINK-1553 Project: Flink Issue Type: Bug Components: Streaming Reporter: Gábor Hermann Assignee: Gábor Hermann The Kafka connectors should be reworked and tested because it is not fully functioning (e.g. the source can only read partitions from the start). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
