[jira] [Updated] (FLINK-1404) Add support to cache intermediate results
[ https://issues.apache.org/jira/browse/FLINK-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi updated FLINK-1404: --- Summary: Add support to cache intermediate results (was: Trigger recycling of buffers held by historic intermediate result partitions) Add support to cache intermediate results - Key: FLINK-1404 URL: https://issues.apache.org/jira/browse/FLINK-1404 Project: Flink Issue Type: Improvement Components: Distributed Runtime Reporter: Ufuk Celebi Assignee: Ufuk Celebi With blocking intermediate results (FLINK-1350) and proper partition state management (FLINK-1359) it is necessary to allow the network buffer pool to request eviction of historic intermediate results when not enough buffers are available. With the currently available pipelined intermediate partitions this is not an issue, because buffer pools can be released as soon as a partition is consumed. We need to be able to trigger the recycling of buffers held by historic intermediate results when not enough buffers are available for new local pools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-1831) runtime.taskmanager.RegistrationTests fails sporiously
Márton Balassi created FLINK-1831: - Summary: runtime.taskmanager.RegistrationTests fails sporiously Key: FLINK-1831 URL: https://issues.apache.org/jira/browse/FLINK-1831 Project: Flink Issue Type: Bug Components: Distributed Runtime Affects Versions: 0.9 Reporter: Márton Balassi Failing in a decent chunk of travis builds, I could also reproduce it locally: akka.actor.InvalidActorNameException: actor name [FAKE_JOB_MANAGER] is not unique! at akka.actor.dungeon.ChildrenContainer$TerminatingChildrenContainer.reserve(ChildrenContainer.scala:192) at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) at akka.actor.ActorCell.reserveChild(ActorCell.scala:369) at akka.actor.dungeon.Children$class.makeChild(Children.scala:202) at akka.actor.dungeon.Children$class.attachChild(Children.scala:42) at akka.actor.ActorCell.attachChild(ActorCell.scala:369) at akka.actor.ActorSystemImpl.actorOf(ActorSystem.scala:553) at org.apache.flink.runtime.taskmanager.RegistrationTest$5.init(RegistrationTest.java:308) at org.apache.flink.runtime.taskmanager.RegistrationTest.testTaskManagerResumesConnectAfterJobManagerFailure(RegistrationTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) 09:46:39.486 [flink-akka.actor.default-dispatcher-2] [flink-akka.actor.default-dispatcher-3 - akka://flink/user/FAKE_JOB_MANAGER] ERROR akka.actor.OneForOneStrategy - Kill akka.actor.ActorKilledException: Kill java.lang.AssertionError: actor name [FAKE_JOB_MANAGER] is not unique! at org.junit.Assert.fail(Assert.java:88) at org.apache.flink.runtime.taskmanager.RegistrationTest$5.init(RegistrationTest.java:328) at org.apache.flink.runtime.taskmanager.RegistrationTest.testTaskManagerResumesConnectAfterJobManagerFailure(RegistrationTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
[jira] [Commented] (FLINK-1829) Conflicting Jackson version in the Flink POMs
[
https://issues.apache.org/jira/browse/FLINK-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482808#comment-14482808
]
Stephan Ewen commented on FLINK-1829:
-
The `hbase-client` dependency should not be in there by default, or is it
pulled in transitively?
For the others, I think we need to add an entry to the dependency management.
Conflicting Jackson version in the Flink POMs
-
Key: FLINK-1829
URL: https://issues.apache.org/jira/browse/FLINK-1829
Project: Flink
Issue Type: Bug
Components: Build System
Affects Versions: 0.9
Reporter: Alexander Alexandrov
Fix For: 0.9
The current POM setup transitively includes multiple conflicting versions of
the Jackson library over
* {{com.amazonaws:aws-java-sdk}} (v. 2.1.1)
* {{org.apache.avro:avro}} (v. 1.9.13)
* {{org.apache.hbase:hbase-client}} (v. 1.8.8)
When running jobs against a Flink local runtime embedded with:
{code:xml}
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-scala/artifactId
version${flink.version}/version
scopeprovided/scope
/dependency
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-java/artifactId
version${flink.version}/version
scopeprovided/scope
/dependency
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-clients/artifactId
version${flink.version}/version
scopeprovided/scope
/dependency
{code}
I get the following error:
{noformat}
15-04-04 15:52:04 ERROR exception during creation
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at akka.util.Reflect$.instantiate(Reflect.scala:66)
at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
at akka.actor.Props.newActor(Props.scala:252)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 9 more
Caused by: java.lang.NoSuchMethodError:
com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
at
com.fasterxml.jackson.databind.ObjectMapper.init(ObjectMapper.java:445)
at
com.fasterxml.jackson.databind.ObjectMapper.init(ObjectMapper.java:366)
at
org.apache.flink.runtime.taskmanager.TaskManager.init(TaskManager.scala:134)
... 18 more
{noformat}
Fixing the Jackson version on the client side, e.g, with the following snippet
{code:xml}
dependency
groupIdcom.fasterxml.jackson.core/groupId
artifactIdjackson-core/artifactId
version2.2.1/version
scopeprovided/scope
/dependency
dependency
groupIdcom.fasterxml.jackson.core/groupId
artifactIdjackson-databind/artifactId
version2.2.1/version
scopeprovided/scope
/dependency
dependency
groupIdcom.fasterxml.jackson.core/groupId
artifactIdjackson-annotations/artifactId
version2.2.1/version
scopeprovided/scope
/dependency
{code}
solves the problem, but I guess it will be better if we can stick with one
version in the build artifacts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-967) Make intermediate results a first-class citizen in the JobGraph
[ https://issues.apache.org/jira/browse/FLINK-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482816#comment-14482816 ] Maximilian Michels commented on FLINK-967: -- The intermediate results are now integrated into the JobGraph. We are currently working on the runtime parts. Make intermediate results a first-class citizen in the JobGraph --- Key: FLINK-967 URL: https://issues.apache.org/jira/browse/FLINK-967 Project: Flink Issue Type: New Feature Components: JobManager, TaskManager Affects Versions: 0.6-incubating Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 In order to add incremental plan rollout to the system, we need to make intermediate results a first-class citizen in the job graph and scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1831) runtime.taskmanager.RegistrationTests fails spuriously
[ https://issues.apache.org/jira/browse/FLINK-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-1831. --- runtime.taskmanager.RegistrationTests fails spuriously -- Key: FLINK-1831 URL: https://issues.apache.org/jira/browse/FLINK-1831 Project: Flink Issue Type: Bug Components: Distributed Runtime Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Stephan Ewen Fix For: 0.9 Failing in a decent chunk of travis builds, I could also reproduce it locally: akka.actor.InvalidActorNameException: actor name [FAKE_JOB_MANAGER] is not unique! at akka.actor.dungeon.ChildrenContainer$TerminatingChildrenContainer.reserve(ChildrenContainer.scala:192) at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) at akka.actor.ActorCell.reserveChild(ActorCell.scala:369) at akka.actor.dungeon.Children$class.makeChild(Children.scala:202) at akka.actor.dungeon.Children$class.attachChild(Children.scala:42) at akka.actor.ActorCell.attachChild(ActorCell.scala:369) at akka.actor.ActorSystemImpl.actorOf(ActorSystem.scala:553) at org.apache.flink.runtime.taskmanager.RegistrationTest$5.init(RegistrationTest.java:308) at org.apache.flink.runtime.taskmanager.RegistrationTest.testTaskManagerResumesConnectAfterJobManagerFailure(RegistrationTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) 09:46:39.486 [flink-akka.actor.default-dispatcher-2] [flink-akka.actor.default-dispatcher-3 - akka://flink/user/FAKE_JOB_MANAGER] ERROR akka.actor.OneForOneStrategy - Kill akka.actor.ActorKilledException: Kill java.lang.AssertionError: actor name [FAKE_JOB_MANAGER] is not unique! at org.junit.Assert.fail(Assert.java:88) at org.apache.flink.runtime.taskmanager.RegistrationTest$5.init(RegistrationTest.java:328) at org.apache.flink.runtime.taskmanager.RegistrationTest.testTaskManagerResumesConnectAfterJobManagerFailure(RegistrationTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at
[jira] [Resolved] (FLINK-1831) runtime.taskmanager.RegistrationTests fails spuriously
[ https://issues.apache.org/jira/browse/FLINK-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1831. - Resolution: Fixed Fix Version/s: 0.9 Assignee: Stephan Ewen Fixed via 5cd9e9d94bb3af4a5a868789f663af262672cac8 runtime.taskmanager.RegistrationTests fails spuriously -- Key: FLINK-1831 URL: https://issues.apache.org/jira/browse/FLINK-1831 Project: Flink Issue Type: Bug Components: Distributed Runtime Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Stephan Ewen Fix For: 0.9 Failing in a decent chunk of travis builds, I could also reproduce it locally: akka.actor.InvalidActorNameException: actor name [FAKE_JOB_MANAGER] is not unique! at akka.actor.dungeon.ChildrenContainer$TerminatingChildrenContainer.reserve(ChildrenContainer.scala:192) at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) at akka.actor.ActorCell.reserveChild(ActorCell.scala:369) at akka.actor.dungeon.Children$class.makeChild(Children.scala:202) at akka.actor.dungeon.Children$class.attachChild(Children.scala:42) at akka.actor.ActorCell.attachChild(ActorCell.scala:369) at akka.actor.ActorSystemImpl.actorOf(ActorSystem.scala:553) at org.apache.flink.runtime.taskmanager.RegistrationTest$5.init(RegistrationTest.java:308) at org.apache.flink.runtime.taskmanager.RegistrationTest.testTaskManagerResumesConnectAfterJobManagerFailure(RegistrationTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) 09:46:39.486 [flink-akka.actor.default-dispatcher-2] [flink-akka.actor.default-dispatcher-3 - akka://flink/user/FAKE_JOB_MANAGER] ERROR akka.actor.OneForOneStrategy - Kill akka.actor.ActorKilledException: Kill java.lang.AssertionError: actor name [FAKE_JOB_MANAGER] is not unique! at org.junit.Assert.fail(Assert.java:88) at org.apache.flink.runtime.taskmanager.RegistrationTest$5.init(RegistrationTest.java:328) at org.apache.flink.runtime.taskmanager.RegistrationTest.testTaskManagerResumesConnectAfterJobManagerFailure(RegistrationTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at
[jira] [Created] (FLINK-1832) start-local.bat/start-local.sh does not work if there is a white space in the file path (windows)
Nikolaas Steenbergen created FLINK-1832: --- Summary: start-local.bat/start-local.sh does not work if there is a white space in the file path (windows) Key: FLINK-1832 URL: https://issues.apache.org/jira/browse/FLINK-1832 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.8.1 Environment: Windows 8.1 Reporter: Nikolaas Steenbergen Priority: Minor start-local.bat can be run, but exits immediately. start-local.sh with cygwin exits with error: owner_@owner /cygdrive/e/Program Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin $ ./start-local.sh cygpath: error converting Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin/.. - No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory cygpath: error converting /cygdrive/e/Program:Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin/../lib/*.jar:. - Unknown error -1 ./start-local.sh: line 27: E:\Program/bin/jobmanager.sh: No such file or directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1577) Misleading error messages when cancelling tasks
[ https://issues.apache.org/jira/browse/FLINK-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi resolved FLINK-1577. Resolution: Fixed Misleading error messages when cancelling tasks --- Key: FLINK-1577 URL: https://issues.apache.org/jira/browse/FLINK-1577 Project: Flink Issue Type: Improvement Components: Distributed Runtime Affects Versions: master Reporter: Ufuk Celebi Assignee: Ufuk Celebi A user running a Flink version before bec9c4d ran into a job manager failure (fixed in bec9c4d), which lead to restarting the JM and cancelling/clearing all tasks on the TMs. The logs of the TMs were inconclusive. I think part of that has been fixed by now, e.g. there is a log message when cancelAndClearEverything is called, but the task thread (RuntimeEnvironment) always logs an error when interrupted during the run method -- even if the task gets cancelled. I think these error messages are misleading and only the root cause is important (i.e. non-failed tasks should be silently cancelled). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1469) Initialize network environment at task manager startup
[ https://issues.apache.org/jira/browse/FLINK-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi resolved FLINK-1469. Resolution: Fixed Fix Version/s: 0.9 Fixed in e74521c1fe5f5842185be5c454d27adc3bd254d5. Initialize network environment at task manager startup -- Key: FLINK-1469 URL: https://issues.apache.org/jira/browse/FLINK-1469 Project: Flink Issue Type: Improvement Components: Distributed Runtime Affects Versions: master Reporter: Ufuk Celebi Priority: Minor Fix For: 0.9 Currently, the network environment is initialized only after the registration of a task manager with the job manager finishes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-1834) Is mapred.output.dir conf parameter really required?
Flavio Pompermaier created FLINK-1834: - Summary: Is mapred.output.dir conf parameter really required? Key: FLINK-1834 URL: https://issues.apache.org/jira/browse/FLINK-1834 Project: Flink Issue Type: Improvement Components: Hadoop Compatibility Affects Versions: 0.9 Reporter: Flavio Pompermaier Priority: Minor At the moment if you don't set mapred.output.dir parameter when output to an hadoop sink Flink throws an error. Probably this parameter could be defaulted to some tmp dir + jobId (for each output sink). For example, when output to an HBase sink, the only things you'll find in the mapred.output.dir will be the SUCCESS files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1832) start-local.bat/start-local.sh does not work if there is a white space in the file path (windows)
[ https://issues.apache.org/jira/browse/FLINK-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482901#comment-14482901 ] Maximilian Michels commented on FLINK-1832: --- Thanks for reporting this. Looks like an easy fix. start-local.bat/start-local.sh does not work if there is a white space in the file path (windows) - Key: FLINK-1832 URL: https://issues.apache.org/jira/browse/FLINK-1832 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.8.1 Environment: Windows 8.1 Reporter: Nikolaas Steenbergen Priority: Minor start-local.bat can be run, but exits immediately. start-local.sh with cygwin exits with error: owner_@owner /cygdrive/e/Program Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin $ ./start-local.sh cygpath: error converting Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin/.. - No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory cygpath: error converting /cygdrive/e/Program:Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin/../lib/*.jar:. - Unknown error -1 ./start-local.sh: line 27: E:\Program/bin/jobmanager.sh: No such file or directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1828) Impossible to output data to an HBase table
[ https://issues.apache.org/jira/browse/FLINK-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482898#comment-14482898 ] ASF GitHub Bot commented on FLINK-1828: --- Github user fpompermaier commented on the pull request: https://github.com/apache/flink/pull/571#issuecomment-90481118 Ok, I created this issue (https://issues.apache.org/jira/browse/FLINK-1834) about the mapred.output.dir Impossible to output data to an HBase table --- Key: FLINK-1828 URL: https://issues.apache.org/jira/browse/FLINK-1828 Project: Flink Issue Type: Bug Components: Hadoop Compatibility Affects Versions: 0.9 Reporter: Flavio Pompermaier Labels: hadoop, hbase Fix For: 0.9 Right now it is not possible to use HBase TableOutputFormat as output format because Configurable.setConf is not called in the configure() method of the HadoopOutputFormatBase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fixed Configurable HadoopOutputFormat (FLINK-1...
Github user fpompermaier commented on the pull request: https://github.com/apache/flink/pull/571#issuecomment-90481118 Ok, I created this issue (https://issues.apache.org/jira/browse/FLINK-1834) about the mapred.output.dir --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1822) Update Community page on website with a Getting Help section
[ https://issues.apache.org/jira/browse/FLINK-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482818#comment-14482818 ] Maximilian Michels commented on FLINK-1822: --- +1 very nice idea. The path of the linked how_to_contribute page is wrong. Update Community page on website with a Getting Help section Key: FLINK-1822 URL: https://issues.apache.org/jira/browse/FLINK-1822 Project: Flink Issue Type: Task Components: Project Website Reporter: Robert Metzger Assignee: Robert Metzger Attachments: update_community.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fixed Configurable HadoopOutputFormat (FLINK-1...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/571#issuecomment-90475934 Is the Hadoop configuration specified in the flink-conf.yaml loaded? If we set `mapred.output.dir` then we should check for an existing config entry beforehand. Otherwise, we overwrite Hadoop configuration values. Like @fhueske suggested, please open a JIRA for investigation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1828) Impossible to output data to an HBase table
[ https://issues.apache.org/jira/browse/FLINK-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482834#comment-14482834 ] ASF GitHub Bot commented on FLINK-1828: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/571#issuecomment-90475934 Is the Hadoop configuration specified in the flink-conf.yaml loaded? If we set `mapred.output.dir` then we should check for an existing config entry beforehand. Otherwise, we overwrite Hadoop configuration values. Like @fhueske suggested, please open a JIRA for investigation. Impossible to output data to an HBase table --- Key: FLINK-1828 URL: https://issues.apache.org/jira/browse/FLINK-1828 Project: Flink Issue Type: Bug Components: Hadoop Compatibility Affects Versions: 0.9 Reporter: Flavio Pompermaier Labels: hadoop, hbase Fix For: 0.9 Right now it is not possible to use HBase TableOutputFormat as output format because Configurable.setConf is not called in the configure() method of the HadoopOutputFormatBase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-1833) Refactor partition availability notification in ExecutionGraph
Ufuk Celebi created FLINK-1833: -- Summary: Refactor partition availability notification in ExecutionGraph Key: FLINK-1833 URL: https://issues.apache.org/jira/browse/FLINK-1833 Project: Flink Issue Type: Improvement Components: Distributed Runtime, JobManager Affects Versions: master Reporter: Ufuk Celebi The mechanism to notify the JobManager about available result partitions is hard to understand. The are two parts to this: 1) JobManager - The deployment of receivers happens in the Execution class although it is by now totally unrelated to the state of a specific execution. I propose to move this to the respective IntermediateResultPartition. - The deployment information for a receiver is spread across different components: when creating the TaskDeploymentDescriptor and the caching of partition infos at the consuming vertex. This is very hard to follow and results in unnecessary messages being sent (which are discarded at the TM). 2) TaskManager - Pipelined results notify where you would expect it in the ResultPartition, but blocking results don't have an extra message and are implicitly piggy-backed to the final state transition, after which the job manager deploys receivers if all blocking partitions of a result have been produced. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-1835) Spurious failure of YARN tests
Stephan Ewen created FLINK-1835:
---
Summary: Spurious failure of YARN tests
Key: FLINK-1835
URL: https://issues.apache.org/jira/browse/FLINK-1835
Project: Flink
Issue Type: Bug
Components: YARN Client
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Robert Metzger
Fix For: 0.9
THe failure was caused by detecting an exception in the log.
Stack trace of the exception (extracted from the log) below
{code}
21:18:29,555 WARN org.apache.hadoop.util.NativeCodeLoader
- Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
21:18:29,806 INFO org.apache.flink.yarn.ApplicationMaster$
- YARN daemon runs as travis setting user to execute Flink
ApplicationMaster/JobManager to travis
21:18:29,808 INFO org.apache.flink.yarn.ApplicationMaster$
-
21:18:29,809 INFO org.apache.flink.yarn.ApplicationMaster$
- Starting YARN ApplicationMaster/JobManager (Version: 0.9-SNAPSHOT,
Rev:d2020b5, Date:06.04.2015 @ 18:00:21 UTC)
21:18:29,809 INFO org.apache.flink.yarn.ApplicationMaster$
- Current user: travis
21:18:29,809 INFO org.apache.flink.yarn.ApplicationMaster$
- JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.31-b07
21:18:29,809 INFO org.apache.flink.yarn.ApplicationMaster$
- Maximum heap size: 393 MiBytes
21:18:29,826 INFO org.apache.flink.yarn.ApplicationMaster$
- JAVA_HOME: /usr/lib/jvm/java-8-oracle
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
- JVM Options:
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
- -Xmx409M
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
-
-Dlog.file=/home/travis/build/StephanEwen/incubator-flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-1_0/application_1428355034517_0004/container_1428355034517_0004_01_01/jobmanager-main.log
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
- -Dlogback.configurationFile=file:logback.xml
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
- -Dlog4j.configuration=file:log4j.properties
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
- Program Arguments: (none)
21:18:29,827 INFO org.apache.flink.yarn.ApplicationMaster$
-
21:18:29,828 INFO org.apache.flink.yarn.ApplicationMaster$
- registered UNIX signal handlers for [TERM, HUP, INT]
21:18:29,843 INFO org.apache.flink.yarn.ApplicationMaster$
- Starting JobManager for YARN
21:18:29,845 INFO org.apache.flink.yarn.ApplicationMaster$
- Loading config from:
/home/travis/build/StephanEwen/incubator-flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-localDir-nm-1_0/usercache/travis/appcache/application_1428355034517_0004/container_1428355034517_0004_01_01
21:18:30,388 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
21:18:30,450 INFO Remoting
- Starting remoting
21:18:30,637 INFO Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@172.17.0.176:34023]
21:18:30,651 INFO org.apache.flink.runtime.blob.BlobServer
- Created BLOB server storage directory
/tmp/blobStore-e34b86da-094c-4a4e-aa02-7b0556e8af93
21:18:30,655 INFO org.apache.flink.runtime.blob.BlobServer
- Started BLOB server at 0.0.0.0:33717 - max concurrent requests: 50 - max
backlog: 1000
21:18:30,670 INFO org.apache.flink.yarn.ApplicationMaster$
- Starting Job Manger web frontend.
21:18:30,673 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer
- Setting up web info server, using web-root directory
jar:file:/home/travis/build/StephanEwen/incubator-flink/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-localDir-nm-1_0/usercache/travis/appcache/application_1428355034517_0004/filecache/12/flink-dist-0.9-SNAPSHOT.jar!/web-docs-infoserver.
21:18:30,705 INFO org.apache.flink.runtime.jobmanager.JobManager
- Starting JobManager at akka://flink/user/jobmanager#395299512.
21:18:31,184 INFO org.eclipse.jetty.util.log
- jetty-0.9-SNAPSHOT
21:18:31,269 INFO org.eclipse.jetty.util.log
[GitHub] flink pull request: Fix issue where Windows paths were not recogni...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/491#issuecomment-90507970 Btw, the path `file:/c:` that motivated this PR is not a valid absolute path according to Microsoft's documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1836) just test
zhu created FLINK-1836: -- Summary: just test Key: FLINK-1836 URL: https://issues.apache.org/jira/browse/FLINK-1836 Project: Flink Issue Type: Bug Reporter: zhu just test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1817) ClassLoaderObjectInputStream fails with ClassNotFoundException for primitive classes
[
https://issues.apache.org/jira/browse/FLINK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483114#comment-14483114
]
ASF GitHub Bot commented on FLINK-1817:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/565
ClassLoaderObjectInputStream fails with ClassNotFoundException for primitive
classes
Key: FLINK-1817
URL: https://issues.apache.org/jira/browse/FLINK-1817
Project: Flink
Issue Type: Bug
Components: core
Affects Versions: 0.9, 0.8.1
Reporter: Fabian Hueske
Assignee: Fabian Hueske
Fix For: 0.9, 0.8.2
User function objects that contain primitive type classes (such as
{{int.class}} or {{long.class}}) are not supported because the
{{ClassLoaderObjectInputStream}} fails with a {{ClassNotFoundException}} when
reading the object from the {{TaskConfig}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-1838) Update streaming programming guide
Gyula Fora created FLINK-1838: - Summary: Update streaming programming guide Key: FLINK-1838 URL: https://issues.apache.org/jira/browse/FLINK-1838 Project: Flink Issue Type: Task Components: Documentation, Streaming Reporter: Gyula Fora Assignee: Gyula Fora The current streaming programming guide has gotten slightly outdated since the last release. We need to update it to reflect the changes in the API and runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1834) Is mapred.output.dir conf parameter really required?
[ https://issues.apache.org/jira/browse/FLINK-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483249#comment-14483249 ] Fabian Hueske commented on FLINK-1834: -- I totally agree. We should get rid of this mandatory parameter. We try to only set it if it is set by the user and let the HadoopFormat figure out whether it needs it or not. If it does and it is not set, the job should fail. Is mapred.output.dir conf parameter really required? Key: FLINK-1834 URL: https://issues.apache.org/jira/browse/FLINK-1834 Project: Flink Issue Type: Improvement Components: Hadoop Compatibility Affects Versions: 0.9 Reporter: Flavio Pompermaier Priority: Minor Labels: hadoop At the moment if you don't set mapred.output.dir parameter when output to an hadoop sink Flink throws an error. Probably this parameter could be defaulted to some tmp dir + jobId (for each output sink). For example, when output to an HBase sink, the only things you'll find in the mapred.output.dir will be the SUCCESS files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1560) Add ITCases for streaming examples
[ https://issues.apache.org/jira/browse/FLINK-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi closed FLINK-1560. - Resolution: Implemented Fix Version/s: 0.9 Implemented via ed7d165, 464e782 and 954beca. Add ITCases for streaming examples -- Key: FLINK-1560 URL: https://issues.apache.org/jira/browse/FLINK-1560 Project: Flink Issue Type: Test Components: Streaming Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Péter Szabó Fix For: 0.9 Currently there are no tests for consistency of the streaming example programs. This might be a real show stopper for users who encounter an issue there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1560) Add ITCases for streaming examples
[ https://issues.apache.org/jira/browse/FLINK-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483216#comment-14483216 ] ASF GitHub Bot commented on FLINK-1560: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/519 Add ITCases for streaming examples -- Key: FLINK-1560 URL: https://issues.apache.org/jira/browse/FLINK-1560 Project: Flink Issue Type: Test Components: Streaming Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Péter Szabó Currently there are no tests for consistency of the streaming example programs. This might be a real show stopper for users who encounter an issue there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1834) Is mapred.output.dir conf parameter really required?
[ https://issues.apache.org/jira/browse/FLINK-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483258#comment-14483258 ] Maximilian Michels commented on FLINK-1834: --- +1 Is mapred.output.dir conf parameter really required? Key: FLINK-1834 URL: https://issues.apache.org/jira/browse/FLINK-1834 Project: Flink Issue Type: Improvement Components: Hadoop Compatibility Affects Versions: 0.9 Reporter: Flavio Pompermaier Priority: Minor Labels: hadoop At the moment if you don't set mapred.output.dir parameter when output to an hadoop sink Flink throws an error. Probably this parameter could be defaulted to some tmp dir + jobId (for each output sink). For example, when output to an HBase sink, the only things you'll find in the mapred.output.dir will be the SUCCESS files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-1837) Throw an exceptions for iterative streaming programs with checkpointing enabled
Márton Balassi created FLINK-1837: - Summary: Throw an exceptions for iterative streaming programs with checkpointing enabled Key: FLINK-1837 URL: https://issues.apache.org/jira/browse/FLINK-1837 Project: Flink Issue Type: Improvement Components: Streaming Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Márton Balassi Currently streaming state checkpointing does not properly support iterative programs. The user should be clearly notified of this behaviour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1834) Is mapred.output.dir conf parameter really required?
[ https://issues.apache.org/jira/browse/FLINK-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483142#comment-14483142 ] Fabian Hueske commented on FLINK-1834: -- I would be careful with this. Do we know where we should default to? If an OutputFormat does not require an output path we do not need to set it at all and if it does, it should be set by the user, IMO. Is mapred.output.dir conf parameter really required? Key: FLINK-1834 URL: https://issues.apache.org/jira/browse/FLINK-1834 Project: Flink Issue Type: Improvement Components: Hadoop Compatibility Affects Versions: 0.9 Reporter: Flavio Pompermaier Priority: Minor Labels: hadoop At the moment if you don't set mapred.output.dir parameter when output to an hadoop sink Flink throws an error. Probably this parameter could be defaulted to some tmp dir + jobId (for each output sink). For example, when output to an HBase sink, the only things you'll find in the mapred.output.dir will be the SUCCESS files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: FLINK-1560 - Add ITCases for streaming example...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/519 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1820) Bug in DoubleParser and FloatParser - empty String is not casted to 0
[ https://issues.apache.org/jira/browse/FLINK-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483215#comment-14483215 ] Stephan Ewen commented on FLINK-1820: - Given the discussion here, I agree with Fabian. An exception is easier to understand than strange results because of a sneaky zero inserted for empty fields... Bug in DoubleParser and FloatParser - empty String is not casted to 0 - Key: FLINK-1820 URL: https://issues.apache.org/jira/browse/FLINK-1820 Project: Flink Issue Type: Bug Components: core Affects Versions: 0.8.0, 0.9, 0.8.1 Reporter: Felix Neutatz Assignee: Felix Neutatz Priority: Critical Fix For: 0.9 Hi, I found the bug, when I wanted to read a csv file, which had a line like: ||\n If I treat it as a Tuple2Long,Long, I get as expected a tuple (0L,0L). But if I want to read it into a Double-Tuple or a Float-Tuple, I get the following error: java.lang.AssertionError: Test failed due to a org.apache.flink.api.common.io.ParseException: Line could not be parsed: '||' ParserError NUMERIC_VALUE_FORMAT_ERROR This error can be solved by adding an additional condition for empty strings in the FloatParser / DoubleParser. We definitely need the CSVReader to be able to read empty values. I can fix it like described if there are no better ideas :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1834) Is mapred.output.dir conf parameter really required?
[
https://issues.apache.org/jira/browse/FLINK-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483125#comment-14483125
]
Maximilian Michels commented on FLINK-1834:
---
Let's check if the user's Hadoop configuration contains {{mapred.output.dir}}
via {{HadoopUtils.getHadoopConfiguration()}}. Otherwise, set a default with a
warning which also contains the output path to which we default.
Do you want to fix that?
Is mapred.output.dir conf parameter really required?
Key: FLINK-1834
URL: https://issues.apache.org/jira/browse/FLINK-1834
Project: Flink
Issue Type: Improvement
Components: Hadoop Compatibility
Affects Versions: 0.9
Reporter: Flavio Pompermaier
Priority: Minor
Labels: hadoop
At the moment if you don't set mapred.output.dir parameter when output to an
hadoop sink Flink throws an error. Probably this parameter could be defaulted
to some tmp dir + jobId (for each output sink).
For example, when output to an HBase sink, the only things you'll find in the
mapred.output.dir will be the SUCCESS files
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-1817) ClassLoaderObjectInputStream fails with ClassNotFoundException for primitive classes
[
https://issues.apache.org/jira/browse/FLINK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske resolved FLINK-1817.
--
Resolution: Fixed
Fixed with
- 0.9: d33b44549d416c30f01ead9aacf5cf7ed30674ca
- 0.8: a8dee2f0facae5fac8fe1ddf320ff7d377021313
ClassLoaderObjectInputStream fails with ClassNotFoundException for primitive
classes
Key: FLINK-1817
URL: https://issues.apache.org/jira/browse/FLINK-1817
Project: Flink
Issue Type: Bug
Components: core
Affects Versions: 0.9, 0.8.1
Reporter: Fabian Hueske
Assignee: Fabian Hueske
Fix For: 0.9, 0.8.2
User function objects that contain primitive type classes (such as
{{int.class}} or {{long.class}}) are not supported because the
{{ClassLoaderObjectInputStream}} fails with a {{ClassNotFoundException}} when
reading the object from the {{TaskConfig}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1820) Bug in DoubleParser and FloatParser - empty String is not casted to 0
[ https://issues.apache.org/jira/browse/FLINK-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483253#comment-14483253 ] Maximilian Michels commented on FLINK-1820: --- +1 for a exception message for an empty String Bug in DoubleParser and FloatParser - empty String is not casted to 0 - Key: FLINK-1820 URL: https://issues.apache.org/jira/browse/FLINK-1820 Project: Flink Issue Type: Bug Components: core Affects Versions: 0.8.0, 0.9, 0.8.1 Reporter: Felix Neutatz Assignee: Felix Neutatz Priority: Critical Fix For: 0.9 Hi, I found the bug, when I wanted to read a csv file, which had a line like: ||\n If I treat it as a Tuple2Long,Long, I get as expected a tuple (0L,0L). But if I want to read it into a Double-Tuple or a Float-Tuple, I get the following error: java.lang.AssertionError: Test failed due to a org.apache.flink.api.common.io.ParseException: Line could not be parsed: '||' ParserError NUMERIC_VALUE_FORMAT_ERROR This error can be solved by adding an additional condition for empty strings in the FloatParser / DoubleParser. We definitely need the CSVReader to be able to read empty values. I can fix it like described if there are no better ideas :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1839) Failures in TwitterStreamITCase
[
https://issues.apache.org/jira/browse/FLINK-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483441#comment-14483441
]
Stephan Ewen commented on FLINK-1839:
-
Okay, from my latest Travis logs, this seems to happen in one out of three runs
on average...
Failures in TwitterStreamITCase
---
Key: FLINK-1839
URL: https://issues.apache.org/jira/browse/FLINK-1839
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Stephan Ewen
Priority: Critical
The test seems unstable and fails occasionally with the error below
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.508 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 3.739 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:144
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-1839) Failures in TwitterStreamITCase
Stephan Ewen created FLINK-1839:
---
Summary: Failures in TwitterStreamITCase
Key: FLINK-1839
URL: https://issues.apache.org/jira/browse/FLINK-1839
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Stephan Ewen
The test seems unstable and fails occasionally with the error below
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.508 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 3.739 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:144
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1839) Failures in TwitterStreamITCase
[
https://issues.apache.org/jira/browse/FLINK-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483440#comment-14483440
]
Stephan Ewen commented on FLINK-1839:
-
Another occurance
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.448 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 4.745 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:160
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
Failures in TwitterStreamITCase
---
Key: FLINK-1839
URL: https://issues.apache.org/jira/browse/FLINK-1839
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Stephan Ewen
The test seems unstable and fails occasionally with the error below
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.508 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 3.739 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:144
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-1839) Failures in TwitterStreamITCase
[
https://issues.apache.org/jira/browse/FLINK-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-1839:
Priority: Critical (was: Major)
Failures in TwitterStreamITCase
---
Key: FLINK-1839
URL: https://issues.apache.org/jira/browse/FLINK-1839
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Stephan Ewen
Priority: Critical
The test seems unstable and fails occasionally with the error below
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.508 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 3.739 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:144
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1319) Add static code analysis for UDFs
[
https://issues.apache.org/jira/browse/FLINK-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483480#comment-14483480
]
Timo Walther commented on FLINK-1319:
-
I have integrated the analyzer in the FlatMapOperator overriding the
getSemanticProperties() method:
https://github.com/twalthr/flink/blob/a01b8334795d530b28d25f6bb20b09fca5c3cf27/flink-java/src/main/java/org/apache/flink/api/java/operators/FlatMapOperator.java
Do you think that's the right way to go?
(before I implement that in all operators ;) )
Add static code analysis for UDFs
-
Key: FLINK-1319
URL: https://issues.apache.org/jira/browse/FLINK-1319
Project: Flink
Issue Type: New Feature
Components: Java API, Scala API
Reporter: Stephan Ewen
Assignee: Timo Walther
Priority: Minor
Flink's Optimizer takes information that tells it for UDFs which fields of
the input elements are accessed, modified, or frwarded/copied. This
information frequently helps to reuse partitionings, sorts, etc. It may speed
up programs significantly, as it can frequently eliminate sorts and shuffles,
which are costly.
Right now, users can add lightweight annotations to UDFs to provide this
information (such as adding {{@ConstandFields(0-3, 1, 2-1)}}.
We worked with static code analysis of UDFs before, to determine this
information automatically. This is an incredible feature, as it magically
makes programs faster.
For record-at-a-time operations (Map, Reduce, FlatMap, Join, Cross), this
works surprisingly well in many cases. We used the Soot toolkit for the
static code analysis. Unfortunately, Soot is LGPL licensed and thus we did
not include any of the code so far.
I propose to add this functionality to Flink, in the form of a drop-in
addition, to work around the LGPL incompatibility with ALS 2.0. Users could
simply download a special flink-code-analysis.jar and drop it into the
lib folder to enable this functionality. We may even add a script to
tools that downloads that library automatically into the lib folder. This
should be legally fine, since we do not redistribute LGPL code and only
dynamically link it (the incompatibility with ASL 2.0 is mainly in the
patentability, if I remember correctly).
Prior work on this has been done by [~aljoscha] and [~skunert], which could
provide a code base to start with.
*Appendix*
Hompage to Soot static analysis toolkit: http://www.sable.mcgill.ca/soot/
Papers on static analysis and for optimization:
http://stratosphere.eu/assets/papers/EnablingOperatorReorderingSCA_12.pdf and
http://stratosphere.eu/assets/papers/openingTheBlackBoxes_12.pdf
Quick introduction to the Optimizer:
http://stratosphere.eu/assets/papers/2014-VLDBJ_Stratosphere_Overview.pdf
(Section 6)
Optimizer for Iterations:
http://stratosphere.eu/assets/papers/spinningFastIterativeDataFlows_12.pdf
(Sections 4.3 and 5.3)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-1837) Throw an exceptions for iterative streaming programs with checkpointing enabled
[ https://issues.apache.org/jira/browse/FLINK-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi closed FLINK-1837. - Resolution: Implemented Fix Version/s: 0.9 Implemented via 52ebb29 Throw an exceptions for iterative streaming programs with checkpointing enabled --- Key: FLINK-1837 URL: https://issues.apache.org/jira/browse/FLINK-1837 Project: Flink Issue Type: Improvement Components: Streaming Affects Versions: 0.9 Reporter: Márton Balassi Assignee: Márton Balassi Fix For: 0.9 Currently streaming state checkpointing does not properly support iterative programs. The user should be clearly notified of this behaviour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fix permissions on various non-executable sour...
Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/574#issuecomment-90660629 Ship it :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user markus-h commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-90694363 There seems to be a race condition somewhere in my code but I have trouble finding it since I can not reproduce it locally. I thought my last change would fix it but it didn't. So if somebody has some free time and knows a bit about race conditions feel free to help me :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-1839) Failures in TwitterStreamITCase
[
https://issues.apache.org/jira/browse/FLINK-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Márton Balassi reassigned FLINK-1839:
-
Assignee: Márton Balassi
Failures in TwitterStreamITCase
---
Key: FLINK-1839
URL: https://issues.apache.org/jira/browse/FLINK-1839
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Márton Balassi
Priority: Critical
The test seems unstable and fails occasionally with the error below
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.508 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 3.739 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:144
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1839) Failures in TwitterStreamITCase
[
https://issues.apache.org/jira/browse/FLINK-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483537#comment-14483537
]
Márton Balassi commented on FLINK-1839:
---
My bad, I thought I have fixed it already. Thanks for the report.
Failures in TwitterStreamITCase
---
Key: FLINK-1839
URL: https://issues.apache.org/jira/browse/FLINK-1839
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Stephan Ewen
Priority: Critical
The test seems unstable and fails occasionally with the error below
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.508 sec
FAILURE! - in
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase)
Time elapsed: 3.739 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:146 but was:144
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.twitter.TwitterStreamITCase.postSubmit(TwitterStreamITCase.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-951) Reworking of Iteration Synchronization, Accumulators and Aggregators
[ https://issues.apache.org/jira/browse/FLINK-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483752#comment-14483752 ] ASF GitHub Bot commented on FLINK-951: -- Github user markus-h commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-90694363 There seems to be a race condition somewhere in my code but I have trouble finding it since I can not reproduce it locally. I thought my last change would fix it but it didn't. So if somebody has some free time and knows a bit about race conditions feel free to help me :-) Reworking of Iteration Synchronization, Accumulators and Aggregators Key: FLINK-951 URL: https://issues.apache.org/jira/browse/FLINK-951 Project: Flink Issue Type: Improvement Components: Iterations, Optimizer Affects Versions: 0.9 Reporter: Markus Holzemer Assignee: Markus Holzemer Labels: refactoring Original Estimate: 168h Remaining Estimate: 168h I just realized that there is no real Jira issue for the task I am currently working on. I am currently reworking a few things regarding Iteration Synchronization, Accumulators and Aggregators. Currently the synchronization at the end of one superstep is done through channel events. That makes it hard to track the current status of iterations. That is why I am changing this synchronization to use RPC calls with the JobManager, so that the JobManager manages the current status of all iterations. Currently we use Accumulators outside of iterations and Aggregators inside of iterations. Both have a similiar function, but a bit different interfaces and handling. I want to unify these two concepts. I propose that we stick in the future to Accumulators only. Aggregators therefore are removed and Accumulators are extended to cover the usecases Aggregators were used fore before. The switch to RPC for iterations makes it possible to also send the current Accumulator values at the end of each superstep, so that the JobManager (and thereby the webinterface) will be able to print intermediate accumulation results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Fix permissions on various non-executable sour...
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/574 Fix permissions on various non-executable source files Not a critical fix, but threw off the search for 3rd party executables as part of my pre-release code checking... You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink permissions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/574.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #574 commit 469256458c51ccf9035c4d0b6083c072d9858452 Author: Stephan Ewen se...@apache.org Date: 2015-04-07T17:09:02Z Remove 'executable' permission from various non-executable source files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1840) Job execution fails on Windows (native and Cygwin)
Fabian Hueske created FLINK-1840:
Summary: Job execution fails on Windows (native and Cygwin)
Key: FLINK-1840
URL: https://issues.apache.org/jira/browse/FLINK-1840
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 0.9
Environment: Windows 8.1 (Cygwin and native)
Reporter: Fabian Hueske
Priority: Critical
Fix For: 0.9
Job execution fails on Windows using
- {{./bin/flink.bat}} (native)
- {{./bin/flink.sh}} (Cygwin)
- Web submission client
The following exception is thrown:
{code}
org.apache.flink.client.program.ProgramInvocationException: Could not upload
the program's JAR files tothe JobManager.
at org.apache.flink.client.program.Client.run(Client.java:341)
at org.apache.flink.client.program.Client.run(Client.java:313)
at org.apache.flink.client.program.Client.run(Client.java:306)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:63)
at
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedPro
gram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:273)
at
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:886)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:928)
Caused by: java.io.IOException: PUT operation failed: Server side error: Cannot
move staging file C:\cyg
win64\tmp\blobStore-17f49f61-bcb7-45ba-909e-56beca159fe4\incoming\temp-
to BLOB file C:\cygwin64
\tmp\blobStore-17f49f61-bcb7-45ba-909e-56beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:465)
at org.apache.flink.runtime.blob.BlobClient.put(BlobClient.java:327)
at
org.apache.flink.runtime.jobgraph.JobGraph.uploadRequiredJarFiles(JobGraph.java:457)
at
org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:288)
at
org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
at org.apache.flink.client.program.Client.run(Client.java:338)
... 15 more
Caused by: java.io.IOException: Server side error: Cannot move staging file
C:\cygwin64\tmp\blobStore-17
f49f61-bcb7-45ba-909e-56beca159fe4\incoming\temp- to BLOB file
C:\cygwin64\tmp\blobStore-17f49f6
1-bcb7-45ba-909e-56beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobClient.receivePutResponseAndCompare(BlobClient.java:491)
at
org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:461)
... 20 more
Caused by: java.io.IOException: Cannot move staging file
C:\cygwin64\tmp\blobStore-17f49f61-bcb7-45ba-90
9e-56beca159fe4\incoming\temp- to BLOB file
C:\cygwin64\tmp\blobStore-17f49f61-bcb7-45ba-909e-56
beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobServerConnection.put(BlobServerConnection.java:325)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:109)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1758) Extend Gelly's neighborhood methods
[ https://issues.apache.org/jira/browse/FLINK-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484000#comment-14484000 ] ASF GitHub Bot commented on FLINK-1758: --- GitHub user andralungu opened a pull request: https://github.com/apache/flink/pull/576 [FLINK-1758][gelly] Neighborhood Methods Extensions You can merge this pull request into a Git repository by running: $ git pull https://github.com/andralungu/flink extendNeighborhood Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #576 commit e5f6571c7cb05d96aac84530beb52cef968243d7 Author: andralungu lungu.an...@gmail.com Date: 2015-04-07T20:29:18Z [FLINK-1758][gelly] Neighborhood Methods Extensions Extend Gelly's neighborhood methods --- Key: FLINK-1758 URL: https://issues.apache.org/jira/browse/FLINK-1758 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Andra Lungu Currently, the neighborhood methods only allow returning a single value per vertex. In many cases, it is desirable to return several or no value per vertex. This is the case in clustering coefficient computation, vertex-centric jaccard, algorithms where a vertex computes a value per edge or when a vertex computes a value only for some of its neighbors. This issue proposes to - change the current reduceOnEdges/reduceOnNeighbors methods to use combinable reduce operations where possible - provide groupReduce-versions, which will use a Collector and allow returning none or more values per vertex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1758][gelly] Neighborhood Methods Exten...
GitHub user andralungu opened a pull request: https://github.com/apache/flink/pull/576 [FLINK-1758][gelly] Neighborhood Methods Extensions You can merge this pull request into a Git repository by running: $ git pull https://github.com/andralungu/flink extendNeighborhood Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #576 commit e5f6571c7cb05d96aac84530beb52cef968243d7 Author: andralungu lungu.an...@gmail.com Date: 2015-04-07T20:29:18Z [FLINK-1758][gelly] Neighborhood Methods Extensions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1832) start-local.bat/start-local.sh does not work if there is a white space in the file path (windows)
[ https://issues.apache.org/jira/browse/FLINK-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483878#comment-14483878 ] ASF GitHub Bot commented on FLINK-1832: --- GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/575 [FLINK-1832] Fix start scripts for Cygwin and paths with spaces A few fixes for the start and submission scripts. Right now they fail for: - all scripts fail on Cygwin - `./bin/flink` fails on Unix for Flink root paths that contain spaces Also, some duplicated code is removed. Fixes should be backported to 0.8 as well. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink cygwin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/575.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #575 commit dc9af9249c8fc1c8817ed73c90c0fa048130c7ba Author: Maximilian Michels m...@apache.org Date: 2015-04-07T10:56:12Z [FLINK-1832] [scripts] fix conversion to Windows path with spaces commit f20b4827c0d1fb49fd5c6773eeb05ac5540eab4b Author: Fabian Hueske fhue...@apache.org Date: 2015-04-07T18:59:44Z [FLINK-1832] [scripts] fix start scripts for Cygwin and paths with spaces. start-local.bat/start-local.sh does not work if there is a white space in the file path (windows) - Key: FLINK-1832 URL: https://issues.apache.org/jira/browse/FLINK-1832 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.8.1 Environment: Windows 8.1 Reporter: Nikolaas Steenbergen Priority: Minor start-local.bat can be run, but exits immediately. start-local.sh with cygwin exits with error: owner_@owner /cygdrive/e/Program Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin $ ./start-local.sh cygpath: error converting Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin/.. - No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory sed: can't read E:\Program/conf/flink-conf.yaml: No such file or directory cygpath: error converting /cygdrive/e/Program:Files/flink-0.8.1-bin-hadoop2/flink-0.8.1-bin-hadoop2/flink-0.8.1/bin/../lib/*.jar:. - Unknown error -1 ./start-local.sh: line 27: E:\Program/bin/jobmanager.sh: No such file or directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1741) Add Jaccard Similarity Metric Example
[ https://issues.apache.org/jira/browse/FLINK-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483936#comment-14483936 ] ASF GitHub Bot commented on FLINK-1741: --- Github user andralungu commented on the pull request: https://github.com/apache/flink/pull/544#issuecomment-90715751 @vasia, I updated the PR with your latest suggestions. Travis seems to fail because of flink-steaming :-s; my quick refactoring couldn't have triggered that :) Add Jaccard Similarity Metric Example - Key: FLINK-1741 URL: https://issues.apache.org/jira/browse/FLINK-1741 Project: Flink Issue Type: Task Components: Gelly Affects Versions: 0.9 Reporter: Andra Lungu Assignee: Andra Lungu http://www.inside-r.org/packages/cran/igraph/docs/similarity -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1741][gelly] Adds Jaccard Similarity Me...
Github user andralungu commented on the pull request: https://github.com/apache/flink/pull/544#issuecomment-90715751 @vasia, I updated the PR with your latest suggestions. Travis seems to fail because of flink-steaming :-s; my quick refactoring couldn't have triggered that :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1832] Fix start scripts for Cygwin and ...
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/575 [FLINK-1832] Fix start scripts for Cygwin and paths with spaces A few fixes for the start and submission scripts. Right now they fail for: - all scripts fail on Cygwin - `./bin/flink` fails on Unix for Flink root paths that contain spaces Also, some duplicated code is removed. Fixes should be backported to 0.8 as well. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink cygwin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/575.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #575 commit dc9af9249c8fc1c8817ed73c90c0fa048130c7ba Author: Maximilian Michels m...@apache.org Date: 2015-04-07T10:56:12Z [FLINK-1832] [scripts] fix conversion to Windows path with spaces commit f20b4827c0d1fb49fd5c6773eeb05ac5540eab4b Author: Fabian Hueske fhue...@apache.org Date: 2015-04-07T18:59:44Z [FLINK-1832] [scripts] fix start scripts for Cygwin and paths with spaces. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1523][gelly] Vertex centric iteration e...
Github user andralungu commented on the pull request: https://github.com/apache/flink/pull/537#issuecomment-90728415 Hi @vasia , I simply assumed that if the previous examples work, the iteration also works as before. Would you like a specific test suite for that? For the degree option not being set : the getInDegree(); getOutDegree() methods are located inside the Vertex class. Not sure that their availability can be changed by setting the option on the iteration. They are not computed when the degree option is not set, but you can still see the method. Do you have any idea how to make the degrees inaccessible? Once this becomes clear, I can add the tests :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1523) Vertex-centric iteration extensions
[
https://issues.apache.org/jira/browse/FLINK-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484019#comment-14484019
]
ASF GitHub Bot commented on FLINK-1523:
---
Github user andralungu commented on the pull request:
https://github.com/apache/flink/pull/537#issuecomment-90728415
Hi @vasia ,
I simply assumed that if the previous examples work, the iteration also
works as before. Would you like a specific test suite for that?
For the degree option not being set : the getInDegree(); getOutDegree()
methods are located inside the Vertex class. Not sure that their availability
can be changed by setting the option on the iteration. They are not computed
when the degree option is not set, but you can still see the method. Do you
have any idea how to make the degrees inaccessible?
Once this becomes clear, I can add the tests :)
Vertex-centric iteration extensions
---
Key: FLINK-1523
URL: https://issues.apache.org/jira/browse/FLINK-1523
Project: Flink
Issue Type: Improvement
Components: Gelly
Reporter: Vasia Kalavri
Assignee: Andra Lungu
We would like to make the following extensions to the vertex-centric
iterations of Gelly:
- allow vertices to access their in/out degrees and the total number of
vertices of the graph, inside the iteration.
- allow choosing the neighborhood type (in/out/all) over which to run the
vertex-centric iteration. Now, the model uses the updates of the in-neighbors
to calculate state and send messages to out-neighbors. We could add a
parameter with value in/out/all to the {{VertexUpdateFunction}} and
{{MessagingFunction}}, that would indicate the type of neighborhood.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1820) Bug in DoubleParser and FloatParser - empty String is not casted to 0
[ https://issues.apache.org/jira/browse/FLINK-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483973#comment-14483973 ] Felix Neutatz commented on FLINK-1820: -- ok, I agree with you. I will add the exceptions instead Bug in DoubleParser and FloatParser - empty String is not casted to 0 - Key: FLINK-1820 URL: https://issues.apache.org/jira/browse/FLINK-1820 Project: Flink Issue Type: Bug Components: core Affects Versions: 0.8.0, 0.9, 0.8.1 Reporter: Felix Neutatz Assignee: Felix Neutatz Priority: Critical Fix For: 0.9 Hi, I found the bug, when I wanted to read a csv file, which had a line like: ||\n If I treat it as a Tuple2Long,Long, I get as expected a tuple (0L,0L). But if I want to read it into a Double-Tuple or a Float-Tuple, I get the following error: java.lang.AssertionError: Test failed due to a org.apache.flink.api.common.io.ParseException: Line could not be parsed: '||' ParserError NUMERIC_VALUE_FORMAT_ERROR This error can be solved by adding an additional condition for empty strings in the FloatParser / DoubleParser. We definitely need the CSVReader to be able to read empty values. I can fix it like described if there are no better ideas :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1840) Job execution fails on Windows (native and Cygwin)
[
https://issues.apache.org/jira/browse/FLINK-1840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484001#comment-14484001
]
Fabian Hueske commented on FLINK-1840:
--
The exception is cause by the {{java.io.File.renameTo()}} method call. The
JavaDocs say that the behavior is ??inherently platform-depended??.
Therefore, I tried to substitute the call by an alternative but neither
{{com.google.common.io.Files.move()}} or {{java.nio.file.Files.move()}} worked.
It seems like the source file is still used/accessed by a process and can
therefore not be deleted.
Job execution fails on Windows (native and Cygwin)
--
Key: FLINK-1840
URL: https://issues.apache.org/jira/browse/FLINK-1840
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 0.9
Environment: Windows 8.1 (Cygwin and native)
Reporter: Fabian Hueske
Priority: Critical
Fix For: 0.9
Job execution fails on Windows using
- {{./bin/flink.bat}} (native)
- {{./bin/flink.sh}} (Cygwin)
- Web submission client
The following exception is thrown:
{code}
org.apache.flink.client.program.ProgramInvocationException: Could not upload
the program's JAR files tothe JobManager.
at org.apache.flink.client.program.Client.run(Client.java:341)
at org.apache.flink.client.program.Client.run(Client.java:313)
at org.apache.flink.client.program.Client.run(Client.java:306)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:63)
at
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedPro
gram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:273)
at
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:886)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:928)
Caused by: java.io.IOException: PUT operation failed: Server side error:
Cannot move staging file C:\cyg
win64\tmp\blobStore-17f49f61-bcb7-45ba-909e-56beca159fe4\incoming\temp-
to BLOB file C:\cygwin64
\tmp\blobStore-17f49f61-bcb7-45ba-909e-56beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:465)
at org.apache.flink.runtime.blob.BlobClient.put(BlobClient.java:327)
at
org.apache.flink.runtime.jobgraph.JobGraph.uploadRequiredJarFiles(JobGraph.java:457)
at
org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:288)
at
org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
at org.apache.flink.client.program.Client.run(Client.java:338)
... 15 more
Caused by: java.io.IOException: Server side error: Cannot move staging file
C:\cygwin64\tmp\blobStore-17
f49f61-bcb7-45ba-909e-56beca159fe4\incoming\temp- to BLOB file
C:\cygwin64\tmp\blobStore-17f49f6
1-bcb7-45ba-909e-56beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobClient.receivePutResponseAndCompare(BlobClient.java:491)
at
org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:461)
... 20 more
Caused by: java.io.IOException: Cannot move staging file
C:\cygwin64\tmp\blobStore-17f49f61-bcb7-45ba-90
9e-56beca159fe4\incoming\temp- to BLOB file
C:\cygwin64\tmp\blobStore-17f49f61-bcb7-45ba-909e-56
beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobServerConnection.put(BlobServerConnection.java:325)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:109)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1758) Extend Gelly's neighborhood methods
[ https://issues.apache.org/jira/browse/FLINK-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484226#comment-14484226 ] ASF GitHub Bot commented on FLINK-1758: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/576#issuecomment-90749788 Thanks for the PR @andralungu! The idea for this issue is to (a) use a combinable reduce for the simple (without vertex value) reduceOn* methods, because it will be more efficient and (b) offer groupReduceOn* versions with Collectors to allow returning more than one value per vertex. I see you have implemented (b) here, but not (a). To be more specific, the groupReduceOn* methods would be the ones with the Collectors. Basically, you'll just have to rename what you have already implemented. The reduceOn* methods will only return one value per vertex, like they do currently, but internally use the reduce operator (instead of reduceGroup). For example, if you want to compute the sum of edge values, you can use the `reduceOnEdges` method, same as you currently do. And if you want to compute the Jaccard coefficient, you need to output multiple values per vertex and for that you can use `groupReduceOnEdges`. Please let me know if you have any questions! Thanks again :-) Extend Gelly's neighborhood methods --- Key: FLINK-1758 URL: https://issues.apache.org/jira/browse/FLINK-1758 Project: Flink Issue Type: Improvement Components: Gelly Affects Versions: 0.9 Reporter: Vasia Kalavri Assignee: Andra Lungu Currently, the neighborhood methods only allow returning a single value per vertex. In many cases, it is desirable to return several or no value per vertex. This is the case in clustering coefficient computation, vertex-centric jaccard, algorithms where a vertex computes a value per edge or when a vertex computes a value only for some of its neighbors. This issue proposes to - change the current reduceOnEdges/reduceOnNeighbors methods to use combinable reduce operations where possible - provide groupReduce-versions, which will use a Collector and allow returning none or more values per vertex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-1841) WindowJoinITCase fails
[
https://issues.apache.org/jira/browse/FLINK-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Márton Balassi reassigned FLINK-1841:
-
Assignee: Márton Balassi
WindowJoinITCase fails
--
Key: FLINK-1841
URL: https://issues.apache.org/jira/browse/FLINK-1841
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Fabian Hueske
Assignee: Márton Balassi
The WindowJoinITCase failed in a Travis build:
https://travis-ci.org/fhueske/flink/jobs/57547875
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.54 sec
FAILURE! - in org.apache.flink.streaming.examples.test.join.WindowJoinITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.join.WindowJoinITCase)
Time elapsed: 4.638 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:17 but was:24
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.join.WindowJoinITCase.postSubmit(WindowJoinITCase.java:41)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1840) Job execution fails on Windows (native and Cygwin)
[
https://issues.apache.org/jira/browse/FLINK-1840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484129#comment-14484129
]
ASF GitHub Bot commented on FLINK-1840:
---
GitHub user fhueske opened a pull request:
https://github.com/apache/flink/pull/578
[FLINK-1840] Fix BLOB manager on Windows
- closes open `FileOutputStream` before the field is moved
- replaces `java.io.File.renameTo()` by `com.google.common.io.Files.move()`
to avoid platform-dependent behavior
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/flink blobWindows
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/578.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #578
commit b83ab465ba727751bc13db26989c40a94ffe378b
Author: Fabian Hueske fhue...@apache.org
Date: 2015-04-07T21:04:59Z
[FLINK-1840] Fix BLOB manager on Windows
Job execution fails on Windows (native and Cygwin)
--
Key: FLINK-1840
URL: https://issues.apache.org/jira/browse/FLINK-1840
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 0.9
Environment: Windows 8.1 (Cygwin and native)
Reporter: Fabian Hueske
Assignee: Fabian Hueske
Priority: Critical
Fix For: 0.9
Job execution fails on Windows using
- {{./bin/flink.bat}} (native)
- {{./bin/flink.sh}} (Cygwin)
- Web submission client
The following exception is thrown:
{code}
org.apache.flink.client.program.ProgramInvocationException: Could not upload
the program's JAR files tothe JobManager.
at org.apache.flink.client.program.Client.run(Client.java:341)
at org.apache.flink.client.program.Client.run(Client.java:313)
at org.apache.flink.client.program.Client.run(Client.java:306)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:63)
at
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedPro
gram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:273)
at
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:886)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:928)
Caused by: java.io.IOException: PUT operation failed: Server side error:
Cannot move staging file C:\cyg
win64\tmp\blobStore-17f49f61-bcb7-45ba-909e-56beca159fe4\incoming\temp-
to BLOB file C:\cygwin64
\tmp\blobStore-17f49f61-bcb7-45ba-909e-56beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:465)
at org.apache.flink.runtime.blob.BlobClient.put(BlobClient.java:327)
at
org.apache.flink.runtime.jobgraph.JobGraph.uploadRequiredJarFiles(JobGraph.java:457)
at
org.apache.flink.runtime.client.JobClient$.uploadJarFiles(JobClient.scala:288)
at
org.apache.flink.runtime.client.JobClient.uploadJarFiles(JobClient.scala)
at org.apache.flink.client.program.Client.run(Client.java:338)
... 15 more
Caused by: java.io.IOException: Server side error: Cannot move staging file
C:\cygwin64\tmp\blobStore-17
f49f61-bcb7-45ba-909e-56beca159fe4\incoming\temp- to BLOB file
C:\cygwin64\tmp\blobStore-17f49f6
1-bcb7-45ba-909e-56beca159fe4\cache\blob_9d5dc35b7c40b2ddda2df12d7ee8702f5683b131
at
org.apache.flink.runtime.blob.BlobClient.receivePutResponseAndCompare(BlobClient.java:491)
at
org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:461)
... 20 more
Caused by: java.io.IOException: Cannot move staging file
C:\cygwin64\tmp\blobStore-17f49f61-bcb7-45ba-90
9e-56beca159fe4\incoming\temp- to BLOB file
[GitHub] flink pull request: [FLINK-1840] Fix BLOB manager on Windows
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/578 [FLINK-1840] Fix BLOB manager on Windows - closes open `FileOutputStream` before the field is moved - replaces `java.io.File.renameTo()` by `com.google.common.io.Files.move()` to avoid platform-dependent behavior You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink blobWindows Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/578.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #578 commit b83ab465ba727751bc13db26989c40a94ffe378b Author: Fabian Hueske fhue...@apache.org Date: 2015-04-07T21:04:59Z [FLINK-1840] Fix BLOB manager on Windows --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Fix permissions on various non-executable sour...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/574 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1841) WindowJoinITCase fails
Fabian Hueske created FLINK-1841:
Summary: WindowJoinITCase fails
Key: FLINK-1841
URL: https://issues.apache.org/jira/browse/FLINK-1841
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.9
Reporter: Fabian Hueske
The WindowJoinITCase failed in a Travis build:
https://travis-ci.org/fhueske/flink/jobs/57547875
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.54 sec
FAILURE! - in org.apache.flink.streaming.examples.test.join.WindowJoinITCase
testJobWithoutObjectReuse(org.apache.flink.streaming.examples.test.join.WindowJoinITCase)
Time elapsed: 4.638 sec FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained
result. expected:17 but was:24
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:256)
at
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:242)
at
org.apache.flink.streaming.examples.test.join.WindowJoinITCase.postSubmit(WindowJoinITCase.java:41)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1813] Avoid illegalstate exception when...
GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/577 [FLINK-1813] Avoid illegalstate exception when trying to broadcast on finished operators You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink FLINK-1813 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #577 commit e3810484c6fa71f9c1dac69707c81e44380a2d1c Author: Gyula Fora gyf...@apache.org Date: 2015-04-07T20:17:18Z [FLINK-1813] Avoid illegalstate exception when trying to broadcast on finished operators --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1813) Barrier broadcast sometimes causes illegal state exception in source RecordWriter
[ https://issues.apache.org/jira/browse/FLINK-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484082#comment-14484082 ] ASF GitHub Bot commented on FLINK-1813: --- GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/577 [FLINK-1813] Avoid illegalstate exception when trying to broadcast on finished operators You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink FLINK-1813 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #577 commit e3810484c6fa71f9c1dac69707c81e44380a2d1c Author: Gyula Fora gyf...@apache.org Date: 2015-04-07T20:17:18Z [FLINK-1813] Avoid illegalstate exception when trying to broadcast on finished operators Barrier broadcast sometimes causes illegal state exception in source RecordWriter - Key: FLINK-1813 URL: https://issues.apache.org/jira/browse/FLINK-1813 Project: Flink Issue Type: Bug Components: Streaming Reporter: Gyula Fora Assignee: Gyula Fora The StreamCheckPointingITCase sometimes fails on an illegal state exception thrown when trying to broadcast barrier from the sources. I think this is caused by sending a barrier when the source is not running any more. The exception: Exception in thread Thread-341 java.lang.RuntimeException: java.lang.IllegalStateException: Partition already finished. at org.apache.flink.streaming.api.streamvertex.StreamVertex.actOnBarrier(StreamVertex.java:300) at org.apache.flink.streaming.api.streamvertex.StreamVertex.broadcastBarrierFromSource(StreamVertex.java:105) at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$receiveWithLogMessages$1$$anon$4.run(TaskManager.scala:386) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Partition already finished. at org.apache.flink.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:173) at org.apache.flink.runtime.io.network.partition.ResultPartition.checkInProduceState(ResultPartition.java:388) at org.apache.flink.runtime.io.network.partition.ResultPartition.add(ResultPartition.java:221) at org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter.writeEvent(ResultPartitionWriter.java:78) at org.apache.flink.runtime.io.network.api.writer.RecordWriter.broadcastEvent(RecordWriter.java:117) at org.apache.flink.streaming.api.collector.StreamOutput.broadcastEvent(StreamOutput.java:92) at org.apache.flink.streaming.api.streamvertex.OutputHandler.broadcastBarrier(OutputHandler.java:94) at org.apache.flink.streaming.api.streamvertex.StreamVertex.actOnBarrier(StreamVertex.java:294) ... 3 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1694] [gelly] added IterationConfigurat...
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/547#issuecomment-90745094 Hi, any opinions on this? I would like to merge it soon because it changes the api and vertex-centric related issues depend on it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1741][gelly] Adds Jaccard Similarity Me...
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/544#issuecomment-90740868 Thank you @andralungu! This looks great now ^^ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1694) Change the split between create/run of a vertex-centric iteration
[
https://issues.apache.org/jira/browse/FLINK-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484173#comment-14484173
]
ASF GitHub Bot commented on FLINK-1694:
---
Github user vasia commented on the pull request:
https://github.com/apache/flink/pull/547#issuecomment-90745094
Hi,
any opinions on this?
I would like to merge it soon because it changes the api and vertex-centric
related issues depend on it.
Thanks!
Change the split between create/run of a vertex-centric iteration
-
Key: FLINK-1694
URL: https://issues.apache.org/jira/browse/FLINK-1694
Project: Flink
Issue Type: Improvement
Components: Gelly
Reporter: Vasia Kalavri
Assignee: Vasia Kalavri
Currently, the vertex-centric API in Gelly looks like this:
{code:java}
Graph inputGaph = ... //create graph
VertexCentricIteration iteration = inputGraph.createVertexCentricIteration();
... // configure the iteration
Graph newGraph = inputGaph.runVertexCentricIteration(iteration);
{code}
We have this create/run split, in order to expose the iteration object and be
able to call the public methods of VertexCentricIteration.
However, this is not very nice and might lead to errors, if create and run
are mistakenly called on different graph objects.
One suggestion is to change this to the following:
{code:java}
VertexCentricIteration iteration = inputGraph.createVertexCentricIteration();
... // configure the iteration
Graph newGraph = iteration.result();
{code}
or to go with a single run call, where we add an IterationConfiguration
object as a parameter and we don't expose the iteration object to the user at
all:
{code:java}
IterationConfiguration parameters = ...
Graph newGraph = inputGraph.runVertexCentricIteration(parameters);
{code}
and we can also have a simplified method where no configuration is passed.
What do you think?
Personally, I like the second option a bit more.
-Vasia.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1758][gelly] Neighborhood Methods Exten...
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/576#issuecomment-90749788 Thanks for the PR @andralungu! The idea for this issue is to (a) use a combinable reduce for the simple (without vertex value) reduceOn* methods, because it will be more efficient and (b) offer groupReduceOn* versions with Collectors to allow returning more than one value per vertex. I see you have implemented (b) here, but not (a). To be more specific, the groupReduceOn* methods would be the ones with the Collectors. Basically, you'll just have to rename what you have already implemented. The reduceOn* methods will only return one value per vertex, like they do currently, but internally use the reduce operator (instead of reduceGroup). For example, if you want to compute the sum of edge values, you can use the `reduceOnEdges` method, same as you currently do. And if you want to compute the Jaccard coefficient, you need to output multiple values per vertex and for that you can use `groupReduceOnEdges`. Please let me know if you have any questions! Thanks again :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1744) Change the reference of slaves to workers to match the description of the system
[ https://issues.apache.org/jira/browse/FLINK-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484648#comment-14484648 ] Henry Saputra commented on FLINK-1744: -- No worries, we could always re-open it when needed. Change the reference of slaves to workers to match the description of the system Key: FLINK-1744 URL: https://issues.apache.org/jira/browse/FLINK-1744 Project: Flink Issue Type: Improvement Components: core, Documentation Reporter: Henry Saputra Priority: Trivial There are some references to slaves which actually mean workers. Need to change it to use workers whenever possible, unless it is needed when communicating with external system like Apache Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
