Memory segment error when migrating functional code from Flink 0.9 to 0.8
Hello everyone, I am implementing a graph algorithm as part of a course and I will also add it to the Flink- Gelly examples. My problem is that I started developing it in the Gelly repository, which runs on flink 0.9. It works like a charm there, but in order to test in on a cluster to see its real capabilities, I need to move it to the course repository, which runs on flink 0.8. Initially, I thought this migration should occur without incidents since flink 0.8 is more stable. Instead, I got the following exception: java.lang.IllegalArgumentException: *Too few memory segments provided. Hash Table needs at least 33 memory segments.* at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:238) at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:227) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279) at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257) at java.lang.Thread.run(Thread.java:745) This is the code for Gelly, where all tests pass: https://github.com/andralungu/flink-graph/tree/minSpanningTree Unfortunately, the code for the course is private, so you cannot actually see it... maybe @aalexandrov can do something about the privacy settings of this repo. https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm Thank you! Andra
Re: Eclipse JDT, Java 8, lambdas
I did try the 4.5 M4 release and it did not go straightforward. -- View this message in context: http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Eclipse-JDT-Java-8-lambdas-tp3664p3688.html Sent from the Apache Flink (Incubator) Mailing List archive. mailing list archive at Nabble.com.
Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8
Hi Andra, have you tried increasing the number of network buffers in your cluster? You can control by the configuration value: taskmanager.network.numberOfBuffers: #numberBuffers Greets, Till On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu lungu.an...@gmail.com wrote: Hello everyone, I am implementing a graph algorithm as part of a course and I will also add it to the Flink- Gelly examples. My problem is that I started developing it in the Gelly repository, which runs on flink 0.9. It works like a charm there, but in order to test in on a cluster to see its real capabilities, I need to move it to the course repository, which runs on flink 0.8. Initially, I thought this migration should occur without incidents since flink 0.8 is more stable. Instead, I got the following exception: java.lang.IllegalArgumentException: *Too few memory segments provided. Hash Table needs at least 33 memory segments.* at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:238) at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:227) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279) at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257) at java.lang.Thread.run(Thread.java:745) This is the code for Gelly, where all tests pass: https://github.com/andralungu/flink-graph/tree/minSpanningTree Unfortunately, the code for the course is private, so you cannot actually see it... maybe @aalexandrov can do something about the privacy settings of this repo. https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm Thank you! Andra
Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8
This is actually a problem of the number of memory segments available to the hash table for the solution set. For complex pipelines, memory currently gets too fragmented. There are two workarounds, until we do the dynamic memory management, or break it into shorter pipelines: Break the job up into subparts, or move the solution set to the user code part of the heap. There is a flag to attach to the Delta Iterations, see DeltaIteration#setSolutionSetUnManaged() Greetinigs, Stephan On Mon, Feb 9, 2015 at 10:32 AM, Till Rohrmann trohrm...@apache.org wrote: Hi Andra, have you tried increasing the number of network buffers in your cluster? You can control by the configuration value: taskmanager.network.numberOfBuffers: #numberBuffers Greets, Till On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu lungu.an...@gmail.com wrote: Hello everyone, I am implementing a graph algorithm as part of a course and I will also add it to the Flink- Gelly examples. My problem is that I started developing it in the Gelly repository, which runs on flink 0.9. It works like a charm there, but in order to test in on a cluster to see its real capabilities, I need to move it to the course repository, which runs on flink 0.8. Initially, I thought this migration should occur without incidents since flink 0.8 is more stable. Instead, I got the following exception: java.lang.IllegalArgumentException: *Too few memory segments provided. Hash Table needs at least 33 memory segments.* at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:238) at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:227) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279) at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257) at java.lang.Thread.run(Thread.java:745) This is the code for Gelly, where all tests pass: https://github.com/andralungu/flink-graph/tree/minSpanningTree Unfortunately, the code for the course is private, so you cannot actually see it... maybe @aalexandrov can do something about the privacy settings of this repo. https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm Thank you! Andra
Re: Eclipse JDT, Java 8, lambdas
Hey, it seems that 4.4.2 also includes the fix (https://projects.eclipse.org/projects/eclipse/releases/4.4.2/bugs) and will be released end of february. I will try Eclipse Luna SR2 RC2 today and check if it is working. Regards, Timo On 09.02.2015 10:05, Nam-Luc Tran wrote: I did try the 4.5 M4 release and it did not go straightforward. -- View this message in context: http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Eclipse-JDT-Java-8-lambdas-tp3664p3688.html Sent from the Apache Flink (Incubator) Mailing List archive. mailing list archive at Nabble.com.
[jira] [Created] (FLINK-1494) Build fails on BlobCacheTest
Fabian Hueske created FLINK-1494:
Summary: Build fails on BlobCacheTest
Key: FLINK-1494
URL: https://issues.apache.org/jira/browse/FLINK-1494
Project: Flink
Issue Type: Bug
Components: Local Runtime, TaskManager
Environment: Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.7.0_65, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: linux, version: 3.16.0-4-amd64, arch: amd64, family: unix
Reporter: Fabian Hueske
Building Flink with Maven repeatedly fails with the following error:
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.283
sec FAILURE! - in org.apache.flink.runtime.blob.BlobCacheTest
testBlobCache(org.apache.flink.runtime.blob.BlobCacheTest) Time
elapsed: 127.282 sec FAILURE!
java.lang.AssertionError: Could not connect to BlobServer at address
0.0.0.0/0.0.0.0:56760
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.runtime.blob.BlobCacheTest.testBlobCache(BlobCacheTest.java:109)
java.io.IOException: Could not connect to BlobServer at address
0.0.0.0/0.0.0.0:52657
at
org.apache.flink.runtime.blob.BlobClient.init(BlobClient.java:61)
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testLibraryCacheManagerCleanup(BlobLibraryCacheManagerTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at
org.apache.flink.runtime.blob.BlobClient.init(BlobClient.java:59)
... 24 more
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.299
sec FAILURE! - in
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest
testLibraryCacheManagerCleanup(org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest)
Time elapsed: 127.298 sec FAILURE!
java.lang.AssertionError: Could not connect to BlobServer at address
0.0.0.0/0.0.0.0:52657
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testLibraryCacheManagerCleanup(BlobLibraryCacheManagerTest.java:108)
Results :
Failed tests:
BlobCacheTest.testBlobCache:109 Could not connect to BlobServer at
address 0.0.0.0/0.0.0.0:56760
BlobLibraryCacheManagerTest.testLibraryCacheManagerCleanup:108 Could
not connect to BlobServer at address
[jira] [Created] (FLINK-1497) No documentation on MiniCluster
Sergey Dudoladov created FLINK-1497: --- Summary: No documentation on MiniCluster Key: FLINK-1497 URL: https://issues.apache.org/jira/browse/FLINK-1497 Project: Flink Issue Type: Improvement Affects Versions: 0.9 Reporter: Sergey Dudoladov Priority: Trivial It looks like the Flink docs do not show how to run a MiniCluster. It might be worth to document this feature and add relevant scripts to the /bin folder, e.g. start_mini.sh and stop_mini.sh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Planning Release 0.8.1
Went through the streaming commits with Gyula and assembled the ones missing. Pushing it as soon as travis passes. On Fri, Feb 6, 2015 at 2:26 PM, Robert Metzger rmetz...@apache.org wrote: It seems that quite a few important fixes still need some work until they are ready. I'll extend the deadline to Monday morning (CET), since we can not vote during the weekends anyways. On Fri, Feb 6, 2015 at 1:16 PM, Stephan Ewen se...@apache.org wrote: I am working on fixes for local split assignment and custom input split. Would be good to include those as well... On Fri, Feb 6, 2015 at 12:52 PM, Aljoscha Krettek aljos...@apache.org wrote: @robert, yes, will do On Fri, Feb 6, 2015 at 12:28 PM, Márton Balassi balassi.mar...@gmail.com wrote: Found a streaming bug, Gyula fixed it. Pushing it soon to both master and branch-0.8. On Fri, Feb 6, 2015 at 11:51 AM, Robert Metzger rmetz...@apache.org wrote: @Aljoscha, can you merge the backported fix to the release-0.8 branch when its ready? On Fri, Feb 6, 2015 at 11:39 AM, Aljoscha Krettek aljos...@apache.org wrote: I have a fix for this user-discovered bug: https://issues.apache.org/jira/browse/FLINK-1463?jql=project%20%3D%20FLINK%20AND%20assignee%20%3D%20currentUser()%20AND%20resolution%20%3D%20Unresolved in this PR: https://github.com/apache/flink/pull/353 This should probably also be back-ported to 0.8.1 On Thu, Feb 5, 2015 at 3:54 PM, Aljoscha Krettek aljos...@apache.org wrote: I have the PR ready. :D On Thu, Feb 5, 2015 at 2:43 PM, Stephan Ewen se...@apache.org wrote: I would like to add it to 0.8.1. People have asked for it... On Thu, Feb 5, 2015 at 2:36 PM, Aljoscha Krettek aljos...@apache.org wrote: Do we want to have HadoopInputFormat support for Scala in 0.8.1? On Thu, Feb 5, 2015 at 2:29 PM, Stephan Ewen se...@apache.org wrote: I think we need to make a pass through the recent 0.9 commits and cherry pick some more into 0.8.1. There were quite a few bug fixes. Also, this one is rather critical and pending: https://github.com/apache/flink/pull/318 On Thu, Feb 5, 2015 at 2:27 PM, Robert Metzger rmetz...@apache.org wrote: Hi guys, I would like to bundle a minor bugfix release for Flink soon. Some users were complaining about incomplete Kryo support, in particular for Avro. Also, we fixed some other issues which are easy to to port to 0.8.1 (some of them are already in the branch). I would like to start the vote on the 0.8.1 release in roughly 26 hours. So please merge bugfixes you would like to have in 0.8.1 into the release-0.8 branch in the next 24 hours. There are currently 2 open JIRAs assigned to 0.8.1, one with a pending pull request: https://issues.apache.org/jira/browse/FLINK-1422?jql=project%20%3D%20FLINK%20AND%20status%20%3D%20Open%20AND%20fixVersion%20%3D%200.8.1%20ORDER%20BY%20status%20DESC%2C%20priority%20DESC Please chime in if you are suggesting a different plan for the 0.8.1 release. - Robert
Re: Planning Release 0.8.1
Cool. I'm currently also testing my last change (kryo serializers). I think I'll start creating the release candidate in one hour. On Mon, Feb 9, 2015 at 2:24 PM, Márton Balassi balassi.mar...@gmail.com wrote: Went through the streaming commits with Gyula and assembled the ones missing. Pushing it as soon as travis passes. On Fri, Feb 6, 2015 at 2:26 PM, Robert Metzger rmetz...@apache.org wrote: It seems that quite a few important fixes still need some work until they are ready. I'll extend the deadline to Monday morning (CET), since we can not vote during the weekends anyways. On Fri, Feb 6, 2015 at 1:16 PM, Stephan Ewen se...@apache.org wrote: I am working on fixes for local split assignment and custom input split. Would be good to include those as well... On Fri, Feb 6, 2015 at 12:52 PM, Aljoscha Krettek aljos...@apache.org wrote: @robert, yes, will do On Fri, Feb 6, 2015 at 12:28 PM, Márton Balassi balassi.mar...@gmail.com wrote: Found a streaming bug, Gyula fixed it. Pushing it soon to both master and branch-0.8. On Fri, Feb 6, 2015 at 11:51 AM, Robert Metzger rmetz...@apache.org wrote: @Aljoscha, can you merge the backported fix to the release-0.8 branch when its ready? On Fri, Feb 6, 2015 at 11:39 AM, Aljoscha Krettek aljos...@apache.org wrote: I have a fix for this user-discovered bug: https://issues.apache.org/jira/browse/FLINK-1463?jql=project%20%3D%20FLINK%20AND%20assignee%20%3D%20currentUser()%20AND%20resolution%20%3D%20Unresolved in this PR: https://github.com/apache/flink/pull/353 This should probably also be back-ported to 0.8.1 On Thu, Feb 5, 2015 at 3:54 PM, Aljoscha Krettek aljos...@apache.org wrote: I have the PR ready. :D On Thu, Feb 5, 2015 at 2:43 PM, Stephan Ewen se...@apache.org wrote: I would like to add it to 0.8.1. People have asked for it... On Thu, Feb 5, 2015 at 2:36 PM, Aljoscha Krettek aljos...@apache.org wrote: Do we want to have HadoopInputFormat support for Scala in 0.8.1? On Thu, Feb 5, 2015 at 2:29 PM, Stephan Ewen se...@apache.org wrote: I think we need to make a pass through the recent 0.9 commits and cherry pick some more into 0.8.1. There were quite a few bug fixes. Also, this one is rather critical and pending: https://github.com/apache/flink/pull/318 On Thu, Feb 5, 2015 at 2:27 PM, Robert Metzger rmetz...@apache.org wrote: Hi guys, I would like to bundle a minor bugfix release for Flink soon. Some users were complaining about incomplete Kryo support, in particular for Avro. Also, we fixed some other issues which are easy to to port to 0.8.1 (some of them are already in the branch). I would like to start the vote on the 0.8.1 release in roughly 26 hours. So please merge bugfixes you would like to have in 0.8.1 into the release-0.8 branch in the next 24 hours. There are currently 2 open JIRAs assigned to 0.8.1, one with a pending pull request: https://issues.apache.org/jira/browse/FLINK-1422?jql=project%20%3D%20FLINK%20AND%20status%20%3D%20Open%20AND%20fixVersion%20%3D%200.8.1%20ORDER%20BY%20status%20DESC%2C%20priority%20DESC Please chime in if you are suggesting a different plan for the 0.8.1 release. - Robert
[DISCUSS] Distributed TPC-H DataGenerator for flink-contrib
Hi, we recently added the flink-contrib module for user contributed tools etc. On one of the last weekends, I've created a distributed tpch generator, based on this libary: https://github.com/airlift/tpch (which is from a PrestoDB developer and available on Maven central). You can find my code here: https://github.com/rmetzger/scratch/tree/distributed-tpch-generator It contains two examples: a) a full TPC data generator (as a flink program): https://github.com/rmetzger/scratch/blob/distributed-tpch-generator/src/main/java/flink/generators/programs/TPCHGenerator.java b) an example which generates two TPC-H tables on-the-fly to join them: https://github.com/rmetzger/scratch/blob/distributed-tpch-generator/src/main/java/flink/generators/programs/TPCHGeneratorExample.java Before I spend time on integrating it into the flink-contrib package, I was wondering if the community is willing this contribution to Flink. Best, Robert
