[jira] [Created] (FLINK-1436) Command-line interface verbose option (-v)
Max Michels created FLINK-1436: -- Summary: Command-line interface verbose option (-v) Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Priority: Trivial Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a difference for a novice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: ClassLoader issue when submitting Flink Streaming programs through the web cliend
Didn't we have a similar issue before the 0.7.0-incubating release as well? I thought I've tested submitting a streaming program with the web frontend for the 0.8 release and it worked. On Thu, Jan 22, 2015 at 2:31 PM, Gyula Fóra gyf...@apache.org wrote: Hey, While trying to add support for plan visualisation I also realised that streaming programs cannot be run through the flink web client. It seems to be a ClassLoader issue, which is interesting because we use the classloader set in the environment to deserialize user defined objects. This works for submitting jobs through the command line client though. I dont see why it should be different when you submit something through the commandline or the web interface. Thanks, Gyula
Re: [flink-streaming] Regarding loops in the Job Graph
Thanks for the quick answers! It is possible to use iterations, we could detect circles while building the samoa topology and convert them into iterations. It is perhaps the proper way to go. I just thought whether we could hack around it but we better avoid messing with cyclic dependences. Paris On 21 Jan 2015, at 19:36, Stephan Ewen se...@apache.org wrote: Hi Paris! The Streaming API allows you to define iterations, where parts of the stream are fed back. Do those work for you? In general, cyclic flows are a tricky thing, as the topological order of operators is needed for scheduling (may not be important for continuous streams) but also for a clear producer/consumer relationship, which is important for fault tolerance techniques. Currently, the JobManager topologically sorts the job graph and starts scheduling operators. I am surprised to hear that a graph with cyclic dependencies works... Stephan Stephan On Wed, Jan 21, 2015 at 2:57 AM, Paris Carbone par...@kth.se wrote: Hello, While implementing the SAMOA adapter for Flink-Streaming we stumbled upon the need to allow loops (or circular dependencies) in the job graph. Many incremental machine learning tasks define loops already and there is no trivial way of getting around it. In the streaming job graph builder there is only a check that does not allow the user to submit graphs with loops, however, from what Gyula told me, if the check is removed the streaming job runs as expected. Is there (still) a major reason for having this check, at least in the streaming component? Paris
Re: [jira] [Commented] (FLINK-1410) Integrate Flink version variables into website layout
+1 On Thu, Jan 22, 2015 at 2:42 PM, Max Michels (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/FLINK-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287423#comment-14287423 ] Max Michels commented on FLINK-1410: +1 We should make releasing as simple as possible. Perhaps we should encourage people who work on the website to use the Jekyll template system... Integrate Flink version variables into website layout - Key: FLINK-1410 URL: https://issues.apache.org/jira/browse/FLINK-1410 Project: Flink Issue Type: Bug Components: Project Website Reporter: Robert Metzger Attachments: ulli-howItShouldLook.png, ulli-howItactuallyLooks.png The new website layout doesn't use the variables in the website configuration. This makes releasing versions extremely hard, because one needs to manually fix all the links for every version change. The old layout of the website was respecting all variables which made releasing a new version of the website a matter of minutes (changing one file). I would highly recommend to fix FLINK-1387 first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [ANNOUNCE] Apache Flink 0.8.0 released
Awesome! Thank you very much Marton and Robert! Cheers, Fabian 2015-01-22 9:04 GMT+01:00 Robert Metzger rmetz...@apache.org: The Apache Flink team is proud to announce the next version of Apache Flink. Find the blogpost with the change log here: http://flink.apache.org/news/2015/01/21/release-0.8.html I would like to thank Márton Balassi for being the release manager of the first Apache Flink release as a top level project. Also, I would like to thank everybody who has contributed to the release. Regards, Robert
[jira] [Created] (FLINK-1432) CombineTaskTest.testCancelCombineTaskSorting sometimes fails
Robert Metzger created FLINK-1432:
-
Summary: CombineTaskTest.testCancelCombineTaskSorting sometimes
fails
Key: FLINK-1432
URL: https://issues.apache.org/jira/browse/FLINK-1432
Project: Flink
Issue Type: Bug
Reporter: Robert Metzger
We have a bunch of tests which fail only in rare cases on travis.
https://s3.amazonaws.com/archive.travis-ci.org/jobs/47783455/log.txt
{code}
Exception in thread Thread-17 java.lang.AssertionError: Canceling task
failed: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.flink.runtime.memorymanager.DefaultMemoryManager.release(DefaultMemoryManager.java:290)
at
org.apache.flink.runtime.operators.GroupReduceCombineDriver.cancel(GroupReduceCombineDriver.java:221)
at
org.apache.flink.runtime.operators.testutils.DriverTestBase.cancel(DriverTestBase.java:272)
at
org.apache.flink.runtime.operators.testutils.TaskCancelThread.run(TaskCancelThread.java:60)
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.runtime.operators.testutils.TaskCancelThread.run(TaskCancelThread.java:68)
java.lang.NullPointerException
at
org.apache.flink.runtime.memorymanager.DefaultMemoryManager.release(DefaultMemoryManager.java:291)
at
org.apache.flink.runtime.operators.GroupReduceCombineDriver.cleanup(GroupReduceCombineDriver.java:213)
at
org.apache.flink.runtime.operators.testutils.DriverTestBase.testDriverInternal(DriverTestBase.java:245)
at
org.apache.flink.runtime.operators.testutils.DriverTestBase.testDriver(DriverTestBase.java:175)
at
org.apache.flink.runtime.operators.CombineTaskTest$1.run(CombineTaskTest.java:143)
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.172 sec
FAILURE! - in org.apache.flink.runtime.operators.CombineTaskTest
testCancelCombineTaskSorting[0](org.apache.flink.runtime.operators.CombineTaskTest)
Time elapsed: 1.023 sec FAILURE!
java.lang.AssertionError: Exception was thrown despite proper canceling.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.flink.runtime.operators.CombineTaskTest.testCancelCombineTaskSorting(CombineTaskTest.java:162)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Turn lazy operator execution off for streaming jobs
On 22 Jan 2015, at 11:37, Till Rohrmann trohrm...@apache.org wrote: I'm not sure whether it is currently possible to schedule first the receiver and then the sender. Recently, I had to fix the TaskManagerTest.testRunWithForwardChannel test case where this was exactly the case. Due to first scheduling the receiver, it happened sometimes that an IllegalQueueIteratorRequestException in the method IntermediateResultPartitionManager.getIntermediateResultPartitionIterator was thrown. The partition manager complained that the producer execution ID was unknown. I assume that this has to be fixed first in order to schedule all task immediately. But Ufuk will probably know it better. On 21 Jan 2015, at 20:58, Stephan Ewen se...@apache.org wrote: - The queues would still send notifications to the JobManager that data is available, but the JM will see that the target task is already deployed (or currently being deployed). Then the info where to grab a channel from would need to be sent to the task. That mechanism also exists already. The only minor thing that needs to be adjusted would be this mechanism. It is indeed in place already (e.g. UNKNOWN input channels are updated at runtime to LOCAL or REMOTE input channels depending on the producer location), but currently the consumer tasks assume that the consumed intermediate result partition has already been created when they (the consumer task) are deployed and request the partition. When we schedule all tasks at once, we might end up in situations like the test case Till described, where we know that it is a LOCAL or REMOTE channel, but the intermediate result has not been created yet and the request fails. tl;dr: channels can be updated at runtime, but requests need to arrive after the producer created the partition.
Re: [ANNOUNCE] Apache Flink 0.8.0 released
Great idea, I will send an email to announce@ On Thu, Jan 22, 2015 at 8:56 AM, Henry Saputra henry.sapu...@gmail.com wrote: Awesome job guys! We should also send email to annou...@apache.org for the release too. It can be picked up by Sally and maybe get retweeted from official ASF Twitter handle =) - Henry On Thu, Jan 22, 2015 at 8:01 AM, Kostas Tzoumas ktzou...@apache.org wrote: Great job, thank you Marton, Robert and everyone! On Thu, Jan 22, 2015 at 12:50 AM, Fabian Hueske fhue...@apache.org wrote: Awesome! Thank you very much Marton and Robert! Cheers, Fabian 2015-01-22 9:04 GMT+01:00 Robert Metzger rmetz...@apache.org: The Apache Flink team is proud to announce the next version of Apache Flink. Find the blogpost with the change log here: http://flink.apache.org/news/2015/01/21/release-0.8.html I would like to thank Márton Balassi for being the release manager of the first Apache Flink release as a top level project. Also, I would like to thank everybody who has contributed to the release. Regards, Robert
Re: [flink-streaming] Regarding loops in the Job Graph
If this becomes a strong requirement, then we can look into relaxing the constraints (and then have some features not supported on cyclic flows). I just wanted to get a discussion started about the different angles of approach, and what may be the simplest way to do this... On Thu, Jan 22, 2015 at 4:47 AM, Paris Carbone par...@kth.se wrote: Thanks for the quick answers! It is possible to use iterations, we could detect circles while building the samoa topology and convert them into iterations. It is perhaps the proper way to go. I just thought whether we could hack around it but we better avoid messing with cyclic dependences. Paris On 21 Jan 2015, at 19:36, Stephan Ewen se...@apache.org wrote: Hi Paris! The Streaming API allows you to define iterations, where parts of the stream are fed back. Do those work for you? In general, cyclic flows are a tricky thing, as the topological order of operators is needed for scheduling (may not be important for continuous streams) but also for a clear producer/consumer relationship, which is important for fault tolerance techniques. Currently, the JobManager topologically sorts the job graph and starts scheduling operators. I am surprised to hear that a graph with cyclic dependencies works... Stephan Stephan On Wed, Jan 21, 2015 at 2:57 AM, Paris Carbone par...@kth.se wrote: Hello, While implementing the SAMOA adapter for Flink-Streaming we stumbled upon the need to allow loops (or circular dependencies) in the job graph. Many incremental machine learning tasks define loops already and there is no trivial way of getting around it. In the streaming job graph builder there is only a check that does not allow the user to submit graphs with loops, however, from what Gyula told me, if the check is removed the streaming job runs as expected. Is there (still) a major reason for having this check, at least in the streaming component? Paris
