[jira] [Created] (FLINK-1624) Build of old sources fails due to git-commit-id plugin

[Max Michels (JIRA)]

Max Michels created FLINK-1624:
--

 Summary: Build of old sources fails due to git-commit-id plugin
 Key: FLINK-1624
 URL: https://issues.apache.org/jira/browse/FLINK-1624
 Project: Flink
  Issue Type: Bug
Reporter: Max Michels
Assignee: Max Michels
Priority: Minor
 Fix For: 0.6-incubating


Builds for Flink (Stratosphere) versions  0.6.0 fail because of a bug in the 
maven git-commit-id plugin.

https://github.com/ktoso/maven-git-commit-id-plugin/issues/61



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1621) Create a generalized combine function

[Max Michels (JIRA)]

Max Michels created FLINK-1621:
--

 Summary: Create a generalized combine function
 Key: FLINK-1621
 URL: https://issues.apache.org/jira/browse/FLINK-1621
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Runtime
Affects Versions: 0.9
Reporter: Max Michels
 Fix For: 0.9


Flink allows combiners which accept a type {{I}} and combine the values of this 
type into type {{O}}. In Google Dataflow, combiners are more generalized. They 
accept an Input {{I}}, produce an intermediate combine value of {{T}}, and 
finally an output {{O}}. Flink's combiners are like the {{SimpleCombineFn}} in 
Google Dataflow.

Right now, we translate the {{KeyedCombineFn}} into a {{SortPartition}} 
followed by a {{MapPartition}} to emulate the Combiner's behavior. Rudimentary 
performance tests showed that this behavior causes a significant increase in 
run time compared to the proper Combine implementation.

Let's implement a more generalized Combiner to create a better mapping from 
Google Dataflow to Flink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1599) TypeComperator

[Max Michels (JIRA)]

Max Michels created FLINK-1599:
--

 Summary: TypeComperator
 Key: FLINK-1599
 URL: https://issues.apache.org/jira/browse/FLINK-1599
 Project: Flink
  Issue Type: Bug
  Components: Distributed Runtime
Affects Versions: 0.8.0
Reporter: Max Michels
Priority: Minor


If you create a custom type comparator by subclassing {{TypeComperator}} and 
implement {{int extractKeys(Object record, Object[] target, int index)}} and 
{{TypeComparator[] getFlatComparators()}} to return 0 and no type comparators 
respectively, the {{coGroup}} operator (possibly others) find matching elements 
although no comparators have been specified.

In this case, the expected behavior for a CoGroup would be that only elements 
from one side are supplied in the CoGroup method.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1436) Command-line interface verbose option (-v)

[Max Michels (JIRA)]

Max Michels created FLINK-1436:
--

 Summary: Command-line interface verbose option (-v)
 Key: FLINK-1436
 URL: https://issues.apache.org/jira/browse/FLINK-1436
 Project: Flink
  Issue Type: Improvement
  Components: Start-Stop Scripts
Reporter: Max Michels
Priority: Trivial


Let me run just a basic Flink job and add the verbose flag. It's a general 
option, so let me add it as a first parameter:

 ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
 hdfs:///input hdfs:///output9
Invalid action!
./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS]
  general options:
 -h,--help  Show the help for the CLI Frontend.
 -v,--verbose   Print more detailed error messages.

Action run compiles and runs a program.

  Syntax: run [OPTIONS] jar-file arguments
  run action arguments:
 -c,--class classname   Class with the program entry point (main
  method or getPlan() method. Only needed
  if the JAR file does not specify the class
  in its manifest.
 -m,--jobmanager host:port  Address of the JobManager (master) to
  which to connect. Use this flag to connect
  to a different JobManager than the one
  specified in the configuration.
 -p,--parallelism parallelism   The parallelism with which to run the
  program. Optional flag to override the
  default value specified in the
  configuration.

Action info displays information about a program.
  info action arguments:
 -c,--class classname   Class with the program entry point (main
  method or getPlan() method. Only needed
  if the JAR file does not specify the class
  in its manifest.
 -e,--executionplan   Show optimized execution plan of the
  program (JSON)
 -m,--jobmanager host:port  Address of the JobManager (master) to
  which to connect. Use this flag to connect
  to a different JobManager than the one
  specified in the configuration.
 -p,--parallelism parallelism   The parallelism with which to run the
  program. Optional flag to override the
  default value specified in the
  configuration.

Action list lists running and finished programs.
  list action arguments:
 -m,--jobmanager host:port   Address of the JobManager (master) to which
   to connect. Use this flag to connect to a
   different JobManager than the one specified
   in the configuration.
 -r,--running  Show running programs and their JobIDs
 -s,--scheduledShow scheduled prorgrams and their JobIDs

Action cancel cancels a running program.
  cancel action arguments:
 -i,--jobid jobIDJobID of program to cancel
 -m,--jobmanager host:port   Address of the JobManager (master) to which
   to connect. Use this flag to connect to a
   different JobManager than the one specified
   in the configuration.

What just happened? This results in a lot of output which is usually generated 
if you use the --help option on command-line tools. If your terminal window is 
large enough, then you will see a tiny message:

Please specify an action. I did specify an action. Strange. If you read the 
help messages carefully you see, that general options belong to the action.

 ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
 hdfs:///input hdfs:///output9

For the sake of mitigating user frustration, let us also accept -v as the first 
argument. It may seem trivial for the day-to-day Flink user but makes a 
difference for a novice.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)