[jira] [Commented] (FLINK-1512) Add CsvReader for reading into POJOs.

[Chiwan Park (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319492#comment-14319492
 ] 

Chiwan Park commented on FLINK-1512:


[~StephanEwen] Thanks for your explanation. I'll implement this feature and 
send a pull request.

> Add CsvReader for reading into POJOs.
> -
>
> Key: FLINK-1512
> URL: https://issues.apache.org/jira/browse/FLINK-1512
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API, Scala API
>Reporter: Robert Metzger
>Assignee: Chiwan Park
>Priority: Minor
>  Labels: starter
>
> Currently, the {{CsvReader}} supports only TupleXX types. 
> It would be nice if users were also able to read into POJOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-785) Add Chained operators for AllReduce and AllGroupReduce

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319276#comment-14319276
 ] 

ASF GitHub Bot commented on FLINK-785:
--

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/370#discussion_r24633778
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllGroupReduceCombineDriver.java
 ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.operators.chaining;
+
+import java.util.ArrayList;
+
+import org.apache.flink.api.common.functions.FlatCombineFunction;
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerFactory;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.operators.RegularPactTask;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ChainedAllGroupReduceCombineDriver extends 
ChainedDriver {
+   private static final Logger LOG = 
LoggerFactory.getLogger(ChainedAllGroupReduceCombineDriver.class);
+
+   // 

+   private FlatCombineFunction combiner;
+   private TypeSerializer serializer;
+
+   private volatile boolean running = true;
+
+   private final ArrayList values = new ArrayList();
+
+   // 

+   @Override
+   public void setup(AbstractInvokable parent) {
+   @SuppressWarnings("unchecked")
+   final FlatCombineFunction com = 
RegularPactTask.instantiateUserCode(this.config, userCodeClassLoader, 
FlatCombineFunction.class);
+   this.combiner = com;
+   FunctionUtils.setFunctionRuntimeContext(com, 
getUdfRuntimeContext());
+
+   this.objectReuseEnabled = 
this.executionConfig.isObjectReuseEnabled();
+
+   final TypeSerializerFactory serializerFactory = 
this.config.getInputSerializer(0, this.userCodeClassLoader);
+   this.serializer = serializerFactory.getSerializer();
+
+   if (LOG.isDebugEnabled()) {
+   LOG.debug("ChainedAllGroupReduceCombineDriver object 
reuse: " + (this.objectReuseEnabled ? "ENABLED" : "DISABLED") + ".");
+   }
+   }
+
+   @Override
+   public void openTask() throws Exception {
+   final Configuration stubConfig = 
this.config.getStubParameters();
+   RegularPactTask.openUserCode(this.combiner, stubConfig);
+   }
+
+   @Override
+   public void closeTask() throws Exception {
+   if (!this.running) {
+   return;
+   }
+   RegularPactTask.closeUserCode(this.combiner);
+   }
+
+   @Override
+   public void cancelTask() {
+   this.running = false;
+   }
+
+   // 

+   @Override
+   public Function getStub() {
+   return this.combiner;
+   }
+
+   @Override
+   public String getTaskName() {
+   return this.taskName;
+   }
+
+   // 

+   @Override
+   public void collect(T record) {
+   try {
+   values.add(objectReuseEnabled ? record : 
serializer.copy(record));
+   if (values.size() > 1) {
--- End diff --

is there a more reasonable value for this? (basically anything 

[GitHub] flink pull request: [FLINK-785] Chained AllReduce / AllGroupReduce...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/370#discussion_r24633778
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllGroupReduceCombineDriver.java
 ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.operators.chaining;
+
+import java.util.ArrayList;
+
+import org.apache.flink.api.common.functions.FlatCombineFunction;
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerFactory;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.operators.RegularPactTask;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ChainedAllGroupReduceCombineDriver extends 
ChainedDriver {
+   private static final Logger LOG = 
LoggerFactory.getLogger(ChainedAllGroupReduceCombineDriver.class);
+
+   // 

+   private FlatCombineFunction combiner;
+   private TypeSerializer serializer;
+
+   private volatile boolean running = true;
+
+   private final ArrayList values = new ArrayList();
+
+   // 

+   @Override
+   public void setup(AbstractInvokable parent) {
+   @SuppressWarnings("unchecked")
+   final FlatCombineFunction com = 
RegularPactTask.instantiateUserCode(this.config, userCodeClassLoader, 
FlatCombineFunction.class);
+   this.combiner = com;
+   FunctionUtils.setFunctionRuntimeContext(com, 
getUdfRuntimeContext());
+
+   this.objectReuseEnabled = 
this.executionConfig.isObjectReuseEnabled();
+
+   final TypeSerializerFactory serializerFactory = 
this.config.getInputSerializer(0, this.userCodeClassLoader);
+   this.serializer = serializerFactory.getSerializer();
+
+   if (LOG.isDebugEnabled()) {
+   LOG.debug("ChainedAllGroupReduceCombineDriver object 
reuse: " + (this.objectReuseEnabled ? "ENABLED" : "DISABLED") + ".");
+   }
+   }
+
+   @Override
+   public void openTask() throws Exception {
+   final Configuration stubConfig = 
this.config.getStubParameters();
+   RegularPactTask.openUserCode(this.combiner, stubConfig);
+   }
+
+   @Override
+   public void closeTask() throws Exception {
+   if (!this.running) {
+   return;
+   }
+   RegularPactTask.closeUserCode(this.combiner);
+   }
+
+   @Override
+   public void cancelTask() {
+   this.running = false;
+   }
+
+   // 

+   @Override
+   public Function getStub() {
+   return this.combiner;
+   }
+
+   @Override
+   public String getTaskName() {
+   return this.taskName;
+   }
+
+   // 

+   @Override
+   public void collect(T record) {
+   try {
+   values.add(objectReuseEnabled ? record : 
serializer.copy(record));
+   if (values.size() > 1) {
--- End diff --

is there a more reasonable value for this? (basically anything whose 
reasoning goes beyond "i felt like using 1")


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature

[jira] [Commented] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319271#comment-14319271
 ] 

ASF GitHub Bot commented on FLINK-1521:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/392

[FLINK-1521] Chained operators respect reuse



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/incubator-flink flink-1521

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #392


commit b3f8ec9a107a2baf743468c9efdf28da9f49cf79
Author: zentol 
Date:   2015-02-12T19:36:00Z

[FLINK-1521] Chained operators respect reuse




> Some Chained Drivers do not respect object-reuse/non-reuse flag
> ---
>
> Key: FLINK-1521
> URL: https://issues.apache.org/jira/browse/FLINK-1521
> Project: Flink
>  Issue Type: Bug
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1521] Chained operators respect reuse

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/392

[FLINK-1521] Chained operators respect reuse



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/incubator-flink flink-1521

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #392


commit b3f8ec9a107a2baf743468c9efdf28da9f49cf79
Author: zentol 
Date:   2015-02-12T19:36:00Z

[FLINK-1521] Chained operators respect reuse




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink commit comment: 988602b031b63deb9640b115b798e9ae54fb7357

[mbalassi]

Github user mbalassi commented on commit 
988602b031b63deb9640b115b798e9ae54fb7357:


https://github.com/apache/flink/commit/988602b031b63deb9640b115b798e9ae54fb7357#commitcomment-9725249
  
In tools/create_release_files.sh:
In tools/create_release_files.sh on line 150:
This way you use the generate pom that is in the outside repo and might be 
from a different branch. Not a big difference, but might lead to unexpected 
results. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-1436) Command-line interface verbose option & error reporting

[Max Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Max Michels resolved FLINK-1436.

   Resolution: Fixed
Fix Version/s: 0.9

> Command-line interface verbose option & error reporting
> ---
>
> Key: FLINK-1436
> URL: https://issues.apache.org/jira/browse/FLINK-1436
> Project: Flink
>  Issue Type: Improvement
>  Components: Start-Stop Scripts
>Reporter: Max Michels
>Assignee: Max Michels
>Priority: Trivial
>  Labels: starter, usability
> Fix For: 0.9
>
>
> Let me run just a basic Flink job and add the verbose flag. It's a general 
> option, so let me add it as a first parameter:
> > ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> Invalid action!
> ./flink  [GENERAL_OPTIONS] [ARGUMENTS]
>   general options:
>  -h,--help  Show the help for the CLI Frontend.
>  -v,--verbose   Print more detailed error messages.
> Action "run" compiles and runs a program.
>   Syntax: run [OPTIONS]  
>   "run" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "info" displays information about a program.
>   "info" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -e,--executionplan   Show optimized execution plan of the
>   program (JSON)
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "list" lists running and finished programs.
>   "list" action arguments:
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
>  -r,--running  Show running programs and their JobIDs
>  -s,--scheduledShow scheduled prorgrams and their JobIDs
> Action "cancel" cancels a running program.
>   "cancel" action arguments:
>  -i,--jobid JobID of program to cancel
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
> What just happened? This results in a lot of output which is usually 
> generated if you use the --help option on command-line tools. If your 
> terminal window is large enough, then you will see a tiny message:
> "Please specify an action". I did specify an action. Strange. If you read the 
> help messages carefully you see, that "general options" belong to the action.
> > ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> For the sake of mitigating user frustration, let us also accept -v as the 
> first argument. It may seem trivial for the day-to-day Flink user but makes a 
> difference for a novice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-1521:
---

Assignee: Chesnay Schepler

> Some Chained Drivers do not respect object-reuse/non-reuse flag
> ---
>
> Key: FLINK-1521
> URL: https://issues.apache.org/jira/browse/FLINK-1521
> Project: Flink
>  Issue Type: Bug
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Johannes (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318779#comment-14318779
 ] 

Johannes commented on FLINK-1531:
-

Thanks for the quick turnaround
Tried it locally and it works now - thanks

But as a general though, isn't using exceptions to signal EOF much to costly?

Quick informative write up on stack overflow:

http://stackoverflow.com/questions/299068/how-slow-are-java-exceptions

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Fix For: 0.9, 0.8.1
>
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318681#comment-14318681
 ] 

ASF GitHub Bot commented on FLINK-1529:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/385#issuecomment-74122398
  
Nice work. Looks really good. There are only some minor issues on Travis 
with the ```JobManagerFailsITCase``` which fails because the archive actor is 
not properly shut down before restarting the JobManager.

LGTM


> Improve JobManager startup robustness
> -
>
> Key: FLINK-1529
> URL: https://issues.apache.org/jira/browse/FLINK-1529
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, the JobManager is creates asynchronously (as an actor). If its 
> initialization fails (for various reasons), the process does not terminate 
> and gives only vague log message that an actor creation failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/385#issuecomment-74122398
  
Nice work. Looks really good. There are only some minor issues on Travis 
with the ```JobManagerFailsITCase``` which fails because the archive actor is 
not properly shut down before restarting the JobManager.

LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318657#comment-14318657
 ] 

ASF GitHub Bot commented on FLINK-1529:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24602575
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -653,11 +693,87 @@ object JobManager {
 (archiveCount, profilingEnabled, cleanupInterval, executionRetries, 
delayBetweenRetries)
   }
 
-  def startActor(configuration: Configuration)(implicit actorSystem: 
ActorSystem): ActorRef = {
-startActor(Props(classOf[JobManager], configuration))
+  /**
+   * Create the job manager members as (instanceManager, scheduler, 
libraryCacheManager,
+   *  archiverProps, accumulatorManager, profiler, 
defaultExecutionRetries,
+   *  delayBetweenRetries, timeout)
+   *
+   * @param configuration The configuration from which to parse the config 
values.
+   * @return The members for a default JobManager.
+   */
+  def createJobManagerComponents(configuration: Configuration) :
+(InstanceManager, FlinkScheduler, BlobLibraryCacheManager,
+  Props, AccumulatorManager, Option[Props], Int, Long, FiniteDuration, 
Int) = {
+
+val timeout: FiniteDuration = AkkaUtils.getTimeout(configuration)
+
+val (archiveCount, profilingEnabled, cleanupInterval, 
executionRetries, delayBetweenRetries) =
+  parseConfiguration(configuration)
+
+val archiveProps: Props = Props(classOf[MemoryArchivist], archiveCount)
+
+val profilerProps: Option[Props] = if (profilingEnabled) {
+  Some(Props(classOf[JobManagerProfiler]))
+} else {
+  None
+}
+
+val accumulatorManager: AccumulatorManager = new 
AccumulatorManager(Math.min(1, archiveCount))
+
+var blobServer: BlobServer = null
+var instanceManager: InstanceManager = null
+var scheduler: FlinkScheduler = null
+var libraryCacheManager: BlobLibraryCacheManager = null
+
+try {
+  blobServer = new BlobServer(configuration)
+  instanceManager = new InstanceManager()
+  scheduler = new FlinkScheduler()
+  libraryCacheManager = new BlobLibraryCacheManager(blobServer, 
cleanupInterval)
+
+  instanceManager.addInstanceListener(scheduler)
+}
+catch {
+  case t: Throwable => {
+if (libraryCacheManager != null) {
+  libraryCacheManager.shutdown()
+}
+if (scheduler != null) {
+  scheduler.shutdown()
+}
+if (instanceManager != null) {
+  instanceManager.shutdown()
+}
+if (blobServer != null) {
+  blobServer.shutdown()
+}
+throw t;
+  }
+}
+
+(instanceManager, scheduler, libraryCacheManager, archiveProps, 
accumulatorManager,
+  profilerProps, executionRetries, delayBetweenRetries, timeout, 
archiveCount)
+  }
+
+  def startActor(configuration: Configuration, actorSystem: ActorSystem): 
ActorRef = {
+
+val (instanceManager, scheduler, libraryCacheManager, archiveProps, 
accumulatorManager,
+  profilerProps, executionRetries, delayBetweenRetries,
+  timeout, _) = createJobManagerComponents(configuration)
+
+val profiler: Option[ActorRef] =
+ profilerProps.map( props => actorSystem.actorOf(props, 
PROFILER_NAME) )
--- End diff --

Nice functional style :-)


> Improve JobManager startup robustness
> -
>
> Key: FLINK-1529
> URL: https://issues.apache.org/jira/browse/FLINK-1529
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, the JobManager is creates asynchronously (as an actor). If its 
> initialization fails (for various reasons), the process does not terminate 
> and gives only vague log message that an actor creation failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24602575
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -653,11 +693,87 @@ object JobManager {
 (archiveCount, profilingEnabled, cleanupInterval, executionRetries, 
delayBetweenRetries)
   }
 
-  def startActor(configuration: Configuration)(implicit actorSystem: 
ActorSystem): ActorRef = {
-startActor(Props(classOf[JobManager], configuration))
+  /**
+   * Create the job manager members as (instanceManager, scheduler, 
libraryCacheManager,
+   *  archiverProps, accumulatorManager, profiler, 
defaultExecutionRetries,
+   *  delayBetweenRetries, timeout)
+   *
+   * @param configuration The configuration from which to parse the config 
values.
+   * @return The members for a default JobManager.
+   */
+  def createJobManagerComponents(configuration: Configuration) :
+(InstanceManager, FlinkScheduler, BlobLibraryCacheManager,
+  Props, AccumulatorManager, Option[Props], Int, Long, FiniteDuration, 
Int) = {
+
+val timeout: FiniteDuration = AkkaUtils.getTimeout(configuration)
+
+val (archiveCount, profilingEnabled, cleanupInterval, 
executionRetries, delayBetweenRetries) =
+  parseConfiguration(configuration)
+
+val archiveProps: Props = Props(classOf[MemoryArchivist], archiveCount)
+
+val profilerProps: Option[Props] = if (profilingEnabled) {
+  Some(Props(classOf[JobManagerProfiler]))
+} else {
+  None
+}
+
+val accumulatorManager: AccumulatorManager = new 
AccumulatorManager(Math.min(1, archiveCount))
+
+var blobServer: BlobServer = null
+var instanceManager: InstanceManager = null
+var scheduler: FlinkScheduler = null
+var libraryCacheManager: BlobLibraryCacheManager = null
+
+try {
+  blobServer = new BlobServer(configuration)
+  instanceManager = new InstanceManager()
+  scheduler = new FlinkScheduler()
+  libraryCacheManager = new BlobLibraryCacheManager(blobServer, 
cleanupInterval)
+
+  instanceManager.addInstanceListener(scheduler)
+}
+catch {
+  case t: Throwable => {
+if (libraryCacheManager != null) {
+  libraryCacheManager.shutdown()
+}
+if (scheduler != null) {
+  scheduler.shutdown()
+}
+if (instanceManager != null) {
+  instanceManager.shutdown()
+}
+if (blobServer != null) {
+  blobServer.shutdown()
+}
+throw t;
+  }
+}
+
+(instanceManager, scheduler, libraryCacheManager, archiveProps, 
accumulatorManager,
+  profilerProps, executionRetries, delayBetweenRetries, timeout, 
archiveCount)
+  }
+
+  def startActor(configuration: Configuration, actorSystem: ActorSystem): 
ActorRef = {
+
+val (instanceManager, scheduler, libraryCacheManager, archiveProps, 
accumulatorManager,
+  profilerProps, executionRetries, delayBetweenRetries,
+  timeout, _) = createJobManagerComponents(configuration)
+
+val profiler: Option[ActorRef] =
+ profilerProps.map( props => actorSystem.actorOf(props, 
PROFILER_NAME) )
--- End diff --

Nice functional style :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1461) Add sortPartition operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318656#comment-14318656
 ] 

ASF GitHub Bot commented on FLINK-1461:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/381#issuecomment-74120225
  
Looks good to me (I'm uncertain regarding the optimizer changes)


> Add sortPartition operator
> --
>
> Key: FLINK-1461
> URL: https://issues.apache.org/jira/browse/FLINK-1461
> Project: Flink
>  Issue Type: New Feature
>  Components: Java API, Local Runtime, Optimizer, Scala API
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Minor
>
> A {{sortPartition()}} operator can be used to
> * sort the input of a {{mapPartition()}} operator
> * enforce a certain sorting of the input of a given operator of a program. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1461][api-extending] Add SortPartition ...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/381#issuecomment-74120225
  
Looks good to me (I'm uncertain regarding the optimizer changes)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1508) Remove AkkaUtils.ask to encourage explicit future handling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318642#comment-14318642
 ] 

ASF GitHub Bot commented on FLINK-1508:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/384#issuecomment-74119549
  
I vote to merge this quickly and fix issues as they appear. 
The change touches a lot of different parts of the code and is predestined 
to become unmergeable quickly.


> Remove AkkaUtils.ask to encourage explicit future handling
> --
>
> Key: FLINK-1508
> URL: https://issues.apache.org/jira/browse/FLINK-1508
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> {{AkkaUtils.ask}} asks another actor and awaits its response. Since this 
> constitutes a blocking call, it might be potentially harmful when used in an 
> actor thread. In order to encourage developers to program asynchronously I 
> propose to remove this helper function. That forces the developer to handle 
> futures explicitly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318641#comment-14318641
 ] 

ASF GitHub Bot commented on FLINK-1529:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24602187
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -530,54 +515,109 @@ Actor with ActorLogMessages with ActorLogging {
 log.error(t, "Could not properly unregister job {} form the 
library cache.", jobID)
 }
   }
-
-  private def checkJavaVersion(): Unit = {
-if (System.getProperty("java.version").substring(0, 3).toDouble < 1.7) 
{
-  log.warning("Warning: Flink is running with Java 6. " +
-"Java 6 is not maintained any more by Oracle or the OpenJDK 
community. " +
-"Flink currently supports Java 6, but may not in future releases," 
+
-" due to the unavailability of bug fixes security patched.")
-}
-  }
 }
 
 object JobManager {
+  
   import ExecutionMode._
+
   val LOG = LoggerFactory.getLogger(classOf[JobManager])
+
   val FAILURE_RETURN_CODE = 1
+
   val JOB_MANAGER_NAME = "jobmanager"
   val EVENT_COLLECTOR_NAME = "eventcollector"
   val ARCHIVE_NAME = "archive"
   val PROFILER_NAME = "profiler"
 
   def main(args: Array[String]): Unit = {
+
+// startup checks and logging
 EnvironmentInformation.logEnvironmentInfo(LOG, "JobManager")
-val (configuration, executionMode, listeningAddress) = parseArgs(args)
+checkJavaVersion()
 
-  if(SecurityUtils.isSecurityEnabled) {
+val (configuration: Configuration,
+ executionMode: ExecutionMode,
+ listeningAddress:  Option[(String, Int)]) =
+try {
+  parseArgs(args)
+}
+catch {
+  case t: Throwable => {
+LOG.error(t.getMessage(), t)
+System.exit(FAILURE_RETURN_CODE)
+null
+  }
+}
+
+try {
+  if (SecurityUtils.isSecurityEnabled) {
 LOG.info("Security is enabled. Starting secure JobManager.")
 SecurityUtils.runSecured(new FlinkSecuredRunner[Unit] {
   override def run(): Unit = {
-start(configuration, executionMode, listeningAddress)
+runJobManager(configuration, executionMode, listeningAddress)
   }
 })
   } else {
-start(configuration, executionMode, listeningAddress)
+runJobManager(configuration, executionMode, listeningAddress)
+  }
+}
+catch {
+  case t: Throwable => {
+LOG.error("Failed to start JobManager.", t)
+System.exit(FAILURE_RETURN_CODE)
   }
+}
   }
 
-  def start(configuration: Configuration, executionMode: ExecutionMode,
-listeningAddress : Option[(String, Int)]): Unit = {
-val jobManagerSystem = AkkaUtils.createActorSystem(configuration, 
listeningAddress)
 
-startActor(Props(new JobManager(configuration) with 
WithWebServer))(jobManagerSystem)
+  def runJobManager(configuration: Configuration,
+executionMode: ExecutionMode,
+listeningAddress: Option[(String, Int)]) : Unit = {
+
+LOG.info("Starting JobManager")
+LOG.debug("Starting JobManager actor system")
 
-if(executionMode.equals(LOCAL)){
-  TaskManager.startActorWithConfiguration("", configuration,
-localAkkaCommunication = false, localTaskManagerCommunication = 
true)(jobManagerSystem)
+val jobManagerSystem = try {
+  AkkaUtils.createActorSystem(configuration, listeningAddress)
 }
+catch {
+  case t: Throwable => {
+if (t.isInstanceOf[org.jboss.netty.channel.ChannelException]) {
+  val cause = t.getCause()
+  if (cause != null && 
t.getCause().isInstanceOf[java.net.BindException]) {
+val address = listeningAddress match {
+  case Some((host, port)) => host + ":" + port
+  case None => "unknown"
+}
 
-jobManagerSystem.awaitTermination()
+throw new Exception("Unable to create JobManager at address " 
+ address + ": " + cause.getMessage(), t)
+  }
+}
+throw new Exception("Could not create JobManager actor system", t)
+  }
+}
+
+try {
+  LOG.debug("Starting JobManager actor")
+
+  startActor(configuration, jobManagerSystem)
+
+  if(executionMode.equals(LOCAL)){
+LOG.info("Starting embedded TaskManag

[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24602187
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -530,54 +515,109 @@ Actor with ActorLogMessages with ActorLogging {
 log.error(t, "Could not properly unregister job {} form the 
library cache.", jobID)
 }
   }
-
-  private def checkJavaVersion(): Unit = {
-if (System.getProperty("java.version").substring(0, 3).toDouble < 1.7) 
{
-  log.warning("Warning: Flink is running with Java 6. " +
-"Java 6 is not maintained any more by Oracle or the OpenJDK 
community. " +
-"Flink currently supports Java 6, but may not in future releases," 
+
-" due to the unavailability of bug fixes security patched.")
-}
-  }
 }
 
 object JobManager {
+  
   import ExecutionMode._
+
   val LOG = LoggerFactory.getLogger(classOf[JobManager])
+
   val FAILURE_RETURN_CODE = 1
+
   val JOB_MANAGER_NAME = "jobmanager"
   val EVENT_COLLECTOR_NAME = "eventcollector"
   val ARCHIVE_NAME = "archive"
   val PROFILER_NAME = "profiler"
 
   def main(args: Array[String]): Unit = {
+
+// startup checks and logging
 EnvironmentInformation.logEnvironmentInfo(LOG, "JobManager")
-val (configuration, executionMode, listeningAddress) = parseArgs(args)
+checkJavaVersion()
 
-  if(SecurityUtils.isSecurityEnabled) {
+val (configuration: Configuration,
+ executionMode: ExecutionMode,
+ listeningAddress:  Option[(String, Int)]) =
+try {
+  parseArgs(args)
+}
+catch {
+  case t: Throwable => {
+LOG.error(t.getMessage(), t)
+System.exit(FAILURE_RETURN_CODE)
+null
+  }
+}
+
+try {
+  if (SecurityUtils.isSecurityEnabled) {
 LOG.info("Security is enabled. Starting secure JobManager.")
 SecurityUtils.runSecured(new FlinkSecuredRunner[Unit] {
   override def run(): Unit = {
-start(configuration, executionMode, listeningAddress)
+runJobManager(configuration, executionMode, listeningAddress)
   }
 })
   } else {
-start(configuration, executionMode, listeningAddress)
+runJobManager(configuration, executionMode, listeningAddress)
+  }
+}
+catch {
+  case t: Throwable => {
+LOG.error("Failed to start JobManager.", t)
+System.exit(FAILURE_RETURN_CODE)
   }
+}
   }
 
-  def start(configuration: Configuration, executionMode: ExecutionMode,
-listeningAddress : Option[(String, Int)]): Unit = {
-val jobManagerSystem = AkkaUtils.createActorSystem(configuration, 
listeningAddress)
 
-startActor(Props(new JobManager(configuration) with 
WithWebServer))(jobManagerSystem)
+  def runJobManager(configuration: Configuration,
+executionMode: ExecutionMode,
+listeningAddress: Option[(String, Int)]) : Unit = {
+
+LOG.info("Starting JobManager")
+LOG.debug("Starting JobManager actor system")
 
-if(executionMode.equals(LOCAL)){
-  TaskManager.startActorWithConfiguration("", configuration,
-localAkkaCommunication = false, localTaskManagerCommunication = 
true)(jobManagerSystem)
+val jobManagerSystem = try {
+  AkkaUtils.createActorSystem(configuration, listeningAddress)
 }
+catch {
+  case t: Throwable => {
+if (t.isInstanceOf[org.jboss.netty.channel.ChannelException]) {
+  val cause = t.getCause()
+  if (cause != null && 
t.getCause().isInstanceOf[java.net.BindException]) {
+val address = listeningAddress match {
+  case Some((host, port)) => host + ":" + port
+  case None => "unknown"
+}
 
-jobManagerSystem.awaitTermination()
+throw new Exception("Unable to create JobManager at address " 
+ address + ": " + cause.getMessage(), t)
+  }
+}
+throw new Exception("Could not create JobManager actor system", t)
+  }
+}
+
+try {
+  LOG.debug("Starting JobManager actor")
+
+  startActor(configuration, jobManagerSystem)
+
+  if(executionMode.equals(LOCAL)){
+LOG.info("Starting embedded TaskManager for JobManager's LOCAL 
mode execution")
+
+TaskManager.startActorWithConfiguration("", configuration,
+  localAkkaCommunication = false, localTaskManagerCommunication = 
true)(jobManagerSystem)
+  }
+
+  

[GitHub] flink pull request: [FLINK-1508] Removes AkkaUtil.ask

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/384#issuecomment-74119549
  
I vote to merge this quickly and fix issues as they appear. 
The change touches a lot of different parts of the code and is predestined 
to become unmergeable quickly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318635#comment-14318635
 ] 

ASF GitHub Bot commented on FLINK-1529:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24601897
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -530,54 +515,109 @@ Actor with ActorLogMessages with ActorLogging {
 log.error(t, "Could not properly unregister job {} form the 
library cache.", jobID)
 }
   }
-
-  private def checkJavaVersion(): Unit = {
-if (System.getProperty("java.version").substring(0, 3).toDouble < 1.7) 
{
-  log.warning("Warning: Flink is running with Java 6. " +
-"Java 6 is not maintained any more by Oracle or the OpenJDK 
community. " +
-"Flink currently supports Java 6, but may not in future releases," 
+
-" due to the unavailability of bug fixes security patched.")
-}
-  }
 }
 
 object JobManager {
+  
   import ExecutionMode._
+
   val LOG = LoggerFactory.getLogger(classOf[JobManager])
+
   val FAILURE_RETURN_CODE = 1
+
   val JOB_MANAGER_NAME = "jobmanager"
   val EVENT_COLLECTOR_NAME = "eventcollector"
   val ARCHIVE_NAME = "archive"
   val PROFILER_NAME = "profiler"
 
   def main(args: Array[String]): Unit = {
+
+// startup checks and logging
 EnvironmentInformation.logEnvironmentInfo(LOG, "JobManager")
-val (configuration, executionMode, listeningAddress) = parseArgs(args)
+checkJavaVersion()
 
-  if(SecurityUtils.isSecurityEnabled) {
+val (configuration: Configuration,
+ executionMode: ExecutionMode,
+ listeningAddress:  Option[(String, Int)]) =
+try {
+  parseArgs(args)
+}
+catch {
+  case t: Throwable => {
+LOG.error(t.getMessage(), t)
+System.exit(FAILURE_RETURN_CODE)
+null
--- End diff --

Do we need the ```null``` expression after the ```System.exit```?


> Improve JobManager startup robustness
> -
>
> Key: FLINK-1529
> URL: https://issues.apache.org/jira/browse/FLINK-1529
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, the JobManager is creates asynchronously (as an actor). If its 
> initialization fails (for various reasons), the process does not terminate 
> and gives only vague log message that an actor creation failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24601897
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -530,54 +515,109 @@ Actor with ActorLogMessages with ActorLogging {
 log.error(t, "Could not properly unregister job {} form the 
library cache.", jobID)
 }
   }
-
-  private def checkJavaVersion(): Unit = {
-if (System.getProperty("java.version").substring(0, 3).toDouble < 1.7) 
{
-  log.warning("Warning: Flink is running with Java 6. " +
-"Java 6 is not maintained any more by Oracle or the OpenJDK 
community. " +
-"Flink currently supports Java 6, but may not in future releases," 
+
-" due to the unavailability of bug fixes security patched.")
-}
-  }
 }
 
 object JobManager {
+  
   import ExecutionMode._
+
   val LOG = LoggerFactory.getLogger(classOf[JobManager])
+
   val FAILURE_RETURN_CODE = 1
+
   val JOB_MANAGER_NAME = "jobmanager"
   val EVENT_COLLECTOR_NAME = "eventcollector"
   val ARCHIVE_NAME = "archive"
   val PROFILER_NAME = "profiler"
 
   def main(args: Array[String]): Unit = {
+
+// startup checks and logging
 EnvironmentInformation.logEnvironmentInfo(LOG, "JobManager")
-val (configuration, executionMode, listeningAddress) = parseArgs(args)
+checkJavaVersion()
 
-  if(SecurityUtils.isSecurityEnabled) {
+val (configuration: Configuration,
+ executionMode: ExecutionMode,
+ listeningAddress:  Option[(String, Int)]) =
+try {
+  parseArgs(args)
+}
+catch {
+  case t: Throwable => {
+LOG.error(t.getMessage(), t)
+System.exit(FAILURE_RETURN_CODE)
+null
--- End diff --

Do we need the ```null``` expression after the ```System.exit```?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318633#comment-14318633
 ] 

ASF GitHub Bot commented on FLINK-1529:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24601755
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -149,7 +134,7 @@ Actor with ActorLogMessages with ActorLogging {
 
   override def receiveWithLogMessages: Receive = {
 case RegisterTaskManager(connectionInfo, hardwareInformation, 
numberOfSlots) =>
-  val taskManager = sender
+  val taskManager = sender()
--- End diff --

This does not work with older Akka versions, which we are using with the 
Hadoop-2.0.0-alpha profile. I think in older Akka version it is a val.


> Improve JobManager startup robustness
> -
>
> Key: FLINK-1529
> URL: https://issues.apache.org/jira/browse/FLINK-1529
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 0.9
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> Currently, the JobManager is creates asynchronously (as an actor). If its 
> initialization fails (for various reasons), the process does not terminate 
> and gives only vague log message that an actor creation failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/385#discussion_r24601755
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -149,7 +134,7 @@ Actor with ActorLogMessages with ActorLogging {
 
   override def receiveWithLogMessages: Receive = {
 case RegisterTaskManager(connectionInfo, hardwareInformation, 
numberOfSlots) =>
-  val taskManager = sender
+  val taskManager = sender()
--- End diff --

This does not work with older Akka versions, which we are using with the 
Hadoop-2.0.0-alpha profile. I think in older Akka version it is a val.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1533.

Resolution: Fixed

Fixed in bc1432a2f605f97a27b0075c9f54b46c74e42d56

> NullPointerException in 
> SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
> --
>
> Key: FLINK-1533
> URL: https://issues.apache.org/jira/browse/FLINK-1533
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The scheduler can fail with a NullPointerException if it is not possible to 
> allocate a {{SharedSlot}} from an {{Instance}}. In the method 
> {{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not 
> checked that it is not null. This can cause a program to fail if there is a 
> race condition in scheduling different tasks on the same instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-1531.

   Resolution: Fixed
Fix Version/s: 0.8.1
   0.9

Fixed for 0.9 in 21f47d9c69441c17b5f90ea2c7cb8f4d47f7fcb5
Fixed for 0.8 in a669d785e74534731c19e982e47457475420f4a7

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Fix For: 0.9, 0.8.1
>
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318603#comment-14318603
 ] 

ASF GitHub Bot commented on FLINK-1531:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/391


> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318604#comment-14318604
 ] 

ASF GitHub Bot commented on FLINK-1533:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/388


> NullPointerException in 
> SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
> --
>
> Key: FLINK-1533
> URL: https://issues.apache.org/jira/browse/FLINK-1533
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The scheduler can fail with a NullPointerException if it is not possible to 
> allocate a {{SharedSlot}} from an {{Instance}}. In the method 
> {{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not 
> checked that it is not null. This can cause a program to fail if there is a 
> race condition in scheduling different tasks on the same instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/388


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/391


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-1536) GSoC project: Graph partitioning operators for Gelly

[Vasia Kalavri (JIRA)]

Vasia Kalavri created FLINK-1536:


 Summary: GSoC project: Graph partitioning operators for Gelly
 Key: FLINK-1536
 URL: https://issues.apache.org/jira/browse/FLINK-1536
 Project: Flink
  Issue Type: New Feature
  Components: Gelly, Java API
Reporter: Vasia Kalavri
Priority: Minor


Smart graph partitioning can significantly improve the performance and 
scalability of graph analysis applications. Depending on the computation 
pattern, a graph partitioning algorithm divides the graph into (maybe 
overlapping) subgraphs, optimizing some objective. For example, if 
communication is performed across graph edges, one might want to minimize the 
edges that cross from one partition to another.

The problem of graph partitioning is a well studied problem and several 
algorithms have been proposed in the literature. The goal of this project would 
be to choose a few existing partitioning techniques and implement the 
corresponding graph partitioning operators for Gelly.

Some related literature can be found [here| 
http://www.citeulike.org/user/vasiakalavri/tag/graph-partitioning].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74107768
  
Ok, I'll merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318544#comment-14318544
 ] 

ASF GitHub Bot commented on FLINK-1533:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/388#issuecomment-74107842
  
Ok, I'll merge it.


> NullPointerException in 
> SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
> --
>
> Key: FLINK-1533
> URL: https://issues.apache.org/jira/browse/FLINK-1533
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The scheduler can fail with a NullPointerException if it is not possible to 
> allocate a {{SharedSlot}} from an {{Instance}}. In the method 
> {{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not 
> checked that it is not null. This can cause a program to fail if there is a 
> race condition in scheduling different tasks on the same instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/388#issuecomment-74107842
  
Ok, I'll merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318543#comment-14318543
 ] 

ASF GitHub Bot commented on FLINK-1531:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74107768
  
Ok, I'll merge it.


> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [builds] Print Java process stack traces of st...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/389#issuecomment-74100276
  
I really like this feature. Will make all our lives less frustrating ;-)

On Thu, Feb 12, 2015 at 3:33 PM, Robert Metzger 
wrote:

> Very nicely implemented! My bash scripts are certainly uglier ;)
>
> +1 for merging it
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74099482
  
Tests for release-0.8 are running.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318465#comment-14318465
 ] 

ASF GitHub Bot commented on FLINK-1531:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74099482
  
Tests for release-0.8 are running.


> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74099311
  
We should also put the fix into the release-0.8 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318463#comment-14318463
 ] 

ASF GitHub Bot commented on FLINK-1531:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74099311
  
We should also put the fix into the release-0.8 branch.


> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318440#comment-14318440
 ] 

ASF GitHub Bot commented on FLINK-1531:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74097085
  
Looks good
+1


> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/391#issuecomment-74097085
  
Looks good
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318436#comment-14318436
 ] 

Till Rohrmann commented on FLINK-1531:
--

Thanks for the great bug description with the additional UnitTest [~jkirsch]. 

The reason for the failure was that the KryoSerializer did not properly 
forwarded the EOFException of the underlying stream. The EOFException is used 
by the system to know when all data of an input channel has been consumed. 

I created a PR for this and once all tests pass on Travis, I'll commit it. 

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318430#comment-14318430
 ] 

ASF GitHub Bot commented on FLINK-1531:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/391

[FLINK-1531] Fixes KryoSerializer's EOFException forwarding

Adds proper EOFException forwarding to the KryoSerializer.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixKryoUnderflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #391


commit 8e9c94b50c55f31c311d60ff63bd42108962e9a4
Author: Till Rohrmann 
Date:   2015-02-12T15:50:46Z

[FLINK-1531] [runtime] Adds proper EOFException forwarding to 
KryoSerializer.




> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/391

[FLINK-1531] Fixes KryoSerializer's EOFException forwarding

Adds proper EOFException forwarding to the KryoSerializer.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixKryoUnderflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #391


commit 8e9c94b50c55f31c311d60ff63bd42108962e9a4
Author: Till Rohrmann 
Date:   2015-02-12T15:50:46Z

[FLINK-1531] [runtime] Adds proper EOFException forwarding to 
KryoSerializer.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-1424) bin/flink run does not recognize -c parameter anymore

[Max Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Max Michels resolved FLINK-1424.

   Resolution: Fixed
Fix Version/s: 0.9

> bin/flink run does not recognize -c parameter anymore
> -
>
> Key: FLINK-1424
> URL: https://issues.apache.org/jira/browse/FLINK-1424
> Project: Flink
>  Issue Type: Bug
>  Components: TaskManager
>Affects Versions: master
>Reporter: Carsten Brandt
>Assignee: Max Michels
> Fix For: 0.9
>
>
> bin/flink binary does not recognize `-c` parameter anymore which specifies 
> the class to run:
> {noformat}
> $ ./flink run "/path/to/target/impro3-ws14-flink-1.0-SNAPSHOT.jar" -c 
> de.tu_berlin.impro3.flink.etl.FollowerGraphGenerator /tmp/flink/testgraph.txt 
> 1
> usage: emma-experiments-impro3-ss14-flink
>[-?]
> emma-experiments-impro3-ss14-flink: error: unrecognized arguments: '-c'
> {noformat}
> before this command worked fine and executed the job.
> I tracked it down to the following commit using `git bisect`:
> {noformat}
> 93eadca782ee8c77f89609f6d924d73021dcdda9 is the first bad commit
> commit 93eadca782ee8c77f89609f6d924d73021dcdda9
> Author: Alexander Alexandrov 
> Date:   Wed Dec 24 13:49:56 2014 +0200
> [FLINK-1027] [cli] Added support for '--' and '-' prefixed tokens in CLI 
> program arguments.
> 
> This closes #278
> :04 04 a1358e6f7fe308b4d51a47069f190a29f87fdeda 
> d6f11bbc9444227d5c6297ec908e44b9644289a9 Mflink-clients
> {noformat}
> https://github.com/apache/flink/commit/93eadca782ee8c77f89609f6d924d73021dcdda9



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1436) Command-line interface verbose option & error reporting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318405#comment-14318405
 ] 

ASF GitHub Bot commented on FLINK-1436:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/331


> Command-line interface verbose option & error reporting
> ---
>
> Key: FLINK-1436
> URL: https://issues.apache.org/jira/browse/FLINK-1436
> Project: Flink
>  Issue Type: Improvement
>  Components: Start-Stop Scripts
>Reporter: Max Michels
>Assignee: Max Michels
>Priority: Trivial
>  Labels: starter, usability
>
> Let me run just a basic Flink job and add the verbose flag. It's a general 
> option, so let me add it as a first parameter:
> > ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> Invalid action!
> ./flink  [GENERAL_OPTIONS] [ARGUMENTS]
>   general options:
>  -h,--help  Show the help for the CLI Frontend.
>  -v,--verbose   Print more detailed error messages.
> Action "run" compiles and runs a program.
>   Syntax: run [OPTIONS]  
>   "run" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "info" displays information about a program.
>   "info" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -e,--executionplan   Show optimized execution plan of the
>   program (JSON)
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "list" lists running and finished programs.
>   "list" action arguments:
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
>  -r,--running  Show running programs and their JobIDs
>  -s,--scheduledShow scheduled prorgrams and their JobIDs
> Action "cancel" cancels a running program.
>   "cancel" action arguments:
>  -i,--jobid JobID of program to cancel
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
> What just happened? This results in a lot of output which is usually 
> generated if you use the --help option on command-line tools. If your 
> terminal window is large enough, then you will see a tiny message:
> "Please specify an action". I did specify an action. Strange. If you read the 
> help messages carefully you see, that "general options" belong to the action.
> > ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> For the sake of mitigating user frustration, let us also accept -v as the 
> first argument. It may seem trivial for the day-to-day Flink user but makes a 
> difference for a novice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/331


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1436) Command-line interface verbose option & error reporting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318361#comment-14318361
 ] 

ASF GitHub Bot commented on FLINK-1436:
---

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74088065
  
@rmetzger Thanks but I think it should be "This closes #331" :)


> Command-line interface verbose option & error reporting
> ---
>
> Key: FLINK-1436
> URL: https://issues.apache.org/jira/browse/FLINK-1436
> Project: Flink
>  Issue Type: Improvement
>  Components: Start-Stop Scripts
>Reporter: Max Michels
>Assignee: Max Michels
>Priority: Trivial
>  Labels: starter, usability
>
> Let me run just a basic Flink job and add the verbose flag. It's a general 
> option, so let me add it as a first parameter:
> > ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> Invalid action!
> ./flink  [GENERAL_OPTIONS] [ARGUMENTS]
>   general options:
>  -h,--help  Show the help for the CLI Frontend.
>  -v,--verbose   Print more detailed error messages.
> Action "run" compiles and runs a program.
>   Syntax: run [OPTIONS]  
>   "run" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "info" displays information about a program.
>   "info" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -e,--executionplan   Show optimized execution plan of the
>   program (JSON)
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "list" lists running and finished programs.
>   "list" action arguments:
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
>  -r,--running  Show running programs and their JobIDs
>  -s,--scheduledShow scheduled prorgrams and their JobIDs
> Action "cancel" cancels a running program.
>   "cancel" action arguments:
>  -i,--jobid JobID of program to cancel
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
> What just happened? This results in a lot of output which is usually 
> generated if you use the --help option on command-line tools. If your 
> terminal window is large enough, then you will see a tiny message:
> "Please specify an action". I did specify an action. Strange. If you read the 
> help messages carefully you see, that "general options" belong to the action.
> > ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> For the sake of mitigating user frustration, let us also accept -v as the 
> first argument. It may seem trivial for the day-to-day Flink user but make

[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...

[mxm]

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74088065
  
@rmetzger Thanks but I think it should be "This closes #331" :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Johannes (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318314#comment-14318314
 ] 

Johannes commented on FLINK-1531:
-

I just had a look again at the logging again

Right at the beginning I see

{code}
15:56:17.009 [main] INFO  org.apache.flink.api.java.typeutils.TypeExtractor  - 
Class class TestKryoIterationSerializer$DomainObject is not a valid POJO type
{code}

Which is clear, as this one should be managed by Kryos serializer.
Looks like if the process fails on this issue, it would be clearer.
It seems that the serializer is only invoked for fields. So if I wrap the 
object into a value holder or tuple class it works fine, without the default 
constructor or any of the setters or getters.

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Johannes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes updated FLINK-1531:

Priority: Minor  (was: Major)

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Priority: Minor
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1535) Use usercode class loader to serialize/deserialize accumulators

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-1535:
---

 Summary: Use usercode class loader to serialize/deserialize 
accumulators
 Key: FLINK-1535
 URL: https://issues.apache.org/jira/browse/FLINK-1535
 Project: Flink
  Issue Type: Bug
  Components: JobManager
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.9


Currently, accumulators are transferred via simple Akka Messages. Since the 
accumulators may be user defined types, we should use the user code class 
loader for code loading when deserializing them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-1510) Make AvroInputFormat splittable

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-1510.
---
   Resolution: Fixed
Fix Version/s: 0.8.1
   0.9

Resolved for 0.9 in http://git-wip-us.apache.org/repos/asf/flink/commit/3241214c

Resolved for 0.8.1 in 
http://git-wip-us.apache.org/repos/asf/flink/commit/e7c99962

> Make AvroInputFormat splittable
> ---
>
> Key: FLINK-1510
> URL: https://issues.apache.org/jira/browse/FLINK-1510
> Project: Flink
>  Issue Type: Improvement
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.9, 0.8.1
>
>
> Avro supports splitting files using synchronization points. IFs have to read 
> between these points.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318285#comment-14318285
 ] 

ASF GitHub Bot commented on FLINK-1391:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74080897
  
Merged to release-0.8 in: 
http://git-wip-us.apache.org/repos/asf/flink/commit/10f89c73


> Kryo fails to properly serialize avro collection types
> --
>
> Key: FLINK-1391
> URL: https://issues.apache.org/jira/browse/FLINK-1391
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.8.1
>
>
> Before FLINK-610, Avro was the default generic serializer.
> Now, special types coming from Avro are handled by Kryo .. which seems to 
> cause errors like:
> {code}
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.NullPointerException
>   at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
>   at 
> org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
>   at 
> org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
>   at 
> org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
>   at 
> org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
>   at 
> org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
>   at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
>   at 
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>   at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74080897
  
Merged to release-0.8 in: 
http://git-wip-us.apache.org/repos/asf/flink/commit/10f89c73


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318286#comment-14318286
 ] 

ASF GitHub Bot commented on FLINK-1391:
---

Github user rmetzger closed the pull request at:

https://github.com/apache/flink/pull/386


> Kryo fails to properly serialize avro collection types
> --
>
> Key: FLINK-1391
> URL: https://issues.apache.org/jira/browse/FLINK-1391
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.8.1
>
>
> Before FLINK-610, Avro was the default generic serializer.
> Now, special types coming from Avro are handled by Kryo .. which seems to 
> cause errors like:
> {code}
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.NullPointerException
>   at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
>   at 
> org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
>   at 
> org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
>   at 
> org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
>   at 
> org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
>   at 
> org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
>   at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
>   at 
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>   at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...

[rmetzger]

Github user rmetzger closed the pull request at:

https://github.com/apache/flink/pull/386


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318281#comment-14318281
 ] 

ASF GitHub Bot commented on FLINK-1533:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/388#issuecomment-74080518
  
Good catch.

+1 to merge


> NullPointerException in 
> SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
> --
>
> Key: FLINK-1533
> URL: https://issues.apache.org/jira/browse/FLINK-1533
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The scheduler can fail with a NullPointerException if it is not possible to 
> allocate a {{SharedSlot}} from an {{Instance}}. In the method 
> {{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not 
> checked that it is not null. This can cause a program to fail if there is a 
> race condition in scheduling different tasks on the same instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/388#issuecomment-74080518
  
Good catch.

+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [builds] Notify dev@f.a.o about travis-ci buil...

[uce]

Github user uce closed the pull request at:

https://github.com/apache/flink/pull/390


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [builds] Notify dev@f.a.o about travis-ci buil...

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/390#issuecomment-74080451
  
From Travis docs:
```
I'm not receiving any build notifications #

The most common cause for not receiving build notifications, beyond not 
having a user account on Travis CI, is the use of an email address that's not 
registered and verified on GitHub. See above on how to change the email address 
to one that's registered or make sure to add the email address used in this 
repository to your verified email addresses on GitHub.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [builds] Notify dev@f.a.o about travis-ci buil...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/390

[builds] Notify dev@f.a.o about travis-ci build status changes

Related discussion [about how to notice failing master 
faster](http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Master-not-building-and-how-to-notice-it-faster-in-the-future-td3375.html).

With this Travis config, we will get an email to dev@f.a.o for every status 
change from success => failure and failure => success.

We could also just notify on failures. Feedback is welcome.

If we want to merge this, we have to make sure that bui...@travis-ci.org is 
allowed to post to our dev mailing list.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/incubator-flink travis_notify

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/390.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #390


commit 2e56503c581078935847036ab8627e4815a84189
Author: Ufuk Celebi 
Date:   2015-02-12T14:28:54Z

[builds] Notify dev@f.a.o about travis-ci build status changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [builds] Print Java process stack traces of st...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/389#issuecomment-74079695
  
Very nicely implemented! My bash scripts are certainly uglier ;)

+1 for merging it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-1534) GSoC project: Distributed pattern matching over Flink streaming

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLINK-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi updated FLINK-1534:
--
Description: 
Pattern matching over streams is an important application. The general 
structure of a streaming pattern matching is the following:

If A event follows B event then trigger some computation.

The support for this feature is associated with complex event processing 
systems, however it is also adoptable for distributed setting, however it poses 
additional challenges.

The Google Summer of Code student volunteering for this project is expected to 
have general knowledge of distributed systems and Java/Scala coding skills. The 
project includes research and implementation oriented taks. 

  was:
Pattern matching over streams is an important application. The general 
structure of a streaming pattern matching is the following:

If A event follows B event then trigger some computation.

The support for this feature is associated with complex event processing 
systems, however it is also adoptable for distributed setting, however it poses 
additional challenges.


> GSoC project: Distributed pattern matching over Flink streaming
> ---
>
> Key: FLINK-1534
> URL: https://issues.apache.org/jira/browse/FLINK-1534
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming
>Reporter: Márton Balassi
>Priority: Minor
>  Labels: gsoc2015, java, scala
>
> Pattern matching over streams is an important application. The general 
> structure of a streaming pattern matching is the following:
> If A event follows B event then trigger some computation.
> The support for this feature is associated with complex event processing 
> systems, however it is also adoptable for distributed setting, however it 
> poses additional challenges.
> The Google Summer of Code student volunteering for this project is expected 
> to have general knowledge of distributed systems and Java/Scala coding 
> skills. The project includes research and implementation oriented taks. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1534) GSoC project: Distributed pattern matching over Flink streaming

[JIRA]

Márton Balassi created FLINK-1534:
-

 Summary: GSoC project: Distributed pattern matching over Flink 
streaming
 Key: FLINK-1534
 URL: https://issues.apache.org/jira/browse/FLINK-1534
 Project: Flink
  Issue Type: New Feature
  Components: Streaming
Reporter: Márton Balassi
Priority: Minor


Pattern matching over streams is an important application. The general 
structure of a streaming pattern matching is the following:

If A event follows B event then trigger some computation.

The support for this feature is associated with complex event processing 
systems, however it is also adoptable for distributed setting, however it poses 
additional challenges.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [builds] Print Java process stack traces of st...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/389

[builds] Print Java process stack traces of stalled builds

After discussion with @rmetzger we came up with the following idea to 
manually monitor stalled builds and to print the Java process stack traces if 
they do stall.

Furthermore, a stalling build is killed with a != 0 exit code and really 
fails the Travis build instead of just timing it out.

For testing, I've a branch where I block a test on purpose. The output can 
be found here: https://travis-ci.org/uce/incubator-flink/jobs/50491750

```

==
Maven produced no output for 300 seconds.

==

==
The following Java processes are running (JPS)

==
21670 Jps
12703 surefirebooter5726841031685552603.jar
2220 Launcher

==
Printing stack trace of Java process 12703

==
2015-02-12 14:14:42
Full thread dump OpenJDK 64-Bit Server VM (23.25-b01 mixed mode):

"Attach Listener" daemon prio=10 tid=0x7f4e18001000 nid=0x54ed runnable 
[0x]
   java.lang.Thread.State: RUNNABLE

"Low Memory Detector" daemon prio=10 tid=0x7f4e64112000 nid=0x31d2 
runnable [0x]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x7f4e6411 nid=0x31ce 
waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x7f4e6410d000 nid=0x31cd 
waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x7f4e6410b000 nid=0x31cb 
runnable [0x]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x7f4e640b8800 nid=0x31c8 in 
Object.wait() [0x7f4e32fee000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xd23275e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0xd23275e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)

"Reference Handler" daemon prio=10 tid=0x7f4e640b6800 nid=0x31c7 in 
Object.wait() [0x7f4e330ef000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xd23273a0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0xd23273a0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x7f4e64009800 nid=0x31a4 in Object.wait() 
[0x7f4e6b98c000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xf1c81e18> (a 
org.apache.flink.api.common.io.BinaryInputFormatTest)
at java.lang.Object.wait(Object.java:502)
at 
org.apache.flink.api.common.io.BinaryInputFormatTest.testCreateInputSplitsWithOneFile(BinaryInputFormatTest.java:51)
- locked <0xf1c81e18> (a 
org.apache.flink.api.common.io.BinaryInputFormatTest)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
   

[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/388

[FLINK-1533] [runtime] Fixes NPE in the scheduler

Adds checks to the method ```Scheduler.getFreeSubSlotForTask``` to handle 
failed shared slot allocations. Before the slot was given to the 
```SlotSharingGroupAssignment``` without checking that it is not null. 
Consequently, a NullPointerException could occur in the 
```SlotSharingGroupAssignment```.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixScheduler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/388.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #388


commit 604728df38c860ab111d0a7bf2d19c71a0d30c67
Author: Till Rohrmann 
Date:   2015-02-12T13:51:45Z

[FLINK-1533] [runtime] Fixes NPE in the scheduler where the allocated 
shared slots are not properly checked.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318207#comment-14318207
 ] 

ASF GitHub Bot commented on FLINK-1533:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/388

[FLINK-1533] [runtime] Fixes NPE in the scheduler

Adds checks to the method ```Scheduler.getFreeSubSlotForTask``` to handle 
failed shared slot allocations. Before the slot was given to the 
```SlotSharingGroupAssignment``` without checking that it is not null. 
Consequently, a NullPointerException could occur in the 
```SlotSharingGroupAssignment```.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixScheduler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/388.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #388


commit 604728df38c860ab111d0a7bf2d19c71a0d30c67
Author: Till Rohrmann 
Date:   2015-02-12T13:51:45Z

[FLINK-1533] [runtime] Fixes NPE in the scheduler where the allocated 
shared slots are not properly checked.




> NullPointerException in 
> SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
> --
>
> Key: FLINK-1533
> URL: https://issues.apache.org/jira/browse/FLINK-1533
> Project: Flink
>  Issue Type: Bug
>Reporter: Till Rohrmann
>
> The scheduler can fail with a NullPointerException if it is not possible to 
> allocate a {{SharedSlot}} from an {{Instance}}. In the method 
> {{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not 
> checked that it is not null. This can cause a program to fail if there is a 
> race condition in scheduling different tasks on the same instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-1533:


 Summary: NullPointerException in 
SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
 Key: FLINK-1533
 URL: https://issues.apache.org/jira/browse/FLINK-1533
 Project: Flink
  Issue Type: Bug
Reporter: Till Rohrmann


The scheduler can fail with a NullPointerException if it is not possible to 
allocate a {{SharedSlot}} from an {{Instance}}. In the method 
{{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not checked 
that it is not null. This can cause a program to fail if there is a race 
condition in scheduling different tasks on the same instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1436) Command-line interface verbose option & error reporting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318196#comment-14318196
 ] 

ASF GitHub Bot commented on FLINK-1436:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74072268
  
Remember to include "This closes #311" ;)


> Command-line interface verbose option & error reporting
> ---
>
> Key: FLINK-1436
> URL: https://issues.apache.org/jira/browse/FLINK-1436
> Project: Flink
>  Issue Type: Improvement
>  Components: Start-Stop Scripts
>Reporter: Max Michels
>Assignee: Max Michels
>Priority: Trivial
>  Labels: starter, usability
>
> Let me run just a basic Flink job and add the verbose flag. It's a general 
> option, so let me add it as a first parameter:
> > ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> Invalid action!
> ./flink  [GENERAL_OPTIONS] [ARGUMENTS]
>   general options:
>  -h,--help  Show the help for the CLI Frontend.
>  -v,--verbose   Print more detailed error messages.
> Action "run" compiles and runs a program.
>   Syntax: run [OPTIONS]  
>   "run" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "info" displays information about a program.
>   "info" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -e,--executionplan   Show optimized execution plan of the
>   program (JSON)
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "list" lists running and finished programs.
>   "list" action arguments:
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
>  -r,--running  Show running programs and their JobIDs
>  -s,--scheduledShow scheduled prorgrams and their JobIDs
> Action "cancel" cancels a running program.
>   "cancel" action arguments:
>  -i,--jobid JobID of program to cancel
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
> What just happened? This results in a lot of output which is usually 
> generated if you use the --help option on command-line tools. If your 
> terminal window is large enough, then you will see a tiny message:
> "Please specify an action". I did specify an action. Strange. If you read the 
> help messages carefully you see, that "general options" belong to the action.
> > ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> For the sake of mitigating user frustration, let us also accept -v as the 
> first argument. It may seem trivial for the day-to-day Flink user but makes a 
> difference

[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74072268
  
Remember to include "This closes #311" ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1436) Command-line interface verbose option & error reporting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318195#comment-14318195
 ] 

ASF GitHub Bot commented on FLINK-1436:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74072237
  
Good work. Thank you.

I think its time for your first push to master ;)


> Command-line interface verbose option & error reporting
> ---
>
> Key: FLINK-1436
> URL: https://issues.apache.org/jira/browse/FLINK-1436
> Project: Flink
>  Issue Type: Improvement
>  Components: Start-Stop Scripts
>Reporter: Max Michels
>Assignee: Max Michels
>Priority: Trivial
>  Labels: starter, usability
>
> Let me run just a basic Flink job and add the verbose flag. It's a general 
> option, so let me add it as a first parameter:
> > ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> Invalid action!
> ./flink  [GENERAL_OPTIONS] [ARGUMENTS]
>   general options:
>  -h,--help  Show the help for the CLI Frontend.
>  -v,--verbose   Print more detailed error messages.
> Action "run" compiles and runs a program.
>   Syntax: run [OPTIONS]  
>   "run" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "info" displays information about a program.
>   "info" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -e,--executionplan   Show optimized execution plan of the
>   program (JSON)
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "list" lists running and finished programs.
>   "list" action arguments:
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
>  -r,--running  Show running programs and their JobIDs
>  -s,--scheduledShow scheduled prorgrams and their JobIDs
> Action "cancel" cancels a running program.
>   "cancel" action arguments:
>  -i,--jobid JobID of program to cancel
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
> What just happened? This results in a lot of output which is usually 
> generated if you use the --help option on command-line tools. If your 
> terminal window is large enough, then you will see a tiny message:
> "Please specify an action". I did specify an action. Strange. If you read the 
> help messages carefully you see, that "general options" belong to the action.
> > ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> For the sake of mitigating user frustration, let us also accept -v as the 
> first argument. It may seem trivial for the day-to-d

[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74072237
  
Good work. Thank you.

I think its time for your first push to master ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1510) Make AvroInputFormat splittable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318175#comment-14318175
 ] 

ASF GitHub Bot commented on FLINK-1510:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/387


> Make AvroInputFormat splittable
> ---
>
> Key: FLINK-1510
> URL: https://issues.apache.org/jira/browse/FLINK-1510
> Project: Flink
>  Issue Type: Improvement
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> Avro supports splitting files using synchronization points. IFs have to read 
> between these points.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1510] Make AvroInputFormat splittable

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/387


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318169#comment-14318169
 ] 

ASF GitHub Bot commented on FLINK-1391:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74068908
  
Ah no. Its not a good idea ;)
We don't see that the user is using an Avro POJO when we use the POJO 
serializer.
So we have to register the types always.


> Kryo fails to properly serialize avro collection types
> --
>
> Key: FLINK-1391
> URL: https://issues.apache.org/jira/browse/FLINK-1391
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.8.1
>
>
> Before FLINK-610, Avro was the default generic serializer.
> Now, special types coming from Avro are handled by Kryo .. which seems to 
> cause errors like:
> {code}
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.NullPointerException
>   at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
>   at 
> org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
>   at 
> org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
>   at 
> org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
>   at 
> org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
>   at 
> org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
>   at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
>   at 
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>   at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74068908
  
Ah no. Its not a good idea ;)
We don't see that the user is using an Avro POJO when we use the POJO 
serializer.
So we have to register the types always.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1179] Add button to JobManager web inte...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/374#issuecomment-74068773
  
Looks good. There is a small conflict with #384 , but we can try and fix 
this while merging.

+1 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318163#comment-14318163
 ] 

ASF GitHub Bot commented on FLINK-1391:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74068511
  
Yep.


> Kryo fails to properly serialize avro collection types
> --
>
> Key: FLINK-1391
> URL: https://issues.apache.org/jira/browse/FLINK-1391
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.8.1
>
>
> Before FLINK-610, Avro was the default generic serializer.
> Now, special types coming from Avro are handled by Kryo .. which seems to 
> cause errors like:
> {code}
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.NullPointerException
>   at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
>   at 
> org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
>   at 
> org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
>   at 
> org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
>   at 
> org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
>   at 
> org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
>   at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
>   at 
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>   at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1179) Add button to JobManager web interface to request stack trace of a TaskManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318167#comment-14318167
 ] 

ASF GitHub Bot commented on FLINK-1179:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/374#issuecomment-74068773
  
Looks good. There is a small conflict with #384 , but we can try and fix 
this while merging.

+1 


> Add button to JobManager web interface to request stack trace of a TaskManager
> --
>
> Key: FLINK-1179
> URL: https://issues.apache.org/jira/browse/FLINK-1179
> Project: Flink
>  Issue Type: New Feature
>  Components: JobManager
>Reporter: Robert Metzger
>Assignee: Chiwan Park
>Priority: Minor
>  Labels: starter
>
> This is something I do quite often manually and I think it might be helpful 
> for users as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1436) Command-line interface verbose option & error reporting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318164#comment-14318164
 ] 

ASF GitHub Bot commented on FLINK-1436:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74068524
  
Looks like a good improvement to me. 

+1 to merge


> Command-line interface verbose option & error reporting
> ---
>
> Key: FLINK-1436
> URL: https://issues.apache.org/jira/browse/FLINK-1436
> Project: Flink
>  Issue Type: Improvement
>  Components: Start-Stop Scripts
>Reporter: Max Michels
>Assignee: Max Michels
>Priority: Trivial
>  Labels: starter, usability
>
> Let me run just a basic Flink job and add the verbose flag. It's a general 
> option, so let me add it as a first parameter:
> > ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> Invalid action!
> ./flink  [GENERAL_OPTIONS] [ARGUMENTS]
>   general options:
>  -h,--help  Show the help for the CLI Frontend.
>  -v,--verbose   Print more detailed error messages.
> Action "run" compiles and runs a program.
>   Syntax: run [OPTIONS]  
>   "run" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "info" displays information about a program.
>   "info" action arguments:
>  -c,--classClass with the program entry point 
> ("main"
>   method or "getPlan()" method. Only 
> needed
>   if the JAR file does not specify the 
> class
>   in its manifest.
>  -e,--executionplan   Show optimized execution plan of the
>   program (JSON)
>  -m,--jobmanager   Address of the JobManager (master) to
>   which to connect. Use this flag to 
> connect
>   to a different JobManager than the one
>   specified in the configuration.
>  -p,--parallelismThe parallelism with which to run the
>   program. Optional flag to override the
>   default value specified in the
>   configuration.
> Action "list" lists running and finished programs.
>   "list" action arguments:
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
>  -r,--running  Show running programs and their JobIDs
>  -s,--scheduledShow scheduled prorgrams and their JobIDs
> Action "cancel" cancels a running program.
>   "cancel" action arguments:
>  -i,--jobid JobID of program to cancel
>  -m,--jobmanagerAddress of the JobManager (master) to which
>to connect. Use this flag to connect to a
>different JobManager than the one specified
>in the configuration.
> What just happened? This results in a lot of output which is usually 
> generated if you use the --help option on command-line tools. If your 
> terminal window is large enough, then you will see a tiny message:
> "Please specify an action". I did specify an action. Strange. If you read the 
> help messages carefully you see, that "general options" belong to the action.
> > ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar 
> > hdfs:///input hdfs:///output9
> For the sake of mitigating user frustration, let us also accept -v as the 
> first argument. It may seem trivial for the day-to-day Flink user but m

[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/331#issuecomment-74068524
  
Looks like a good improvement to me. 

+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318162#comment-14318162
 ] 

ASF GitHub Bot commented on FLINK-1391:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74068411
  
I like the idea. Can we make the registration conditional on whether the 
enclosing type is an Avro Type?


> Kryo fails to properly serialize avro collection types
> --
>
> Key: FLINK-1391
> URL: https://issues.apache.org/jira/browse/FLINK-1391
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.8.1
>
>
> Before FLINK-610, Avro was the default generic serializer.
> Now, special types coming from Avro are handled by Kryo .. which seems to 
> cause errors like:
> {code}
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.NullPointerException
>   at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
>   at 
> org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
>   at 
> org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
>   at 
> org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
>   at 
> org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
>   at 
> org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
>   at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
>   at 
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>   at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1510) Make AvroInputFormat splittable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318161#comment-14318161
 ] 

ASF GitHub Bot commented on FLINK-1510:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/387#issuecomment-74068314
  
Looks good to me.

+1 to add


> Make AvroInputFormat splittable
> ---
>
> Key: FLINK-1510
> URL: https://issues.apache.org/jira/browse/FLINK-1510
> Project: Flink
>  Issue Type: Improvement
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> Avro supports splitting files using synchronization points. IFs have to read 
> between these points.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74068411
  
I like the idea. Can we make the registration conditional on whether the 
enclosing type is an Avro Type?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/386#issuecomment-74068511
  
Yep.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-1510] Make AvroInputFormat splittable

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/387#issuecomment-74068314
  
Looks good to me.

+1 to add


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1510) Make AvroInputFormat splittable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318152#comment-14318152
 ] 

ASF GitHub Bot commented on FLINK-1510:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/387

[FLINK-1510] Make AvroInputFormat splittable



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink1510

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #387


commit 534de608f86cc4ec7176601ede8637b0fc00d22f
Author: Robert Metzger 
Date:   2015-02-11T08:51:12Z

[FLINK-1510] Make AvroInputFormat splittable




> Make AvroInputFormat splittable
> ---
>
> Key: FLINK-1510
> URL: https://issues.apache.org/jira/browse/FLINK-1510
> Project: Flink
>  Issue Type: Improvement
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> Avro supports splitting files using synchronization points. IFs have to read 
> between these points.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1510] Make AvroInputFormat splittable

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/387

[FLINK-1510] Make AvroInputFormat splittable



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink1510

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #387


commit 534de608f86cc4ec7176601ede8637b0fc00d22f
Author: Robert Metzger 
Date:   2015-02-11T08:51:12Z

[FLINK-1510] Make AvroInputFormat splittable




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1525) Provide utils to pass -D parameters to UDFs

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318146#comment-14318146
 ] 

Stephan Ewen commented on FLINK-1525:
-

I agree, nice starter task...

> Provide utils to pass -D parameters to UDFs 
> 
>
> Key: FLINK-1525
> URL: https://issues.apache.org/jira/browse/FLINK-1525
> Project: Flink
>  Issue Type: Improvement
>  Components: flink-contrib
>Reporter: Robert Metzger
>  Labels: starter
>
> Hadoop users are used to setting job configuration through "-D" on the 
> command line.
> Right now, Flink users have to manually parse command line arguments and pass 
> them to the methods.
> It would be nice to provide a standard args parser with is taking care of 
> such stuff.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-1532) AggregatorsITCase fails spuriously

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-1532:


 Summary: AggregatorsITCase fails spuriously
 Key: FLINK-1532
 URL: https://issues.apache.org/jira/browse/FLINK-1532
 Project: Flink
  Issue Type: Bug
Reporter: Till Rohrmann


Locally the AggregatorITCase failed with the following reason. I cannot 
reproduce the failure constantly.

{code}
Failed tests:
  
AggregatorsITCase.after:75->TestBaseUtils.compareResultsByLinesInMemory:223->TestBaseUtils.compareResultsByLinesInMemory:237
 Different number of lines in expected and obtained result. expected:<15> but 
was:<14>
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318143#comment-14318143
 ] 

Chesnay Schepler commented on FLINK-1521:
-

specifically, the following drivers are affected:
ChainedCollectorMapDriver
ChainedFlatMapDriver
ChainedMapDriver

> Some Chained Drivers do not respect object-reuse/non-reuse flag
> ---
>
> Key: FLINK-1521
> URL: https://issues.apache.org/jira/browse/FLINK-1521
> Project: Flink
>  Issue Type: Bug
>Reporter: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318022#comment-14318022
 ] 

ASF GitHub Bot commented on FLINK-1391:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/386

[FLINK-1391] Register common Avro types at Kryo



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink kryo081-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/386.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #386


commit 5ef83e310c90286b85a5c4f6715c193a56899012
Author: Robert Metzger 
Date:   2015-02-12T11:32:27Z

[FLINK-1391] Register common Avro types at Kryo




> Kryo fails to properly serialize avro collection types
> --
>
> Key: FLINK-1391
> URL: https://issues.apache.org/jira/browse/FLINK-1391
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 0.8, 0.9
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 0.8.1
>
>
> Before FLINK-610, Avro was the default generic serializer.
> Now, special types coming from Avro are handled by Kryo .. which seems to 
> cause errors like:
> {code}
> Exception in thread "main" 
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.NullPointerException
>   at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
>   at 
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
>   at 
> org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
>   at 
> org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
>   at 
> org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
>   at 
> org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
>   at 
> org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
>   at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
>   at 
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>   at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/386

[FLINK-1391] Register common Avro types at Kryo



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink kryo081-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/386.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #386


commit 5ef83e310c90286b85a5c4f6715c193a56899012
Author: Robert Metzger 
Date:   2015-02-12T11:32:27Z

[FLINK-1391] Register common Avro types at Kryo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-1525) Provide utils to pass -D parameters to UDFs

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318016#comment-14318016
 ] 

Robert Metzger commented on FLINK-1525:
---

I think it doesn't hurt to have something like this in the {{flink-contrib}} 
package.

The task is nice for somebody who's looking into an easy starter task with 
Flink.
It doesn't have to be the Configuration object we're passing to the UDFs.
The util could also return a serializable object one can pass into the 
functions.


> Provide utils to pass -D parameters to UDFs 
> 
>
> Key: FLINK-1525
> URL: https://issues.apache.org/jira/browse/FLINK-1525
> Project: Flink
>  Issue Type: Improvement
>  Components: flink-contrib
>Reporter: Robert Metzger
>  Labels: starter
>
> Hadoop users are used to setting job configuration through "-D" on the 
> command line.
> Right now, Flink users have to manually parse command line arguments and pass 
> them to the methods.
> It would be nice to provide a standard args parser with is taking care of 
> such stuff.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1525) Provide utils to pass -D parameters to UDFs

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318012#comment-14318012
 ] 

Stephan Ewen commented on FLINK-1525:
-

I think Flink has a much nicer way of passing parameters to functions. This 
seems very much a Hadoop artifact that they build because there was no better 
way. Should we really adapt this?

> Provide utils to pass -D parameters to UDFs 
> 
>
> Key: FLINK-1525
> URL: https://issues.apache.org/jira/browse/FLINK-1525
> Project: Flink
>  Issue Type: Improvement
>  Components: flink-contrib
>Reporter: Robert Metzger
>  Labels: starter
>
> Hadoop users are used to setting job configuration through "-D" on the 
> command line.
> Right now, Flink users have to manually parse command line arguments and pass 
> them to the methods.
> It would be nice to provide a standard args parser with is taking care of 
> such stuff.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-1432) CombineTaskTest.testCancelCombineTaskSorting sometimes fails

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-1432.
-
   Resolution: Fixed
Fix Version/s: 0.9
 Assignee: Stephan Ewen

Fixed via fab1bd9dc9ab43196a3d136f63b77d4b1d58b452

> CombineTaskTest.testCancelCombineTaskSorting sometimes fails
> 
>
> Key: FLINK-1432
> URL: https://issues.apache.org/jira/browse/FLINK-1432
> Project: Flink
>  Issue Type: Bug
>Reporter: Robert Metzger
>Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> We have a bunch of tests which fail only in rare cases on travis.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/47783455/log.txt
> {code}
> Exception in thread "Thread-17" java.lang.AssertionError: Canceling task 
> failed: java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
>   at java.util.ArrayList$Itr.next(ArrayList.java:831)
>   at 
> org.apache.flink.runtime.memorymanager.DefaultMemoryManager.release(DefaultMemoryManager.java:290)
>   at 
> org.apache.flink.runtime.operators.GroupReduceCombineDriver.cancel(GroupReduceCombineDriver.java:221)
>   at 
> org.apache.flink.runtime.operators.testutils.DriverTestBase.cancel(DriverTestBase.java:272)
>   at 
> org.apache.flink.runtime.operators.testutils.TaskCancelThread.run(TaskCancelThread.java:60)
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.flink.runtime.operators.testutils.TaskCancelThread.run(TaskCancelThread.java:68)
> java.lang.NullPointerException
>   at 
> org.apache.flink.runtime.memorymanager.DefaultMemoryManager.release(DefaultMemoryManager.java:291)
>   at 
> org.apache.flink.runtime.operators.GroupReduceCombineDriver.cleanup(GroupReduceCombineDriver.java:213)
>   at 
> org.apache.flink.runtime.operators.testutils.DriverTestBase.testDriverInternal(DriverTestBase.java:245)
>   at 
> org.apache.flink.runtime.operators.testutils.DriverTestBase.testDriver(DriverTestBase.java:175)
>   at 
> org.apache.flink.runtime.operators.CombineTaskTest$1.run(CombineTaskTest.java:143)
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.172 sec <<< 
> FAILURE! - in org.apache.flink.runtime.operators.CombineTaskTest
> testCancelCombineTaskSorting[0](org.apache.flink.runtime.operators.CombineTaskTest)
>   Time elapsed: 1.023 sec  <<< FAILURE!
> java.lang.AssertionError: Exception was thrown despite proper canceling.
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.flink.runtime.operators.CombineTaskTest.testCancelCombineTaskSorting(CombineTaskTest.java:162)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Johannes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes updated FLINK-1531:

Assignee: (was: Johannes)

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Johannes (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318002#comment-14318002
 ] 

Johannes commented on FLINK-1531:
-

It seems that in the attached test scenario there is a MutableObjectIterator 
which is iterated and null is used to signal "no more".
Because kryo is in the mix - it eagerly tries to read "next" which fails with 
_buffer underflow_
So somewhere there should be a _hasNext_ call ..

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Assignee: Johannes
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario

[Johannes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes updated FLINK-1531:

Attachment: TestKryoIterationSerializer.java

JUnit test to reproduce

> Custom Kryo Serializer fails in itertation scenario
> ---
>
> Key: FLINK-1531
> URL: https://issues.apache.org/jira/browse/FLINK-1531
> Project: Flink
>  Issue Type: Bug
>  Components: Iterations
>Affects Versions: 0.9
>Reporter: Johannes
>Assignee: Johannes
> Attachments: TestKryoIterationSerializer.java
>
>
> When using iterations with a custom serializer for a domain object, the 
> iteration will fail.
> {code:java}
> org.apache.flink.runtime.client.JobExecutionException: 
> com.esotericsoftware.kryo.KryoException: Buffer underflow
>   at 
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
>   at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
>   at 
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
>   at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
>   at 
> org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
>   at 
> org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
>   at 
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
>   at 
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>   at 
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)