[jira] [Assigned] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag
[ https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-1521: --- Assignee: Chesnay Schepler Some Chained Drivers do not respect object-reuse/non-reuse flag --- Key: FLINK-1521 URL: https://issues.apache.org/jira/browse/FLINK-1521 Project: Flink Issue Type: Bug Reporter: Aljoscha Krettek Assignee: Chesnay Schepler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness
[
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318657#comment-14318657
]
ASF GitHub Bot commented on FLINK-1529:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/385#discussion_r24602575
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -653,11 +693,87 @@ object JobManager {
(archiveCount, profilingEnabled, cleanupInterval, executionRetries,
delayBetweenRetries)
}
- def startActor(configuration: Configuration)(implicit actorSystem:
ActorSystem): ActorRef = {
-startActor(Props(classOf[JobManager], configuration))
+ /**
+ * Create the job manager members as (instanceManager, scheduler,
libraryCacheManager,
+ * archiverProps, accumulatorManager, profiler,
defaultExecutionRetries,
+ * delayBetweenRetries, timeout)
+ *
+ * @param configuration The configuration from which to parse the config
values.
+ * @return The members for a default JobManager.
+ */
+ def createJobManagerComponents(configuration: Configuration) :
+(InstanceManager, FlinkScheduler, BlobLibraryCacheManager,
+ Props, AccumulatorManager, Option[Props], Int, Long, FiniteDuration,
Int) = {
+
+val timeout: FiniteDuration = AkkaUtils.getTimeout(configuration)
+
+val (archiveCount, profilingEnabled, cleanupInterval,
executionRetries, delayBetweenRetries) =
+ parseConfiguration(configuration)
+
+val archiveProps: Props = Props(classOf[MemoryArchivist], archiveCount)
+
+val profilerProps: Option[Props] = if (profilingEnabled) {
+ Some(Props(classOf[JobManagerProfiler]))
+} else {
+ None
+}
+
+val accumulatorManager: AccumulatorManager = new
AccumulatorManager(Math.min(1, archiveCount))
+
+var blobServer: BlobServer = null
+var instanceManager: InstanceManager = null
+var scheduler: FlinkScheduler = null
+var libraryCacheManager: BlobLibraryCacheManager = null
+
+try {
+ blobServer = new BlobServer(configuration)
+ instanceManager = new InstanceManager()
+ scheduler = new FlinkScheduler()
+ libraryCacheManager = new BlobLibraryCacheManager(blobServer,
cleanupInterval)
+
+ instanceManager.addInstanceListener(scheduler)
+}
+catch {
+ case t: Throwable = {
+if (libraryCacheManager != null) {
+ libraryCacheManager.shutdown()
+}
+if (scheduler != null) {
+ scheduler.shutdown()
+}
+if (instanceManager != null) {
+ instanceManager.shutdown()
+}
+if (blobServer != null) {
+ blobServer.shutdown()
+}
+throw t;
+ }
+}
+
+(instanceManager, scheduler, libraryCacheManager, archiveProps,
accumulatorManager,
+ profilerProps, executionRetries, delayBetweenRetries, timeout,
archiveCount)
+ }
+
+ def startActor(configuration: Configuration, actorSystem: ActorSystem):
ActorRef = {
+
+val (instanceManager, scheduler, libraryCacheManager, archiveProps,
accumulatorManager,
+ profilerProps, executionRetries, delayBetweenRetries,
+ timeout, _) = createJobManagerComponents(configuration)
+
+val profiler: Option[ActorRef] =
+ profilerProps.map( props = actorSystem.actorOf(props,
PROFILER_NAME) )
--- End diff --
Nice functional style :-)
Improve JobManager startup robustness
-
Key: FLINK-1529
URL: https://issues.apache.org/jira/browse/FLINK-1529
Project: Flink
Issue Type: Improvement
Components: JobManager
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, the JobManager is creates asynchronously (as an actor). If its
initialization fails (for various reasons), the process does not terminate
and gives only vague log message that an actor creation failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1461][api-extending] Add SortPartition ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/381#issuecomment-74120225 Looks good to me (I'm uncertain regarding the optimizer changes) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1461) Add sortPartition operator
[
https://issues.apache.org/jira/browse/FLINK-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318656#comment-14318656
]
ASF GitHub Bot commented on FLINK-1461:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/381#issuecomment-74120225
Looks good to me (I'm uncertain regarding the optimizer changes)
Add sortPartition operator
--
Key: FLINK-1461
URL: https://issues.apache.org/jira/browse/FLINK-1461
Project: Flink
Issue Type: New Feature
Components: Java API, Local Runtime, Optimizer, Scala API
Reporter: Fabian Hueske
Assignee: Fabian Hueske
Priority: Minor
A {{sortPartition()}} operator can be used to
* sort the input of a {{mapPartition()}} operator
* enforce a certain sorting of the input of a given operator of a program.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/385#discussion_r24602575
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -653,11 +693,87 @@ object JobManager {
(archiveCount, profilingEnabled, cleanupInterval, executionRetries,
delayBetweenRetries)
}
- def startActor(configuration: Configuration)(implicit actorSystem:
ActorSystem): ActorRef = {
-startActor(Props(classOf[JobManager], configuration))
+ /**
+ * Create the job manager members as (instanceManager, scheduler,
libraryCacheManager,
+ * archiverProps, accumulatorManager, profiler,
defaultExecutionRetries,
+ * delayBetweenRetries, timeout)
+ *
+ * @param configuration The configuration from which to parse the config
values.
+ * @return The members for a default JobManager.
+ */
+ def createJobManagerComponents(configuration: Configuration) :
+(InstanceManager, FlinkScheduler, BlobLibraryCacheManager,
+ Props, AccumulatorManager, Option[Props], Int, Long, FiniteDuration,
Int) = {
+
+val timeout: FiniteDuration = AkkaUtils.getTimeout(configuration)
+
+val (archiveCount, profilingEnabled, cleanupInterval,
executionRetries, delayBetweenRetries) =
+ parseConfiguration(configuration)
+
+val archiveProps: Props = Props(classOf[MemoryArchivist], archiveCount)
+
+val profilerProps: Option[Props] = if (profilingEnabled) {
+ Some(Props(classOf[JobManagerProfiler]))
+} else {
+ None
+}
+
+val accumulatorManager: AccumulatorManager = new
AccumulatorManager(Math.min(1, archiveCount))
+
+var blobServer: BlobServer = null
+var instanceManager: InstanceManager = null
+var scheduler: FlinkScheduler = null
+var libraryCacheManager: BlobLibraryCacheManager = null
+
+try {
+ blobServer = new BlobServer(configuration)
+ instanceManager = new InstanceManager()
+ scheduler = new FlinkScheduler()
+ libraryCacheManager = new BlobLibraryCacheManager(blobServer,
cleanupInterval)
+
+ instanceManager.addInstanceListener(scheduler)
+}
+catch {
+ case t: Throwable = {
+if (libraryCacheManager != null) {
+ libraryCacheManager.shutdown()
+}
+if (scheduler != null) {
+ scheduler.shutdown()
+}
+if (instanceManager != null) {
+ instanceManager.shutdown()
+}
+if (blobServer != null) {
+ blobServer.shutdown()
+}
+throw t;
+ }
+}
+
+(instanceManager, scheduler, libraryCacheManager, archiveProps,
accumulatorManager,
+ profilerProps, executionRetries, delayBetweenRetries, timeout,
archiveCount)
+ }
+
+ def startActor(configuration: Configuration, actorSystem: ActorSystem):
ActorRef = {
+
+val (instanceManager, scheduler, libraryCacheManager, archiveProps,
accumulatorManager,
+ profilerProps, executionRetries, delayBetweenRetries,
+ timeout, _) = createJobManagerComponents(configuration)
+
+val profiler: Option[ActorRef] =
+ profilerProps.map( props = actorSystem.actorOf(props,
PROFILER_NAME) )
--- End diff --
Nice functional style :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness
[
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318633#comment-14318633
]
ASF GitHub Bot commented on FLINK-1529:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/385#discussion_r24601755
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -149,7 +134,7 @@ Actor with ActorLogMessages with ActorLogging {
override def receiveWithLogMessages: Receive = {
case RegisterTaskManager(connectionInfo, hardwareInformation,
numberOfSlots) =
- val taskManager = sender
+ val taskManager = sender()
--- End diff --
This does not work with older Akka versions, which we are using with the
Hadoop-2.0.0-alpha profile. I think in older Akka version it is a val.
Improve JobManager startup robustness
-
Key: FLINK-1529
URL: https://issues.apache.org/jira/browse/FLINK-1529
Project: Flink
Issue Type: Improvement
Components: JobManager
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, the JobManager is creates asynchronously (as an actor). If its
initialization fails (for various reasons), the process does not terminate
and gives only vague log message that an actor creation failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/385#discussion_r24601897
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -530,54 +515,109 @@ Actor with ActorLogMessages with ActorLogging {
log.error(t, Could not properly unregister job {} form the
library cache., jobID)
}
}
-
- private def checkJavaVersion(): Unit = {
-if (System.getProperty(java.version).substring(0, 3).toDouble 1.7)
{
- log.warning(Warning: Flink is running with Java 6. +
-Java 6 is not maintained any more by Oracle or the OpenJDK
community. +
-Flink currently supports Java 6, but may not in future releases,
+
- due to the unavailability of bug fixes security patched.)
-}
- }
}
object JobManager {
+
import ExecutionMode._
+
val LOG = LoggerFactory.getLogger(classOf[JobManager])
+
val FAILURE_RETURN_CODE = 1
+
val JOB_MANAGER_NAME = jobmanager
val EVENT_COLLECTOR_NAME = eventcollector
val ARCHIVE_NAME = archive
val PROFILER_NAME = profiler
def main(args: Array[String]): Unit = {
+
+// startup checks and logging
EnvironmentInformation.logEnvironmentInfo(LOG, JobManager)
-val (configuration, executionMode, listeningAddress) = parseArgs(args)
+checkJavaVersion()
- if(SecurityUtils.isSecurityEnabled) {
+val (configuration: Configuration,
+ executionMode: ExecutionMode,
+ listeningAddress: Option[(String, Int)]) =
+try {
+ parseArgs(args)
+}
+catch {
+ case t: Throwable = {
+LOG.error(t.getMessage(), t)
+System.exit(FAILURE_RETURN_CODE)
+null
--- End diff --
Do we need the ```null``` expression after the ```System.exit```?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness
[
https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318635#comment-14318635
]
ASF GitHub Bot commented on FLINK-1529:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/385#discussion_r24601897
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -530,54 +515,109 @@ Actor with ActorLogMessages with ActorLogging {
log.error(t, Could not properly unregister job {} form the
library cache., jobID)
}
}
-
- private def checkJavaVersion(): Unit = {
-if (System.getProperty(java.version).substring(0, 3).toDouble 1.7)
{
- log.warning(Warning: Flink is running with Java 6. +
-Java 6 is not maintained any more by Oracle or the OpenJDK
community. +
-Flink currently supports Java 6, but may not in future releases,
+
- due to the unavailability of bug fixes security patched.)
-}
- }
}
object JobManager {
+
import ExecutionMode._
+
val LOG = LoggerFactory.getLogger(classOf[JobManager])
+
val FAILURE_RETURN_CODE = 1
+
val JOB_MANAGER_NAME = jobmanager
val EVENT_COLLECTOR_NAME = eventcollector
val ARCHIVE_NAME = archive
val PROFILER_NAME = profiler
def main(args: Array[String]): Unit = {
+
+// startup checks and logging
EnvironmentInformation.logEnvironmentInfo(LOG, JobManager)
-val (configuration, executionMode, listeningAddress) = parseArgs(args)
+checkJavaVersion()
- if(SecurityUtils.isSecurityEnabled) {
+val (configuration: Configuration,
+ executionMode: ExecutionMode,
+ listeningAddress: Option[(String, Int)]) =
+try {
+ parseArgs(args)
+}
+catch {
+ case t: Throwable = {
+LOG.error(t.getMessage(), t)
+System.exit(FAILURE_RETURN_CODE)
+null
--- End diff --
Do we need the ```null``` expression after the ```System.exit```?
Improve JobManager startup robustness
-
Key: FLINK-1529
URL: https://issues.apache.org/jira/browse/FLINK-1529
Project: Flink
Issue Type: Improvement
Components: JobManager
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
Currently, the JobManager is creates asynchronously (as an actor). If its
initialization fails (for various reasons), the process does not terminate
and gives only vague log message that an actor creation failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1529) Improve JobManager startup robustness
[ https://issues.apache.org/jira/browse/FLINK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318681#comment-14318681 ] ASF GitHub Bot commented on FLINK-1529: --- Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/385#issuecomment-74122398 Nice work. Looks really good. There are only some minor issues on Travis with the ```JobManagerFailsITCase``` which fails because the archive actor is not properly shut down before restarting the JobManager. LGTM Improve JobManager startup robustness - Key: FLINK-1529 URL: https://issues.apache.org/jira/browse/FLINK-1529 Project: Flink Issue Type: Improvement Components: JobManager Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 Currently, the JobManager is creates asynchronously (as an actor). If its initialization fails (for various reasons), the process does not terminate and gives only vague log message that an actor creation failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1529] [jobmanager] Improve error handli...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/385#discussion_r24601755
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -149,7 +134,7 @@ Actor with ActorLogMessages with ActorLogging {
override def receiveWithLogMessages: Receive = {
case RegisterTaskManager(connectionInfo, hardwareInformation,
numberOfSlots) =
- val taskManager = sender
+ val taskManager = sender()
--- End diff --
This does not work with older Akka versions, which we are using with the
Hadoop-2.0.0-alpha profile. I think in older Akka version it is a val.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1508] Removes AkkaUtil.ask
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/384#issuecomment-74119549 I vote to merge this quickly and fix issues as they appear. The change touches a lot of different parts of the code and is predestined to become unmergeable quickly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318779#comment-14318779
]
Johannes commented on FLINK-1531:
-
Thanks for the quick turnaround
Tried it locally and it works now - thanks
But as a general though, isn't using exceptions to signal EOF much to costly?
Quick informative write up on stack overflow:
http://stackoverflow.com/questions/299068/how-slow-are-java-exceptions
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Fix For: 0.9, 0.8.1
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Michels resolved FLINK-1436. Resolution: Fixed Fix Version/s: 0.9 Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Fix For: 0.9 Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a difference for a novice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317974#comment-14317974 ] ASF GitHub Bot commented on FLINK-1436: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74049747 Any further comments on the changes? Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a difference for a novice. -- This
[jira] [Commented] (FLINK-1437) Bug in PojoSerializer's copy() method
[
https://issues.apache.org/jira/browse/FLINK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317981#comment-14317981
]
Timo Walther commented on FLINK-1437:
-
Thanks. I change the Fixed Version field.
Bug in PojoSerializer's copy() method
-
Key: FLINK-1437
URL: https://issues.apache.org/jira/browse/FLINK-1437
Project: Flink
Issue Type: Bug
Components: Java API
Reporter: Timo Walther
Assignee: Timo Walther
Fix For: 0.9, 0.8.1
The PojoSerializer's {{copy()}} method does not work properly with {{null}}
values. An exception could look like:
{code}
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: null
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:792)
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.RandomAccessInputView.nextSegment(RandomAccessInputView.java:83)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(AbstractPagedInputView.java:159)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(AbstractPagedInputView.java:270)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readUnsignedByte(AbstractPagedInputView.java:277)
at org.apache.flink.types.StringValue.copyString(StringValue.java:839)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.copy(StringSerializer.java:83)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSerializer.java:261)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1303)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:788)
{code}
I'm working on a fix for that...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318002#comment-14318002
]
Johannes commented on FLINK-1531:
-
It seems that in the attached test scenario there is a MutableObjectIterator
which is iterated and null is used to signal no more.
Because kryo is in the mix - it eagerly tries to read next which fails with
_buffer underflow_
So somewhere there should be a _hasNext_ call ..
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Assignee: Johannes
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes updated FLINK-1531:
Assignee: (was: Johannes)
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-1432) CombineTaskTest.testCancelCombineTaskSorting sometimes fails
[
https://issues.apache.org/jira/browse/FLINK-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1432.
-
Resolution: Fixed
Fix Version/s: 0.9
Assignee: Stephan Ewen
Fixed via fab1bd9dc9ab43196a3d136f63b77d4b1d58b452
CombineTaskTest.testCancelCombineTaskSorting sometimes fails
Key: FLINK-1432
URL: https://issues.apache.org/jira/browse/FLINK-1432
Project: Flink
Issue Type: Bug
Reporter: Robert Metzger
Assignee: Stephan Ewen
Fix For: 0.9
We have a bunch of tests which fail only in rare cases on travis.
https://s3.amazonaws.com/archive.travis-ci.org/jobs/47783455/log.txt
{code}
Exception in thread Thread-17 java.lang.AssertionError: Canceling task
failed: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.flink.runtime.memorymanager.DefaultMemoryManager.release(DefaultMemoryManager.java:290)
at
org.apache.flink.runtime.operators.GroupReduceCombineDriver.cancel(GroupReduceCombineDriver.java:221)
at
org.apache.flink.runtime.operators.testutils.DriverTestBase.cancel(DriverTestBase.java:272)
at
org.apache.flink.runtime.operators.testutils.TaskCancelThread.run(TaskCancelThread.java:60)
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.runtime.operators.testutils.TaskCancelThread.run(TaskCancelThread.java:68)
java.lang.NullPointerException
at
org.apache.flink.runtime.memorymanager.DefaultMemoryManager.release(DefaultMemoryManager.java:291)
at
org.apache.flink.runtime.operators.GroupReduceCombineDriver.cleanup(GroupReduceCombineDriver.java:213)
at
org.apache.flink.runtime.operators.testutils.DriverTestBase.testDriverInternal(DriverTestBase.java:245)
at
org.apache.flink.runtime.operators.testutils.DriverTestBase.testDriver(DriverTestBase.java:175)
at
org.apache.flink.runtime.operators.CombineTaskTest$1.run(CombineTaskTest.java:143)
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.172 sec
FAILURE! - in org.apache.flink.runtime.operators.CombineTaskTest
testCancelCombineTaskSorting[0](org.apache.flink.runtime.operators.CombineTaskTest)
Time elapsed: 1.023 sec FAILURE!
java.lang.AssertionError: Exception was thrown despite proper canceling.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.flink.runtime.operators.CombineTaskTest.testCancelCombineTaskSorting(CombineTaskTest.java:162)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes updated FLINK-1531:
Attachment: TestKryoIterationSerializer.java
JUnit test to reproduce
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Assignee: Johannes
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-1437) Bug in PojoSerializer's copy() method
[
https://issues.apache.org/jira/browse/FLINK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-1437:
Fix Version/s: 0.8.1
Bug in PojoSerializer's copy() method
-
Key: FLINK-1437
URL: https://issues.apache.org/jira/browse/FLINK-1437
Project: Flink
Issue Type: Bug
Components: Java API
Reporter: Timo Walther
Assignee: Timo Walther
Fix For: 0.9, 0.8.1
The PojoSerializer's {{copy()}} method does not work properly with {{null}}
values. An exception could look like:
{code}
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: null
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:792)
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.RandomAccessInputView.nextSegment(RandomAccessInputView.java:83)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(AbstractPagedInputView.java:159)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(AbstractPagedInputView.java:270)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readUnsignedByte(AbstractPagedInputView.java:277)
at org.apache.flink.types.StringValue.copyString(StringValue.java:839)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.copy(StringSerializer.java:83)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSerializer.java:261)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1303)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:788)
{code}
I'm working on a fix for that...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1525) Provide utils to pass -D parameters to UDFs
[ https://issues.apache.org/jira/browse/FLINK-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318012#comment-14318012 ] Stephan Ewen commented on FLINK-1525: - I think Flink has a much nicer way of passing parameters to functions. This seems very much a Hadoop artifact that they build because there was no better way. Should we really adapt this? Provide utils to pass -D parameters to UDFs Key: FLINK-1525 URL: https://issues.apache.org/jira/browse/FLINK-1525 Project: Flink Issue Type: Improvement Components: flink-contrib Reporter: Robert Metzger Labels: starter Hadoop users are used to setting job configuration through -D on the command line. Right now, Flink users have to manually parse command line arguments and pass them to the methods. It would be nice to provide a standard args parser with is taking care of such stuff. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1525) Provide utils to pass -D parameters to UDFs
[
https://issues.apache.org/jira/browse/FLINK-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318016#comment-14318016
]
Robert Metzger commented on FLINK-1525:
---
I think it doesn't hurt to have something like this in the {{flink-contrib}}
package.
The task is nice for somebody who's looking into an easy starter task with
Flink.
It doesn't have to be the Configuration object we're passing to the UDFs.
The util could also return a serializable object one can pass into the
functions.
Provide utils to pass -D parameters to UDFs
Key: FLINK-1525
URL: https://issues.apache.org/jira/browse/FLINK-1525
Project: Flink
Issue Type: Improvement
Components: flink-contrib
Reporter: Robert Metzger
Labels: starter
Hadoop users are used to setting job configuration through -D on the
command line.
Right now, Flink users have to manually parse command line arguments and pass
them to the methods.
It would be nice to provide a standard args parser with is taking care of
such stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
[ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317935#comment-14317935 ] ASF GitHub Bot commented on FLINK-377: -- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-74046665 Thanks, works fine now. For work stations, you cannot assume that the hostname maps to localhost or any address at all. Create a general purpose framework for language bindings Key: FLINK-377 URL: https://issues.apache.org/jira/browse/FLINK-377 Project: Flink Issue Type: Improvement Reporter: GitHub Import Assignee: Chesnay Schepler Labels: github-import Fix For: pre-apache A general purpose API to run operators with arbitrary binaries. This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go or whatever you like. We suggest using Google Protocol Buffers for data serialization. This is the list of languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing protobuf) For Ruby: https://github.com/infochimps-labs/wukong Two new students working at Stratosphere (@skunert and @filiphaase) are working on this. The reference binding language will be for Python, but other bindings are very welcome. The best name for this so far is stratosphere-lang-bindings. I created this issue to track the progress (and give everybody a chance to comment on this) Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/377 Created by: [rmetzger|https://github.com/rmetzger] Labels: enhancement, Assignee: [filiphaase|https://github.com/filiphaase] Created at: Tue Jan 07 19:47:20 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/386 [FLINK-1391] Register common Avro types at Kryo You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink kryo081-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/386.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #386 commit 5ef83e310c90286b85a5c4f6715c193a56899012 Author: Robert Metzger rmetz...@apache.org Date: 2015-02-12T11:32:27Z [FLINK-1391] Register common Avro types at Kryo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types
[
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318022#comment-14318022
]
ASF GitHub Bot commented on FLINK-1391:
---
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/386
[FLINK-1391] Register common Avro types at Kryo
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink kryo081-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/386.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #386
commit 5ef83e310c90286b85a5c4f6715c193a56899012
Author: Robert Metzger rmetz...@apache.org
Date: 2015-02-12T11:32:27Z
[FLINK-1391] Register common Avro types at Kryo
Kryo fails to properly serialize avro collection types
--
Key: FLINK-1391
URL: https://issues.apache.org/jira/browse/FLINK-1391
Project: Flink
Issue Type: Sub-task
Affects Versions: 0.8, 0.9
Reporter: Robert Metzger
Assignee: Robert Metzger
Fix For: 0.8.1
Before FLINK-610, Avro was the default generic serializer.
Now, special types coming from Avro are handled by Kryo .. which seems to
cause errors like:
{code}
Exception in thread main
org.apache.flink.runtime.client.JobExecutionException:
java.lang.NullPointerException
at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
at
org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
at
org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink commit comment: 988602b031b63deb9640b115b798e9ae54fb7357
Github user mbalassi commented on commit 988602b031b63deb9640b115b798e9ae54fb7357: https://github.com/apache/flink/commit/988602b031b63deb9640b115b798e9ae54fb7357#commitcomment-9725249 In tools/create_release_files.sh: In tools/create_release_files.sh on line 150: This way you use the generate pom that is in the outside repo and might be from a different branch. Not a big difference, but might lead to unexpected results. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-785] Chained AllReduce / AllGroupReduce...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/370#discussion_r24633778
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllGroupReduceCombineDriver.java
---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.operators.chaining;
+
+import java.util.ArrayList;
+
+import org.apache.flink.api.common.functions.FlatCombineFunction;
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerFactory;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.operators.RegularPactTask;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ChainedAllGroupReduceCombineDriverT extends
ChainedDriverT, T {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChainedAllGroupReduceCombineDriver.class);
+
+ //
+ private FlatCombineFunctionT combiner;
+ private TypeSerializerT serializer;
+
+ private volatile boolean running = true;
+
+ private final ArrayListT values = new ArrayList();
+
+ //
+ @Override
+ public void setup(AbstractInvokable parent) {
+ @SuppressWarnings(unchecked)
+ final FlatCombineFunctionT com =
RegularPactTask.instantiateUserCode(this.config, userCodeClassLoader,
FlatCombineFunction.class);
+ this.combiner = com;
+ FunctionUtils.setFunctionRuntimeContext(com,
getUdfRuntimeContext());
+
+ this.objectReuseEnabled =
this.executionConfig.isObjectReuseEnabled();
+
+ final TypeSerializerFactoryT serializerFactory =
this.config.getInputSerializer(0, this.userCodeClassLoader);
+ this.serializer = serializerFactory.getSerializer();
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(ChainedAllGroupReduceCombineDriver object
reuse: + (this.objectReuseEnabled ? ENABLED : DISABLED) + .);
+ }
+ }
+
+ @Override
+ public void openTask() throws Exception {
+ final Configuration stubConfig =
this.config.getStubParameters();
+ RegularPactTask.openUserCode(this.combiner, stubConfig);
+ }
+
+ @Override
+ public void closeTask() throws Exception {
+ if (!this.running) {
+ return;
+ }
+ RegularPactTask.closeUserCode(this.combiner);
+ }
+
+ @Override
+ public void cancelTask() {
+ this.running = false;
+ }
+
+ //
+ @Override
+ public Function getStub() {
+ return this.combiner;
+ }
+
+ @Override
+ public String getTaskName() {
+ return this.taskName;
+ }
+
+ //
+ @Override
+ public void collect(T record) {
+ try {
+ values.add(objectReuseEnabled ? record :
serializer.copy(record));
+ if (values.size() 1) {
--- End diff --
is there a more reasonable value for this? (basically anything whose
reasoning goes beyond i felt like using 1)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
[GitHub] flink pull request: [FLINK-1521] Chained operators respect reuse
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/392 [FLINK-1521] Chained operators respect reuse You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/incubator-flink flink-1521 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/392.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #392 commit b3f8ec9a107a2baf743468c9efdf28da9f49cf79 Author: zentol s.mo...@web.de Date: 2015-02-12T19:36:00Z [FLINK-1521] Chained operators respect reuse --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-785) Add Chained operators for AllReduce and AllGroupReduce
[
https://issues.apache.org/jira/browse/FLINK-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319276#comment-14319276
]
ASF GitHub Bot commented on FLINK-785:
--
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/370#discussion_r24633778
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllGroupReduceCombineDriver.java
---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.runtime.operators.chaining;
+
+import java.util.ArrayList;
+
+import org.apache.flink.api.common.functions.FlatCombineFunction;
+import org.apache.flink.api.common.functions.Function;
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerFactory;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable;
+import org.apache.flink.runtime.operators.RegularPactTask;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ChainedAllGroupReduceCombineDriverT extends
ChainedDriverT, T {
+ private static final Logger LOG =
LoggerFactory.getLogger(ChainedAllGroupReduceCombineDriver.class);
+
+ //
+ private FlatCombineFunctionT combiner;
+ private TypeSerializerT serializer;
+
+ private volatile boolean running = true;
+
+ private final ArrayListT values = new ArrayList();
+
+ //
+ @Override
+ public void setup(AbstractInvokable parent) {
+ @SuppressWarnings(unchecked)
+ final FlatCombineFunctionT com =
RegularPactTask.instantiateUserCode(this.config, userCodeClassLoader,
FlatCombineFunction.class);
+ this.combiner = com;
+ FunctionUtils.setFunctionRuntimeContext(com,
getUdfRuntimeContext());
+
+ this.objectReuseEnabled =
this.executionConfig.isObjectReuseEnabled();
+
+ final TypeSerializerFactoryT serializerFactory =
this.config.getInputSerializer(0, this.userCodeClassLoader);
+ this.serializer = serializerFactory.getSerializer();
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(ChainedAllGroupReduceCombineDriver object
reuse: + (this.objectReuseEnabled ? ENABLED : DISABLED) + .);
+ }
+ }
+
+ @Override
+ public void openTask() throws Exception {
+ final Configuration stubConfig =
this.config.getStubParameters();
+ RegularPactTask.openUserCode(this.combiner, stubConfig);
+ }
+
+ @Override
+ public void closeTask() throws Exception {
+ if (!this.running) {
+ return;
+ }
+ RegularPactTask.closeUserCode(this.combiner);
+ }
+
+ @Override
+ public void cancelTask() {
+ this.running = false;
+ }
+
+ //
+ @Override
+ public Function getStub() {
+ return this.combiner;
+ }
+
+ @Override
+ public String getTaskName() {
+ return this.taskName;
+ }
+
+ //
+ @Override
+ public void collect(T record) {
+ try {
+ values.add(objectReuseEnabled ? record :
serializer.copy(record));
+ if (values.size() 1) {
--- End diff --
is there a more reasonable value for this? (basically anything whose
[jira] [Commented] (FLINK-1437) Bug in PojoSerializer's copy() method
[
https://issues.apache.org/jira/browse/FLINK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317801#comment-14317801
]
Stephan Ewen commented on FLINK-1437:
-
I think the fix is in the master, I will try to pick it onto the
{{release-0.8}} branch...
Bug in PojoSerializer's copy() method
-
Key: FLINK-1437
URL: https://issues.apache.org/jira/browse/FLINK-1437
Project: Flink
Issue Type: Bug
Components: Java API
Reporter: Timo Walther
Assignee: Timo Walther
Fix For: 0.9
The PojoSerializer's {{copy()}} method does not work properly with {{null}}
values. An exception could look like:
{code}
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: null
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:792)
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.RandomAccessInputView.nextSegment(RandomAccessInputView.java:83)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(AbstractPagedInputView.java:159)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(AbstractPagedInputView.java:270)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readUnsignedByte(AbstractPagedInputView.java:277)
at org.apache.flink.types.StringValue.copyString(StringValue.java:839)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.copy(StringSerializer.java:83)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSerializer.java:261)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1303)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:788)
{code}
I'm working on a fix for that...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1512) Add CsvReader for reading into POJOs.
[
https://issues.apache.org/jira/browse/FLINK-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319492#comment-14319492
]
Chiwan Park commented on FLINK-1512:
[~StephanEwen] Thanks for your explanation. I'll implement this feature and
send a pull request.
Add CsvReader for reading into POJOs.
-
Key: FLINK-1512
URL: https://issues.apache.org/jira/browse/FLINK-1512
Project: Flink
Issue Type: New Feature
Components: Java API, Scala API
Reporter: Robert Metzger
Assignee: Chiwan Park
Priority: Minor
Labels: starter
Currently, the {{CsvReader}} supports only TupleXX types.
It would be nice if users were also able to read into POJOs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1521) Some Chained Drivers do not respect object-reuse/non-reuse flag
[ https://issues.apache.org/jira/browse/FLINK-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318143#comment-14318143 ] Chesnay Schepler commented on FLINK-1521: - specifically, the following drivers are affected: ChainedCollectorMapDriver ChainedFlatMapDriver ChainedMapDriver Some Chained Drivers do not respect object-reuse/non-reuse flag --- Key: FLINK-1521 URL: https://issues.apache.org/jira/browse/FLINK-1521 Project: Flink Issue Type: Bug Reporter: Aljoscha Krettek -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1525) Provide utils to pass -D parameters to UDFs
[ https://issues.apache.org/jira/browse/FLINK-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318146#comment-14318146 ] Stephan Ewen commented on FLINK-1525: - I agree, nice starter task... Provide utils to pass -D parameters to UDFs Key: FLINK-1525 URL: https://issues.apache.org/jira/browse/FLINK-1525 Project: Flink Issue Type: Improvement Components: flink-contrib Reporter: Robert Metzger Labels: starter Hadoop users are used to setting job configuration through -D on the command line. Right now, Flink users have to manually parse command line arguments and pass them to the methods. It would be nice to provide a standard args parser with is taking care of such stuff. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-1534) GSoC project: Distributed pattern matching over Flink streaming
[ https://issues.apache.org/jira/browse/FLINK-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi updated FLINK-1534: -- Description: Pattern matching over streams is an important application. The general structure of a streaming pattern matching is the following: If A event follows B event then trigger some computation. The support for this feature is associated with complex event processing systems, however it is also adoptable for distributed setting, however it poses additional challenges. The Google Summer of Code student volunteering for this project is expected to have general knowledge of distributed systems and Java/Scala coding skills. The project includes research and implementation oriented taks. was: Pattern matching over streams is an important application. The general structure of a streaming pattern matching is the following: If A event follows B event then trigger some computation. The support for this feature is associated with complex event processing systems, however it is also adoptable for distributed setting, however it poses additional challenges. GSoC project: Distributed pattern matching over Flink streaming --- Key: FLINK-1534 URL: https://issues.apache.org/jira/browse/FLINK-1534 Project: Flink Issue Type: New Feature Components: Streaming Reporter: Márton Balassi Priority: Minor Labels: gsoc2015, java, scala Pattern matching over streams is an important application. The general structure of a streaming pattern matching is the following: If A event follows B event then trigger some computation. The support for this feature is associated with complex event processing systems, however it is also adoptable for distributed setting, however it poses additional challenges. The Google Summer of Code student volunteering for this project is expected to have general knowledge of distributed systems and Java/Scala coding skills. The project includes research and implementation oriented taks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [builds] Notify dev@f.a.o about travis-ci buil...
GitHub user uce opened a pull request: https://github.com/apache/flink/pull/390 [builds] Notify dev@f.a.o about travis-ci build status changes Related discussion [about how to notice failing master faster](http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Master-not-building-and-how-to-notice-it-faster-in-the-future-td3375.html). With this Travis config, we will get an email to dev@f.a.o for every status change from success = failure and failure = success. We could also just notify on failures. Feedback is welcome. If we want to merge this, we have to make sure that bui...@travis-ci.org is allowed to post to our dev mailing list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/incubator-flink travis_notify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/390.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #390 commit 2e56503c581078935847036ab8627e4815a84189 Author: Ufuk Celebi u...@apache.org Date: 2015-02-12T14:28:54Z [builds] Notify dev@f.a.o about travis-ci build status changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
[
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318281#comment-14318281
]
ASF GitHub Bot commented on FLINK-1533:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/388#issuecomment-74080518
Good catch.
+1 to merge
NullPointerException in
SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
--
Key: FLINK-1533
URL: https://issues.apache.org/jira/browse/FLINK-1533
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
The scheduler can fail with a NullPointerException if it is not possible to
allocate a {{SharedSlot}} from an {{Instance}}. In the method
{{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not
checked that it is not null. This can cause a program to fail if there is a
race condition in scheduling different tasks on the same instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/386#issuecomment-74080897 Merged to release-0.8 in: http://git-wip-us.apache.org/repos/asf/flink/commit/10f89c73 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [builds] Notify dev@f.a.o about travis-ci buil...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/390#issuecomment-74080451 From Travis docs: ``` I'm not receiving any build notifications # The most common cause for not receiving build notifications, beyond not having a user account on Travis CI, is the use of an email address that's not registered and verified on GitHub. See above on how to change the email address to one that's registered or make sure to add the email address used in this repository to your verified email addresses on GitHub. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [builds] Notify dev@f.a.o about travis-ci buil...
Github user uce closed the pull request at: https://github.com/apache/flink/pull/390 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-1510) Make AvroInputFormat splittable
[ https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger resolved FLINK-1510. --- Resolution: Fixed Fix Version/s: 0.8.1 0.9 Resolved for 0.9 in http://git-wip-us.apache.org/repos/asf/flink/commit/3241214c Resolved for 0.8.1 in http://git-wip-us.apache.org/repos/asf/flink/commit/e7c99962 Make AvroInputFormat splittable --- Key: FLINK-1510 URL: https://issues.apache.org/jira/browse/FLINK-1510 Project: Flink Issue Type: Improvement Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.9, 0.8.1 Avro supports splitting files using synchronization points. IFs have to read between these points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [builds] Print Java process stack traces of st...
GitHub user uce opened a pull request: https://github.com/apache/flink/pull/389 [builds] Print Java process stack traces of stalled builds After discussion with @rmetzger we came up with the following idea to manually monitor stalled builds and to print the Java process stack traces if they do stall. Furthermore, a stalling build is killed with a != 0 exit code and really fails the Travis build instead of just timing it out. For testing, I've a branch where I block a test on purpose. The output can be found here: https://travis-ci.org/uce/incubator-flink/jobs/50491750 ``` == Maven produced no output for 300 seconds. == == The following Java processes are running (JPS) == 21670 Jps 12703 surefirebooter5726841031685552603.jar 2220 Launcher == Printing stack trace of Java process 12703 == 2015-02-12 14:14:42 Full thread dump OpenJDK 64-Bit Server VM (23.25-b01 mixed mode): Attach Listener daemon prio=10 tid=0x7f4e18001000 nid=0x54ed runnable [0x] java.lang.Thread.State: RUNNABLE Low Memory Detector daemon prio=10 tid=0x7f4e64112000 nid=0x31d2 runnable [0x] java.lang.Thread.State: RUNNABLE C2 CompilerThread1 daemon prio=10 tid=0x7f4e6411 nid=0x31ce waiting on condition [0x] java.lang.Thread.State: RUNNABLE C2 CompilerThread0 daemon prio=10 tid=0x7f4e6410d000 nid=0x31cd waiting on condition [0x] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x7f4e6410b000 nid=0x31cb runnable [0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=10 tid=0x7f4e640b8800 nid=0x31c8 in Object.wait() [0x7f4e32fee000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xd23275e0 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) - locked 0xd23275e0 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189) Reference Handler daemon prio=10 tid=0x7f4e640b6800 nid=0x31c7 in Object.wait() [0x7f4e330ef000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xd23273a0 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked 0xd23273a0 (a java.lang.ref.Reference$Lock) main prio=10 tid=0x7f4e64009800 nid=0x31a4 in Object.wait() [0x7f4e6b98c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xf1c81e18 (a org.apache.flink.api.common.io.BinaryInputFormatTest) at java.lang.Object.wait(Object.java:502) at org.apache.flink.api.common.io.BinaryInputFormatTest.testCreateInputSplitsWithOneFile(BinaryInputFormatTest.java:51) - locked 0xf1c81e18 (a org.apache.flink.api.common.io.BinaryInputFormatTest) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at
[jira] [Created] (FLINK-1534) GSoC project: Distributed pattern matching over Flink streaming
Márton Balassi created FLINK-1534: - Summary: GSoC project: Distributed pattern matching over Flink streaming Key: FLINK-1534 URL: https://issues.apache.org/jira/browse/FLINK-1534 Project: Flink Issue Type: New Feature Components: Streaming Reporter: Márton Balassi Priority: Minor Pattern matching over streams is an important application. The general structure of a streaming pattern matching is the following: If A event follows B event then trigger some computation. The support for this feature is associated with complex event processing systems, however it is also adoptable for distributed setting, however it poses additional challenges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [builds] Print Java process stack traces of st...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/389#issuecomment-74079695 Very nicely implemented! My bash scripts are certainly uglier ;) +1 for merging it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/388#issuecomment-74080518 Good catch. +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types
[
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318286#comment-14318286
]
ASF GitHub Bot commented on FLINK-1391:
---
Github user rmetzger closed the pull request at:
https://github.com/apache/flink/pull/386
Kryo fails to properly serialize avro collection types
--
Key: FLINK-1391
URL: https://issues.apache.org/jira/browse/FLINK-1391
Project: Flink
Issue Type: Sub-task
Affects Versions: 0.8, 0.9
Reporter: Robert Metzger
Assignee: Robert Metzger
Fix For: 0.8.1
Before FLINK-610, Avro was the default generic serializer.
Now, special types coming from Avro are handled by Kryo .. which seems to
cause errors like:
{code}
Exception in thread main
org.apache.flink.runtime.client.JobExecutionException:
java.lang.NullPointerException
at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
at
org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
at
org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...
Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/386 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1535) Use usercode class loader to serialize/deserialize accumulators
Stephan Ewen created FLINK-1535: --- Summary: Use usercode class loader to serialize/deserialize accumulators Key: FLINK-1535 URL: https://issues.apache.org/jira/browse/FLINK-1535 Project: Flink Issue Type: Bug Components: JobManager Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 Currently, accumulators are transferred via simple Akka Messages. Since the accumulators may be user defined types, we should use the user code class loader for code loading when deserializing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types
[
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318162#comment-14318162
]
ASF GitHub Bot commented on FLINK-1391:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/386#issuecomment-74068411
I like the idea. Can we make the registration conditional on whether the
enclosing type is an Avro Type?
Kryo fails to properly serialize avro collection types
--
Key: FLINK-1391
URL: https://issues.apache.org/jira/browse/FLINK-1391
Project: Flink
Issue Type: Sub-task
Affects Versions: 0.8, 0.9
Reporter: Robert Metzger
Assignee: Robert Metzger
Fix For: 0.8.1
Before FLINK-610, Avro was the default generic serializer.
Now, special types coming from Avro are handled by Kryo .. which seems to
cause errors like:
{code}
Exception in thread main
org.apache.flink.runtime.client.JobExecutionException:
java.lang.NullPointerException
at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
at
org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
at
org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74068524 Looks like a good improvement to me. +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318164#comment-14318164 ] ASF GitHub Bot commented on FLINK-1436: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74068524 Looks like a good improvement to me. +1 to merge Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a
[jira] [Commented] (FLINK-1510) Make AvroInputFormat splittable
[ https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318175#comment-14318175 ] ASF GitHub Bot commented on FLINK-1510: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/387 Make AvroInputFormat splittable --- Key: FLINK-1510 URL: https://issues.apache.org/jira/browse/FLINK-1510 Project: Flink Issue Type: Improvement Reporter: Robert Metzger Assignee: Robert Metzger Avro supports splitting files using synchronization points. IFs have to read between these points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types
[
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318169#comment-14318169
]
ASF GitHub Bot commented on FLINK-1391:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/386#issuecomment-74068908
Ah no. Its not a good idea ;)
We don't see that the user is using an Avro POJO when we use the POJO
serializer.
So we have to register the types always.
Kryo fails to properly serialize avro collection types
--
Key: FLINK-1391
URL: https://issues.apache.org/jira/browse/FLINK-1391
Project: Flink
Issue Type: Sub-task
Affects Versions: 0.8, 0.9
Reporter: Robert Metzger
Assignee: Robert Metzger
Fix For: 0.8.1
Before FLINK-610, Avro was the default generic serializer.
Now, special types coming from Avro are handled by Kryo .. which seems to
cause errors like:
{code}
Exception in thread main
org.apache.flink.runtime.client.JobExecutionException:
java.lang.NullPointerException
at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
at
org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
at
org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
Till Rohrmann created FLINK-1533:
Summary: NullPointerException in
SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
Key: FLINK-1533
URL: https://issues.apache.org/jira/browse/FLINK-1533
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
The scheduler can fail with a NullPointerException if it is not possible to
allocate a {{SharedSlot}} from an {{Instance}}. In the method
{{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not checked
that it is not null. This can cause a program to fail if there is a race
condition in scheduling different tasks on the same instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/386#issuecomment-74068908 Ah no. Its not a good idea ;) We don't see that the user is using an Avro POJO when we use the POJO serializer. So we have to register the types always. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74072237 Good work. Thank you. I think its time for your first push to master ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1391) Kryo fails to properly serialize avro collection types
[
https://issues.apache.org/jira/browse/FLINK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318163#comment-14318163
]
ASF GitHub Bot commented on FLINK-1391:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/386#issuecomment-74068511
Yep.
Kryo fails to properly serialize avro collection types
--
Key: FLINK-1391
URL: https://issues.apache.org/jira/browse/FLINK-1391
Project: Flink
Issue Type: Sub-task
Affects Versions: 0.8, 0.9
Reporter: Robert Metzger
Assignee: Robert Metzger
Fix For: 0.8.1
Before FLINK-610, Avro was the default generic serializer.
Now, special types coming from Avro are handled by Kryo .. which seems to
cause errors like:
{code}
Exception in thread main
org.apache.flink.runtime.client.JobExecutionException:
java.lang.NullPointerException
at org.apache.avro.generic.GenericData$Array.add(GenericData.java:200)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:116)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:143)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:148)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:244)
at
org.apache.flink.runtime.plugable.DeserializationDelegate.read(DeserializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.serialization.AdaptiveSpanningRecordDeserializer.getNextRecord(AdaptiveSpanningRecordDeserializer.java:71)
at
org.apache.flink.runtime.io.network.channels.InputChannel.readRecord(InputChannel.java:189)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:176)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:170)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1179] Add button to JobManager web inte...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/374#issuecomment-74068773 Looks good. There is a small conflict with #384 , but we can try and fix this while merging. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/388 [FLINK-1533] [runtime] Fixes NPE in the scheduler Adds checks to the method ```Scheduler.getFreeSubSlotForTask``` to handle failed shared slot allocations. Before the slot was given to the ```SlotSharingGroupAssignment``` without checking that it is not null. Consequently, a NullPointerException could occur in the ```SlotSharingGroupAssignment```. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink fixScheduler Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/388.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #388 commit 604728df38c860ab111d0a7bf2d19c71a0d30c67 Author: Till Rohrmann trohrm...@apache.org Date: 2015-02-12T13:51:45Z [FLINK-1533] [runtime] Fixes NPE in the scheduler where the allocated shared slots are not properly checked. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1510) Make AvroInputFormat splittable
[ https://issues.apache.org/jira/browse/FLINK-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318161#comment-14318161 ] ASF GitHub Bot commented on FLINK-1510: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/387#issuecomment-74068314 Looks good to me. +1 to add Make AvroInputFormat splittable --- Key: FLINK-1510 URL: https://issues.apache.org/jira/browse/FLINK-1510 Project: Flink Issue Type: Improvement Reporter: Robert Metzger Assignee: Robert Metzger Avro supports splitting files using synchronization points. IFs have to read between these points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1391] Register common Avro types at Kry...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/386#issuecomment-74068511 Yep. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1510] Make AvroInputFormat splittable
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/387#issuecomment-74068314 Looks good to me. +1 to add --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318195#comment-14318195 ] ASF GitHub Bot commented on FLINK-1436: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74072237 Good work. Thank you. I think its time for your first push to master ;) Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user
[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74072268 Remember to include This closes #311 ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318196#comment-14318196 ] ASF GitHub Bot commented on FLINK-1436: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74072268 Remember to include This closes #311 ;) Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a difference for a novice.
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318465#comment-14318465
]
ASF GitHub Bot commented on FLINK-1531:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/391#issuecomment-74099482
Tests for release-0.8 are running.
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/391#issuecomment-74099482 Tests for release-0.8 are running. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [builds] Print Java process stack traces of st...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/389#issuecomment-74100276 I really like this feature. Will make all our lives less frustrating ;-) On Thu, Feb 12, 2015 at 3:33 PM, Robert Metzger notificati...@github.com wrote: Very nicely implemented! My bash scripts are certainly uglier ;) +1 for merging it â Reply to this email directly or view it on GitHub https://github.com/apache/flink/pull/389#issuecomment-74079695. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74088065 @rmetzger Thanks but I think it should be This closes #331 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1436] refactor CLiFrontend to provide m...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/331 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes updated FLINK-1531:
Priority: Minor (was: Major)
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318361#comment-14318361 ] ASF GitHub Bot commented on FLINK-1436: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/331#issuecomment-74088065 @rmetzger Thanks but I think it should be This closes #331 :) Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a difference
[jira] [Resolved] (FLINK-1424) bin/flink run does not recognize -c parameter anymore
[
https://issues.apache.org/jira/browse/FLINK-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Michels resolved FLINK-1424.
Resolution: Fixed
Fix Version/s: 0.9
bin/flink run does not recognize -c parameter anymore
-
Key: FLINK-1424
URL: https://issues.apache.org/jira/browse/FLINK-1424
Project: Flink
Issue Type: Bug
Components: TaskManager
Affects Versions: master
Reporter: Carsten Brandt
Assignee: Max Michels
Fix For: 0.9
bin/flink binary does not recognize `-c` parameter anymore which specifies
the class to run:
{noformat}
$ ./flink run /path/to/target/impro3-ws14-flink-1.0-SNAPSHOT.jar -c
de.tu_berlin.impro3.flink.etl.FollowerGraphGenerator /tmp/flink/testgraph.txt
1
usage: emma-experiments-impro3-ss14-flink
[-?]
emma-experiments-impro3-ss14-flink: error: unrecognized arguments: '-c'
{noformat}
before this command worked fine and executed the job.
I tracked it down to the following commit using `git bisect`:
{noformat}
93eadca782ee8c77f89609f6d924d73021dcdda9 is the first bad commit
commit 93eadca782ee8c77f89609f6d924d73021dcdda9
Author: Alexander Alexandrov alexander.s.alexand...@gmail.com
Date: Wed Dec 24 13:49:56 2014 +0200
[FLINK-1027] [cli] Added support for '--' and '-' prefixed tokens in CLI
program arguments.
This closes #278
:04 04 a1358e6f7fe308b4d51a47069f190a29f87fdeda
d6f11bbc9444227d5c6297ec908e44b9644289a9 Mflink-clients
{noformat}
https://github.com/apache/flink/commit/93eadca782ee8c77f89609f6d924d73021dcdda9
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318314#comment-14318314
]
Johannes commented on FLINK-1531:
-
I just had a look again at the logging again
Right at the beginning I see
{code}
15:56:17.009 [main] INFO org.apache.flink.api.java.typeutils.TypeExtractor -
Class class TestKryoIterationSerializer$DomainObject is not a valid POJO type
{code}
Which is clear, as this one should be managed by Kryos serializer.
Looks like if the process fails on this issue, it would be clearer.
It seems that the serializer is only invoked for fields. So if I wrap the
object into a value holder or tuple class it works fine, without the default
constructor or any of the setters or getters.
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1436) Command-line interface verbose option error reporting
[ https://issues.apache.org/jira/browse/FLINK-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318405#comment-14318405 ] ASF GitHub Bot commented on FLINK-1436: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/331 Command-line interface verbose option error reporting --- Key: FLINK-1436 URL: https://issues.apache.org/jira/browse/FLINK-1436 Project: Flink Issue Type: Improvement Components: Start-Stop Scripts Reporter: Max Michels Assignee: Max Michels Priority: Trivial Labels: starter, usability Let me run just a basic Flink job and add the verbose flag. It's a general option, so let me add it as a first parameter: ./flink -v run ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 Invalid action! ./flink ACTION [GENERAL_OPTIONS] [ARGUMENTS] general options: -h,--help Show the help for the CLI Frontend. -v,--verbose Print more detailed error messages. Action run compiles and runs a program. Syntax: run [OPTIONS] jar-file arguments run action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action info displays information about a program. info action arguments: -c,--class classname Class with the program entry point (main method or getPlan() method. Only needed if the JAR file does not specify the class in its manifest. -e,--executionplan Show optimized execution plan of the program (JSON) -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -p,--parallelism parallelism The parallelism with which to run the program. Optional flag to override the default value specified in the configuration. Action list lists running and finished programs. list action arguments: -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. -r,--running Show running programs and their JobIDs -s,--scheduledShow scheduled prorgrams and their JobIDs Action cancel cancels a running program. cancel action arguments: -i,--jobid jobIDJobID of program to cancel -m,--jobmanager host:port Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration. What just happened? This results in a lot of output which is usually generated if you use the --help option on command-line tools. If your terminal window is large enough, then you will see a tiny message: Please specify an action. I did specify an action. Strange. If you read the help messages carefully you see, that general options belong to the action. ./flink run -v ../examples/flink-java-examples-0.8.0-WordCount.jar hdfs:///input hdfs:///output9 For the sake of mitigating user frustration, let us also accept -v as the first argument. It may seem trivial for the day-to-day Flink user but makes a difference for a novice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318430#comment-14318430
]
ASF GitHub Bot commented on FLINK-1531:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/391
[FLINK-1531] Fixes KryoSerializer's EOFException forwarding
Adds proper EOFException forwarding to the KryoSerializer.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink fixKryoUnderflow
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/391.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #391
commit 8e9c94b50c55f31c311d60ff63bd42108962e9a4
Author: Till Rohrmann trohrm...@apache.org
Date: 2015-02-12T15:50:46Z
[FLINK-1531] [runtime] Adds proper EOFException forwarding to
KryoSerializer.
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318440#comment-14318440
]
ASF GitHub Bot commented on FLINK-1531:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/391#issuecomment-74097085
Looks good
+1
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318436#comment-14318436
]
Till Rohrmann commented on FLINK-1531:
--
Thanks for the great bug description with the additional UnitTest [~jkirsch].
The reason for the failure was that the KryoSerializer did not properly
forwarded the EOFException of the underlying stream. The EOFException is used
by the system to know when all data of an input channel has been consumed.
I created a PR for this and once all tests pass on Travis, I'll commit it.
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/391#issuecomment-74097085 Looks good +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318463#comment-14318463
]
ASF GitHub Bot commented on FLINK-1531:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/391#issuecomment-74099311
We should also put the fix into the release-0.8 branch.
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1531] Fixes KryoSerializer's EOFExcepti...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/391#issuecomment-74099311 We should also put the fix into the release-0.8 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
[
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318544#comment-14318544
]
ASF GitHub Bot commented on FLINK-1533:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/388#issuecomment-74107842
Ok, I'll merge it.
NullPointerException in
SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
--
Key: FLINK-1533
URL: https://issues.apache.org/jira/browse/FLINK-1533
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
The scheduler can fail with a NullPointerException if it is not possible to
allocate a {{SharedSlot}} from an {{Instance}}. In the method
{{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not
checked that it is not null. This can cause a program to fail if there is a
race condition in scheduling different tasks on the same instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318543#comment-14318543
]
ASF GitHub Bot commented on FLINK-1531:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/391#issuecomment-74107768
Ok, I'll merge it.
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/388#issuecomment-74107842 Ok, I'll merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1536) GSoC project: Graph partitioning operators for Gelly
Vasia Kalavri created FLINK-1536: Summary: GSoC project: Graph partitioning operators for Gelly Key: FLINK-1536 URL: https://issues.apache.org/jira/browse/FLINK-1536 Project: Flink Issue Type: New Feature Components: Gelly, Java API Reporter: Vasia Kalavri Priority: Minor Smart graph partitioning can significantly improve the performance and scalability of graph analysis applications. Depending on the computation pattern, a graph partitioning algorithm divides the graph into (maybe overlapping) subgraphs, optimizing some objective. For example, if communication is performed across graph edges, one might want to minimize the edges that cross from one partition to another. The problem of graph partitioning is a well studied problem and several algorithms have been proposed in the literature. The goal of this project would be to choose a few existing partitioning techniques and implement the corresponding graph partitioning operators for Gelly. Some related literature can be found [here| http://www.citeulike.org/user/vasiakalavri/tag/graph-partitioning]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1533] [runtime] Fixes NPE in the schedu...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/388 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
[
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318604#comment-14318604
]
ASF GitHub Bot commented on FLINK-1533:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/388
NullPointerException in
SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
--
Key: FLINK-1533
URL: https://issues.apache.org/jira/browse/FLINK-1533
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
The scheduler can fail with a NullPointerException if it is not possible to
allocate a {{SharedSlot}} from an {{Instance}}. In the method
{{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not
checked that it is not null. This can cause a program to fail if there is a
race condition in scheduling different tasks on the same instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318603#comment-14318603
]
ASF GitHub Bot commented on FLINK-1531:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/391
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-1531) Custom Kryo Serializer fails in itertation scenario
[
https://issues.apache.org/jira/browse/FLINK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-1531.
Resolution: Fixed
Fix Version/s: 0.8.1
0.9
Fixed for 0.9 in 21f47d9c69441c17b5f90ea2c7cb8f4d47f7fcb5
Fixed for 0.8 in a669d785e74534731c19e982e47457475420f4a7
Custom Kryo Serializer fails in itertation scenario
---
Key: FLINK-1531
URL: https://issues.apache.org/jira/browse/FLINK-1531
Project: Flink
Issue Type: Bug
Components: Iterations
Affects Versions: 0.9
Reporter: Johannes
Priority: Minor
Fix For: 0.9, 0.8.1
Attachments: TestKryoIterationSerializer.java
When using iterations with a custom serializer for a domain object, the
iteration will fail.
{code:java}
org.apache.flink.runtime.client.JobExecutionException:
com.esotericsoftware.kryo.KryoException: Buffer underflow
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:76)
at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:198)
at
org.apache.flink.api.java.typeutils.runtime.KryoSerializer.deserialize(KryoSerializer.java:203)
at
org.apache.flink.runtime.io.disk.InputViewIterator.next(InputViewIterator.java:43)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.streamOutFinalOutputBulk(IterationHeadPactTask.java:404)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:377)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-1533) NullPointerException in SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
[
https://issues.apache.org/jira/browse/FLINK-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-1533.
Resolution: Fixed
Fixed in bc1432a2f605f97a27b0075c9f54b46c74e42d56
NullPointerException in
SlotSharingGroupAssignment.addSharedSlotAndAllocateSubSlot
--
Key: FLINK-1533
URL: https://issues.apache.org/jira/browse/FLINK-1533
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
The scheduler can fail with a NullPointerException if it is not possible to
allocate a {{SharedSlot}} from an {{Instance}}. In the method
{{Scheduler.getFreeSubSlotForTask}} the allocated {{SharedSlot}} is not
checked that it is not null. This can cause a program to fail if there is a
race condition in scheduling different tasks on the same instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
