[jira] [Created] (BEAM-724) NullPointerException in Snapshot State
Sumit Chawla created BEAM-724:
-
Summary: NullPointerException in Snapshot State
Key: BEAM-724
URL: https://issues.apache.org/jira/browse/BEAM-724
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Sumit Chawla
This seems like a timing Issue. Its reproducible in case where Kafka topic
does not exist yet. The UDF operator run is not called yet, but other thread
calls snapshot before that:
{code}
java.lang.RuntimeException: Error while triggering checkpoint for Source:
Read(UnboundedKafkaSource) -> ParDo(KafkaRecordToCTuple) ->
ParDo(NetFlowToFlowContentTransformation) ->
ParDo(FlowContentToRelationsTransformation) -> ParDo(GraphToJsonTransformation)
-> AnonymousParDo -> ParDo(KafkaWriter) (1/1)
at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:949)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Failed to draw state snapshot from function:
null
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotOperatorState(AbstractUdfStreamOperator.java:132)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:598)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:565)
at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:941)
... 5 more
Caused by: java.lang.NullPointerException
at
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.snapshotState(UnboundedSourceWrapper.java:340)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.snapshotState(UnboundedSourceWrapper.java:54)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotOperatorState(AbstractUdfStreamOperator.java:129)
... 8 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (BEAM-682) Invoker Class should be created in Thread
Sumit Chawla created BEAM-682:
-
Summary: Invoker Class should be created in Thread
Key: BEAM-682
URL: https://issues.apache.org/jira/browse/BEAM-682
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Affects Versions: 0.3.0-incubating
Reporter: Sumit Chawla
Assignee: Sumit Chawla
Priority: Critical
As of now the InvokerClass is being loaded in wrong classloader. It should be
loaded into Thread.currentThread.getContextClassLoader()
https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java#L167
{code}
Class> res =
(Class>)
unloaded
.load(DoFnInvokers.class.getClassLoader(),
ClassLoadingStrategy.Default.INJECTION)
.getLoaded();
{code}
Fix
{code}
Class> res =
(Class>)
unloaded
.load(Thread.currentThread().getContextClassLoader(),
ClassLoadingStrategy.Default.INJECTION)
.getLoaded();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-682) Invoker Class should be created in Thread Context Classloader
[
https://issues.apache.org/jira/browse/BEAM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Chawla updated BEAM-682:
--
Summary: Invoker Class should be created in Thread Context Classloader
(was: Invoker Class should be created in Thread)
> Invoker Class should be created in Thread Context Classloader
> -
>
> Key: BEAM-682
> URL: https://issues.apache.org/jira/browse/BEAM-682
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 0.3.0-incubating
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
>Priority: Critical
>
> As of now the InvokerClass is being loaded in wrong classloader. It should be
> loaded into Thread.currentThread.getContextClassLoader()
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java#L167
> {code}
> Class> res =
> (Class>)
> unloaded
> .load(DoFnInvokers.class.getClassLoader(),
> ClassLoadingStrategy.Default.INJECTION)
> .getLoaded();
> {code}
> Fix
> {code}
> Class> res =
> (Class>)
> unloaded
> .load(Thread.currentThread().getContextClassLoader(),
> ClassLoadingStrategy.Default.INJECTION)
> .getLoaded();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-642) FlinkRunner does not support Detached Mode
[
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513831#comment-15513831
]
Sumit Chawla commented on BEAM-642:
---
[~maxmil23] [~amitsela] Sorry i did not have permission to create a new issue.
So i cloned an existing one and updated that one. I could not change the
assignee as well.
> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
> Fix For: 0.3.0-incubating
>
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception:
> {code}
> LOG.info("Execution finished in {} msecs", result.getNetRuntime());
> {code}
> {code}
> org.apache.flink.api.common.InvalidProgramException: Job was submitted in
> detached mode. Results of job execution, such as accumulators, runtime, job
> id etc. are not available.
> at
> org.apache.flink.client.program.DetachedEnvironment$DetachedJobExecutionResult.getNetRuntime(DetachedEnvironment.java:98)
> ~[flink-clients_2.10-1.1.2.jar:1.1.2]
> at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116)
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
> at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:48)
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:183)
> ~[beam-sdks-java-core-0.2.0-incubating.jar:0.2.0-incubating]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_92]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_92]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_92]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320)
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-642) FlinkRunner does not support Detached Mode
[
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Chawla updated BEAM-642:
--
Description:
FlinkRunner gives exception when job is submitted in detached mode.
Following code will throw exception:
{code}
LOG.info("Execution finished in {} msecs", result.getNetRuntime());
{code}
was:
FlinkRunner gives exception when job is submitted in detached mode.
Following code will throw exception:
{{ LOG.info("Execution finished in {} msecs", result.getNetRuntime()); }}
> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Amit Sela
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception:
> {code}
> LOG.info("Execution finished in {} msecs", result.getNetRuntime());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-642) FlinkRunner does not support Detached Mode
[
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504842#comment-15504842
]
Sumit Chawla commented on BEAM-642:
---
Can somebody please assign this issue to me?
> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Amit Sela
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception:
> {{ LOG.info("Execution finished in {} msecs", result.getNetRuntime());
> }}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-642) FlinkRunner does not support Detached Mode
[
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Chawla updated BEAM-642:
--
Description:
FlinkRunner gives exception when job is submitted in detached mode.
Following code will throw exception:
{{ LOG.info("Execution finished in {} msecs", result.getNetRuntime()); }}
was:
When using {{JavaSerializer}} instead of {{KryoSerializer}} by configuring
{{JavaSparkContext}} with {{"spark.serializer"}} set to
{{JavaSerializer.class.getCanonicalName()}}, an exception is thrown:
{noformat}
object not serializable (class:
org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value:
ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
{noformat}
{noformat}
Serialization stack:
- object not serializable (class:
org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value:
ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
at
org.apache.spark.api.java.JavaRDDLike$class.count(JavaRDDLike.scala:440)
at
org.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:46)
at
org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:175)
at
org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:172)
at
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
at
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at
org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
at
[jira] [Created] (BEAM-642) FlinkRunner does not support Detached Mode
Sumit Chawla created BEAM-642:
-
Summary: FlinkRunner does not support Detached Mode
Key: BEAM-642
URL: https://issues.apache.org/jira/browse/BEAM-642
Project: Beam
Issue Type: Bug
Components: runner-spark
Reporter: Sumit Chawla
Assignee: Amit Sela
When using {{JavaSerializer}} instead of {{KryoSerializer}} by configuring
{{JavaSparkContext}} with {{"spark.serializer"}} set to
{{JavaSerializer.class.getCanonicalName()}}, an exception is thrown:
{noformat}
object not serializable (class:
org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value:
ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
{noformat}
{noformat}
Serialization stack:
- object not serializable (class:
org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value:
ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
at
org.apache.spark.api.java.JavaRDDLike$class.count(JavaRDDLike.scala:440)
at
org.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:46)
at
org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:175)
at
org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:172)
at
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
at
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at
org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
at
[jira] [Updated] (BEAM-642) FlinkRunner does not support Detached Mode
[
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Chawla updated BEAM-642:
--
Component/s: (was: runner-spark)
runner-flink
> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Amit Sela
>
> When using {{JavaSerializer}} instead of {{KryoSerializer}} by configuring
> {{JavaSparkContext}} with {{"spark.serializer"}} set to
> {{JavaSerializer.class.getCanonicalName()}}, an exception is thrown:
> {noformat}
> object not serializable (class:
> org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value:
> ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
> {noformat}
> {noformat}
> Serialization stack:
> - object not serializable (class:
> org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value:
> ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
> at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
> at
> org.apache.spark.api.java.JavaRDDLike$class.count(JavaRDDLike.scala:440)
> at
> org.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:46)
> at
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:175)
> at
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:172)
> at
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
> at
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
> at
> org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
> at scala.util.Try$.apply(Try.scala:161)
> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
> at
>
[jira] [Updated] (BEAM-635) Release 0.2.0-incubating - Support Flink Release Version 1.1.2
[ https://issues.apache.org/jira/browse/BEAM-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla updated BEAM-635: -- Summary: Release 0.2.0-incubating - Support Flink Release Version 1.1.2 (was: Release 0.2.0-incubating - Support Flink Release Version 1.1.0) > Release 0.2.0-incubating - Support Flink Release Version 1.1.2 > -- > > Key: BEAM-635 > URL: https://issues.apache.org/jira/browse/BEAM-635 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sumit Chawla > > Support Latest Flink Release of version of 1.1.0 in BEAM 0.2.0-incubating > release > 0.2.0-incubating has just been released. It will be great if we can add > support for latest Flink 1.1.0 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-635) Release 0.2.0-incubating - Support Flink Release Version 1.1.0
[ https://issues.apache.org/jira/browse/BEAM-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla updated BEAM-635: -- Description: Support Latest Flink Release of version of 1.1.0 in BEAM 0.2.0-incubating release 0.2.0-incubating has just been released. It will be great if we can add support for latest Flink 1.1.0 release. was: Support Latest Flink Release of version of 1.1.0 in BEAM 0.2.0-incubating release 0.2.0-incubating has just been release. It will be great if we can add support for latest Flink 1.1.0 release. > Release 0.2.0-incubating - Support Flink Release Version 1.1.0 > -- > > Key: BEAM-635 > URL: https://issues.apache.org/jira/browse/BEAM-635 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sumit Chawla > > Support Latest Flink Release of version of 1.1.0 in BEAM 0.2.0-incubating > release > 0.2.0-incubating has just been released. It will be great if we can add > support for latest Flink 1.1.0 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-635) Release 0.2.0-incubating - Support Flink Release Version 1.1.0
[ https://issues.apache.org/jira/browse/BEAM-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla updated BEAM-635: -- Description: Support Latest Flink Release of version of 1.1.0 in BEAM 0.2.0-incubating release 0.2.0-incubating has just been release. It will be great if we can add support for latest Flink 1.1.0 release. was: Flink has added support for CodeHale Metrics (https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html) These metrics are more advanced then the current Accumulators. Adding support for these to Beam level should be a good addition. https://github.com/apache/flink/pull/1947#issuecomment-233029166 > Release 0.2.0-incubating - Support Flink Release Version 1.1.0 > -- > > Key: BEAM-635 > URL: https://issues.apache.org/jira/browse/BEAM-635 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sumit Chawla > > Support Latest Flink Release of version of 1.1.0 in BEAM 0.2.0-incubating > release > 0.2.0-incubating has just been release. It will be great if we can add > support for latest Flink 1.1.0 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-635) Release 0.2.0-incubating - Support Flink Release Version 1.1.0
[ https://issues.apache.org/jira/browse/BEAM-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla updated BEAM-635: -- Summary: Release 0.2.0-incubating - Support Flink Release Version 1.1.0 (was: Support for Flink Release Version 1.1.0) > Release 0.2.0-incubating - Support Flink Release Version 1.1.0 > -- > > Key: BEAM-635 > URL: https://issues.apache.org/jira/browse/BEAM-635 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sumit Chawla > > Flink has added support for CodeHale Metrics > (https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html) > These metrics are more advanced then the current Accumulators. > Adding support for these to Beam level should be a good addition. > https://github.com/apache/flink/pull/1947#issuecomment-233029166 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-635) Support for Flink Release Version 1.1.0
Sumit Chawla created BEAM-635: - Summary: Support for Flink Release Version 1.1.0 Key: BEAM-635 URL: https://issues.apache.org/jira/browse/BEAM-635 Project: Beam Issue Type: New Feature Components: beam-model Reporter: Sumit Chawla Flink has added support for CodeHale Metrics (https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html) These metrics are more advanced then the current Accumulators. Adding support for these to Beam level should be a good addition. https://github.com/apache/flink/pull/1947#issuecomment-233029166 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-458) Support for Flink Metrics
[ https://issues.apache.org/jira/browse/BEAM-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla updated BEAM-458: -- Description: Flink has added support for CodeHale Metrics (https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html) These metrics are more advanced then the current Accumulators. Adding support for these to Beam level should be a good addition. https://github.com/apache/flink/pull/1947#issuecomment-233029166 was: Flink has added support for CodeHale Metrics (https://ci.apache.org/projects/flink/flink-docs-master/apis/metrics.html#registering-metrics) These metrics are more advanced then the current Accumulators. Adding support for these to Beam level should be a good addition. https://github.com/apache/flink/pull/1947#issuecomment-233029166 > Support for Flink Metrics > -- > > Key: BEAM-458 > URL: https://issues.apache.org/jira/browse/BEAM-458 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Sumit Chawla > > Flink has added support for CodeHale Metrics > (https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html) > These metrics are more advanced then the current Accumulators. > Adding support for these to Beam level should be a good addition. > https://github.com/apache/flink/pull/1947#issuecomment-233029166 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BEAM-458) Support for Flink Metrics
Sumit Chawla created BEAM-458: - Summary: Support for Flink Metrics Key: BEAM-458 URL: https://issues.apache.org/jira/browse/BEAM-458 Project: Beam Issue Type: New Feature Components: runner-core, runner-flink Reporter: Sumit Chawla Assignee: Frances Perry Priority: Critical Flink has added support for CodeHale Metrics (https://ci.apache.org/projects/flink/flink-docs-master/apis/metrics.html#registering-metrics) These metrics are more advanced then the current Accumulators. Adding support for these to Beam level should be a good addition. https://github.com/apache/flink/pull/1947#issuecomment-233029166 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
