[jira] [Commented] (BEAM-1146) Decrease spark runner startup overhead

[ https://issues.apache.org/jira/browse/BEAM-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760560#comment-15760560 ] Amit Sela commented on BEAM-1146: - As for {{Spark}} and {{Kryo}} - we can't change Spark dependencies

[jira] [Commented] (BEAM-1177) Input DStream "bundles" should be in serialized form and include relevant metadata.

[ https://issues.apache.org/jira/browse/BEAM-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15758852#comment-15758852 ] Amit Sela commented on BEAM-1177: - Instead of simply emitting {{Iterable}} per partition, I'll emit

[jira] [Updated] (BEAM-1177) Input DStream "bundles" should be in serialized form and include relevant metadata.

[ https://issues.apache.org/jira/browse/BEAM-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1177: Description: Currently, the input partitions hold "bundles" of read elements within the

[jira] [Created] (BEAM-1177) Input DStream "bundles" should be in serialized form and include relevant metadata.

Amit Sela created BEAM-1177: --- Summary: Input DStream "bundles" should be in serialized form and include relevant metadata. Key: BEAM-1177 URL: https://issues.apache.org/jira/browse/BEAM-1177 Project: Beam

[jira] [Updated] (BEAM-848) A better shuffle after reading from within mapWithState.

[ https://issues.apache.org/jira/browse/BEAM-848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-848: --- Summary: A better shuffle after reading from within mapWithState. (was: Make post-read (unbounded) shuffle use

[jira] [Resolved] (BEAM-853) Force streaming execution on batch pipelines for testing.

[ https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-853. Resolution: Fixed Fix Version/s: 0.5.0-incubating > Force streaming execution on batch pipelines for

[jira] [Created] (BEAM-1173) TestSparkRunner should use a PipelineVisitor to determine expected assertions.

Amit Sela created BEAM-1173: --- Summary: TestSparkRunner should use a PipelineVisitor to determine expected assertions. Key: BEAM-1173 URL: https://issues.apache.org/jira/browse/BEAM-1173 Project: Beam

[jira] [Reopened] (BEAM-855) Remove the need for --streaming option in the spark runner

[ https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reopened BEAM-855: Assignee: (was: Jean-Baptiste Onofré) > Remove the need for --streaming option in the spark runner >

[jira] [Closed] (BEAM-855) Remove the need for --streaming option in the spark runner

[ https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-855. -- Resolution: Duplicate Fix Version/s: Not applicable > Remove the need for --streaming option in the spark

[jira] [Updated] (BEAM-757) The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.

[ https://issues.apache.org/jira/browse/BEAM-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-757: --- Fix Version/s: (was: 0.4.0-incubating) 0.5.0-incubating > The SparkRunner should utilize

[jira] [Updated] (BEAM-807) [SparkRunner] Replace OldDoFn with DoFn

[ https://issues.apache.org/jira/browse/BEAM-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-807: --- Fix Version/s: (was: 0.4.0-incubating) 0.5.0-incubating > [SparkRunner] Replace OldDoFn

[jira] [Updated] (BEAM-855) Remove the need for --streaming option in the spark runner

[ https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-855: --- Fix Version/s: (was: 0.4.0-incubating) > Remove the need for --streaming option in the spark runner >

[jira] [Closed] (BEAM-1156) Spark runner does not support side-inputs in multiple windows.

[ https://issues.apache.org/jira/browse/BEAM-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-1156. --- Resolution: Duplicate Fix Version/s: Not applicable > Spark runner does not support side-inputs in

[jira] [Commented] (BEAM-1149) Side input access fails in direct runner (possibly others too) when input element in multiple windows

[ https://issues.apache.org/jira/browse/BEAM-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749222#comment-15749222 ] Amit Sela commented on BEAM-1149: - FWIW it works for me if I replace: {code} @Override public void

[jira] [Updated] (BEAM-853) Force streaming execution on batch pipelines for testing.

[ https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-853: --- Description: The SDK's {{streaming}} tests actually use a {{BoundedReadFromUnboundedSource}} while "forcing" a

[jira] [Issue Comment Deleted] (BEAM-853) Force streaming execution on batch pipelines for testing.

[ https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-853: --- Comment: was deleted (was: Streaming tests require forcing streaming mode on {{BoundedFromUnbounded}}, and

[jira] [Issue Comment Deleted] (BEAM-853) Force streaming execution on batch pipelines for testing.

[ https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-853: --- Comment: was deleted (was: I have this one sorted out already, but I'll make it a part of BEAM-920 since there

[jira] [Resolved] (BEAM-807) [SparkRunner] Replace OldDoFn with DoFn

[ https://issues.apache.org/jira/browse/BEAM-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-807. Resolution: Fixed Fix Version/s: 0.4.0-incubating Resolved by:

[jira] [Resolved] (BEAM-757) The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.

[ https://issues.apache.org/jira/browse/BEAM-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-757. Resolution: Fixed Fix Version/s: 0.4.0-incubating > The SparkRunner should utilize the SDK's

[jira] [Created] (BEAM-1156) Spark runner does not support side-inputs in multiple windows.

Amit Sela created BEAM-1156: --- Summary: Spark runner does not support side-inputs in multiple windows. Key: BEAM-1156 URL: https://issues.apache.org/jira/browse/BEAM-1156 Project: Beam Issue Type:

[jira] [Commented] (BEAM-1155) Spark runner aggregators only support a handfuls of combiners

[ https://issues.apache.org/jira/browse/BEAM-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747772#comment-15747772 ] Amit Sela commented on BEAM-1155: - I think the effort should be towards the Metrics API. See BEAM-147. >

[jira] [Commented] (BEAM-1144) Spark runner fails to deserialize MicrobatchSource in cluster mode

[ https://issues.apache.org/jira/browse/BEAM-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745998#comment-15745998 ] Amit Sela commented on BEAM-1144: - [~davor] normally I would say yes, but since the Spark runner does not

[jira] [Commented] (BEAM-1146) Decrease spark runner startup overhead

[ https://issues.apache.org/jira/browse/BEAM-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745479#comment-15745479 ] Amit Sela commented on BEAM-1146: - [~davor] you have thoughts about this ? I know there are thoughts about

[jira] [Resolved] (BEAM-1133) Add maxNumRecords per micro-batch for Spark runner options.

[ https://issues.apache.org/jira/browse/BEAM-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1133. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > Add maxNumRecords per micro-batch for Spark

[jira] [Resolved] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1130. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > SparkRunner ResumeFromCheckpointStreamingTest

[jira] [Commented] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741603#comment-15741603 ] Amit Sela commented on BEAM-1130: - There are currently three tests reading from Kafka (also testing

[jira] [Issue Comment Deleted] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1130: Comment: was deleted (was: To be resolved by) > SparkRunner ResumeFromCheckpointStreamingTest Failing >

[jira] [Commented] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741586#comment-15741586 ] Amit Sela commented on BEAM-1130: - To be resolved by > SparkRunner ResumeFromCheckpointStreamingTest

[jira] [Created] (BEAM-1133) Add maxNumRecords per micro-batch for Spark runner options.

Amit Sela created BEAM-1133: --- Summary: Add maxNumRecords per micro-batch for Spark runner options. Key: BEAM-1133 URL: https://issues.apache.org/jira/browse/BEAM-1133 Project: Beam Issue Type:

[jira] [Commented] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741422#comment-15741422 ] Amit Sela commented on BEAM-1130: - This is yet another reminder of Spak runner flaky streaming tests. I'll

[jira] [Assigned] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[ https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1130: --- Assignee: Amit Sela (was: Aviem Zur) > SparkRunner ResumeFromCheckpointStreamingTest Failing >

[jira] [Closed] (BEAM-637) Support Java serialisation (via JavaSerialiser)

[ https://issues.apache.org/jira/browse/BEAM-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-637. -- Resolution: Won't Fix Fix Version/s: Not applicable I don't see enough motivation to do this. Spark

[jira] [Resolved] (BEAM-1111) Reject timers for ParDo in SparkRunner streaming evaluators.

[ https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > Reject timers for ParDo in SparkRunner

[jira] [Created] (BEAM-1111) Reject timers for ParDo in SparkRunner streaming evaluators.

Amit Sela created BEAM-: --- Summary: Reject timers for ParDo in SparkRunner streaming evaluators. Key: BEAM- URL: https://issues.apache.org/jira/browse/BEAM- Project: Beam Issue Type:

[jira] [Commented] (BEAM-507) Fill in the documentation/runners/spark portion of the website

[ https://issues.apache.org/jira/browse/BEAM-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730054#comment-15730054 ] Amit Sela commented on BEAM-507: I'll have a PR by tomorrow. > Fill in the documentation/runners/spark

[jira] [Updated] (BEAM-329) Update Spark runner README.

[ https://issues.apache.org/jira/browse/BEAM-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-329: --- Summary: Update Spark runner README. (was: Spark runner README should have a proper batch example.) > Update

[jira] [Resolved] (BEAM-1094) Spark runner should define Kafka IO dependency with test scope

[ https://issues.apache.org/jira/browse/BEAM-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1094. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > Spark runner should define Kafka IO dependency

[jira] [Assigned] (BEAM-507) Fill in the documentation/runners/spark portion of the website

[ https://issues.apache.org/jira/browse/BEAM-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-507: -- Assignee: Amit Sela (was: James Malone) > Fill in the documentation/runners/spark portion of the

[jira] [Resolved] (BEAM-1050) PipelineResult.State is not set to FAILED when a streaming job fails

[ https://issues.apache.org/jira/browse/BEAM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1050. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > PipelineResult.State is not set to FAILED when

[jira] [Resolved] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

[ https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-595. Resolution: Fixed Fix Version/s: 0.4.0-incubating > Support non-blocking run() in SparkRunner and

[jira] [Updated] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

[ https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-595: --- Assignee: (was: Kobi Salant) > Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish()

[jira] [Resolved] (BEAM-1000) Add non blocking cancel() and waitUntilFinish() for Spark batch application

[ https://issues.apache.org/jira/browse/BEAM-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1000. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > Add non blocking cancel() and waitUntilFinish()

[jira] [Updated] (BEAM-1000) Add non blocking cancel() and waitUntilFinish() for Spark batch application

[ https://issues.apache.org/jira/browse/BEAM-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1000: Assignee: Stas Levin > Add non blocking cancel() and waitUntilFinish() for Spark batch application >

[jira] [Created] (BEAM-1075) Shuffle the input read-values to get maximum parallelism.

Amit Sela created BEAM-1075: --- Summary: Shuffle the input read-values to get maximum parallelism. Key: BEAM-1075 URL: https://issues.apache.org/jira/browse/BEAM-1075 Project: Beam Issue Type:

[jira] [Created] (BEAM-1074) Set default-partitioner in SourceRDD.Unbounded.

Amit Sela created BEAM-1074: --- Summary: Set default-partitioner in SourceRDD.Unbounded. Key: BEAM-1074 URL: https://issues.apache.org/jira/browse/BEAM-1074 Project: Beam Issue Type: Sub-task

[jira] [Updated] (BEAM-848) Make post-read (unbounded) shuffle use coders instead of Kryo.

[ https://issues.apache.org/jira/browse/BEAM-848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-848: --- Description: The SparkRunner uses {{mapWithState}} to read and manage CheckpointMarks, and this stateful

[jira] [Resolved] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[ https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1052. - Resolution: Fixed Fix Version/s: 0.4.0-incubating > UnboundedSource splitId uniqueness breaks if

[jira] [Created] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

Amit Sela created BEAM-1052: --- Summary: UnboundedSource splitId uniqueness breaks if more than one source is used. Key: BEAM-1052 URL: https://issues.apache.org/jira/browse/BEAM-1052 Project: Beam

[jira] [Resolved] (BEAM-851) Identify streaming pipelines implicitly.

[ https://issues.apache.org/jira/browse/BEAM-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-851. Resolution: Fixed Fix Version/s: 0.4.0-incubating > Identify streaming pipelines implicitly. >

[jira] [Updated] (BEAM-932) Findbugs doesn't pass in Spark runner

[ https://issues.apache.org/jira/browse/BEAM-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-932: --- Assignee: Ismaël Mejía (was: Amit Sela) > Findbugs doesn't pass in Spark runner >

[jira] [Closed] (BEAM-110) Use SDK implementation for HadoopIO in contrib/hadoop

[ https://issues.apache.org/jira/browse/BEAM-110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-110. -- Resolution: Duplicate Fix Version/s: Not applicable > Use SDK implementation for HadoopIO in

[jira] [Updated] (BEAM-18) Add support for new Beam Sink API

[ https://issues.apache.org/jira/browse/BEAM-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-18: -- Description: Support Beam Sinks via Spark {{foreach}} API, Spark's native sinks (if / where) possible. (was:

[jira] [Commented] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[ https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693377#comment-15693377 ] Amit Sela commented on BEAM-1048: - To clarify - calling `rdd.count()` on the read input RDD creates a batch

[jira] [Updated] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[ https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1048: Assignee: Kobi Salant (was: Amit Sela) > Spark Runner streaming batch duration does not include duration of

[jira] [Closed] (BEAM-470) Spark Runner does not send the job execution information into the Spark History Server

[ https://issues.apache.org/jira/browse/BEAM-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-470. -- Resolution: Not A Bug Fix Version/s: Not applicable > Spark Runner does not send the job execution

[jira] [Updated] (BEAM-470) Spark Runner does not send the job execution information into the Spark History Server

[ https://issues.apache.org/jira/browse/BEAM-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-470: --- Assignee: (was: Amit Sela) > Spark Runner does not send the job execution information into the Spark >

[jira] [Updated] (BEAM-637) Support Java serialisation (via JavaSerialiser)

[ https://issues.apache.org/jira/browse/BEAM-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-637: --- Assignee: (was: Amit Sela) > Support Java serialisation (via JavaSerialiser) >

[jira] [Commented] (BEAM-470) Spark Runner does not send the job execution information into the Spark History Server

[ https://issues.apache.org/jira/browse/BEAM-470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689594#comment-15689594 ] Amit Sela commented on BEAM-470: Can this be reproduced ? I've managed to run a local history server and see

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[ https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689580#comment-15689580 ] Amit Sela commented on BEAM-981: Anyway, I'll add that {{spark-submit}} is considered to be the "right way"

[jira] [Updated] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[ https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-981: --- Assignee: (was: Amit Sela) > Not possible to directly submit a pipeline on spark cluster >

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[ https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689574#comment-15689574 ] Amit Sela commented on BEAM-981: [~jbonofre] you've encountered this again lately, no ? any new input ?

[jira] [Commented] (BEAM-18) Add support for new Beam Sink API

[ https://issues.apache.org/jira/browse/BEAM-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689541#comment-15689541 ] Amit Sela commented on BEAM-18: --- Most "actions" will probably be Sinks, so this might be a duplicate. Linking

[jira] [Closed] (BEAM-84) Add support for Session Windows - Beam Sessions

[ https://issues.apache.org/jira/browse/BEAM-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-84. - Resolution: Duplicate Fix Version/s: Not applicable Handled by BEAM-920 > Add support for Session Windows -

[jira] [Commented] (BEAM-229) Add support for additional Spark configuration via PipelineOptions

[ https://issues.apache.org/jira/browse/BEAM-229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689535#comment-15689535 ] Amit Sela commented on BEAM-229: [~jbonofre] we talked about it, but maybe there's a bigger question here

[jira] [Closed] (BEAM-543) Better enforcements of the SparkRunner windowing capabilities.

[ https://issues.apache.org/jira/browse/BEAM-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-543. -- Resolution: Won't Fix Fix Version/s: Not applicable Proper windowing support is handled in BEAM-920 >

[jira] [Updated] (BEAM-111) Use SDK implementation of WritableCoder

[ https://issues.apache.org/jira/browse/BEAM-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-111: --- Labels: easy starter (was: ) > Use SDK implementation of WritableCoder >

[jira] [Updated] (BEAM-111) Use SDK implementation of WritableCoder

[ https://issues.apache.org/jira/browse/BEAM-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-111: --- Description: The Spark runner currently uses it's own implementation of WritableCoder, should use the one in

[jira] [Updated] (BEAM-111) Use SDK implementation of WritableCoder

[ https://issues.apache.org/jira/browse/BEAM-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-111: --- Description: The Spark runner currently uses it's own implementation of WritableCoder, should use the one in

[jira] [Commented] (BEAM-111) Use SDK implementation of WritableCoder

[ https://issues.apache.org/jira/browse/BEAM-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689507#comment-15689507 ] Amit Sela commented on BEAM-111: Seems like {{org.apache.beam.sdk.io.hdfs.WritableCoder}} is (almost)

[jira] [Commented] (BEAM-110) Use SDK implementation for HadoopIO in contrib/hadoop

[ https://issues.apache.org/jira/browse/BEAM-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689495#comment-15689495 ] Amit Sela commented on BEAM-110: [~davor] I think that concerning IOs it's clear that "direct translation"

[jira] [Updated] (BEAM-649) Pipeline "actions" should use foreachRDD via ParDo.

[ https://issues.apache.org/jira/browse/BEAM-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-649: --- Issue Type: Improvement (was: Bug) > Pipeline "actions" should use foreachRDD via ParDo. >

[jira] [Updated] (BEAM-649) Pipeline "actions" should use foreachRDD via ParDo.

[ https://issues.apache.org/jira/browse/BEAM-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-649: --- Assignee: (was: Amit Sela) > Pipeline "actions" should use foreachRDD via ParDo. >

[jira] [Closed] (BEAM-669) SimpleStreamingWordCountTest does not have a test for sliding windows

[ https://issues.apache.org/jira/browse/BEAM-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-669. -- Resolution: Won't Fix Fix Version/s: Not applicable This will be tested as part of the SDK streaming tests

[jira] [Updated] (BEAM-648) Persist and restore Aggergator values in case of recovery from failure

[ https://issues.apache.org/jira/browse/BEAM-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-648: --- Assignee: (was: Amit Sela) > Persist and restore Aggergator values in case of recovery from failure >

[jira] [Commented] (BEAM-1039) Spark context is never actually re-used in tests

[ https://issues.apache.org/jira/browse/BEAM-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687734#comment-15687734 ] Amit Sela commented on BEAM-1039: - [~staslev] to the best of my knowledge {{Boolean#getBoolean(String)}}

[jira] [Updated] (BEAM-647) Fault-tolerant sideInputs via Broadcast variables

[ https://issues.apache.org/jira/browse/BEAM-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-647: --- Summary: Fault-tolerant sideInputs via Broadcast variables (was: Faul-tolerant sideInputs via Broadcast

[jira] [Updated] (BEAM-647) Faul-tolerant sideInputs via Broadcast variables.t

[ https://issues.apache.org/jira/browse/BEAM-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-647: --- Summary: Faul-tolerant sideInputs via Broadcast variables.t (was: Faul-tolerant sideInputs via Broadcast

[jira] [Updated] (BEAM-982) Document spark configuration update

[ https://issues.apache.org/jira/browse/BEAM-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-982: --- Assignee: (was: Amit Sela) > Document spark configuration update > --- > >

[jira] [Resolved] (BEAM-1001) Add non blocking cancel() and waitUntilFinish() for Spark streaming application

[ https://issues.apache.org/jira/browse/BEAM-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1001. - Resolution: Fixed Fix Version/s: 0.4.0-incubating Resolved by #1393:

[jira] [Commented] (BEAM-853) Force streaming execution on batch pipelines for testing.

[ https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679708#comment-15679708 ] Amit Sela commented on BEAM-853: I have this one sorted out already, but I'll make it a part of BEAM-920

[jira] [Commented] (BEAM-853) Force streaming execution on batch pipelines for testing.

[ https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679706#comment-15679706 ] Amit Sela commented on BEAM-853: Streaming tests require forcing streaming mode on {{BoundedFromUnbounded}},

[jira] [Updated] (BEAM-920) Support triggers and panes in streaming.

[ https://issues.apache.org/jira/browse/BEAM-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-920: --- Description: Implement event-time based aggregation using triggers, panes and watermarks. (was: Use Spark's

[jira] [Updated] (BEAM-920) Support triggers, panes and watermarks.

[ https://issues.apache.org/jira/browse/BEAM-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-920: --- Issue Type: New Feature (was: Bug) > Support triggers, panes and watermarks. >

[jira] [Updated] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

[ https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-595: --- Assignee: Kobi Salant (was: Amit Sela) > Support non-blocking run() in SparkRunner and cancel() and

[jira] [Resolved] (BEAM-983) Spark runner fails build on missing license and broken checkstyle.

[ https://issues.apache.org/jira/browse/BEAM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-983. Resolution: Fixed Fix Version/s: 0.4.0-incubating > Spark runner fails build on missing license and

[jira] [Updated] (BEAM-983) Spark runner fails build on missing license and broken checkstyle.

[ https://issues.apache.org/jira/browse/BEAM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-983: --- Description: Pull request #1332 (https://github.com/apache/incubator-beam/pull/1332) is failing in

[jira] [Updated] (BEAM-983) Spark runner fails build on missing license and broken checkstyle.

[ https://issues.apache.org/jira/browse/BEAM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-983: --- Assignee: Eugene Kirpichov (was: Amit Sela) > Spark runner fails build on missing license and broken

[jira] [Updated] (BEAM-983) Spark runner fails build on missing license and broken checkstyle.

[ https://issues.apache.org/jira/browse/BEAM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-983: --- Summary: Spark runner fails build on missing license and broken checkstyle. (was:

[jira] [Commented] (BEAM-983) runners/spark/translation/streaming/utils/TestPipelineOptions.java missing Apache license header.

[ https://issues.apache.org/jira/browse/BEAM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668220#comment-15668220 ] Amit Sela commented on BEAM-983: Not your fault. Jenkins didn't run in PreCommit - it's kind of lost it for

[jira] [Closed] (BEAM-984) rat plugin fails build because of a missing license.

[ https://issues.apache.org/jira/browse/BEAM-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-984. -- Resolution: Duplicate > rat plugin fails build because of a missing license. >

[jira] [Commented] (BEAM-983) runners/spark/translation/streaming/utils/TestPipelineOptions.java missing Apache license header.

[ https://issues.apache.org/jira/browse/BEAM-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667929#comment-15667929 ] Amit Sela commented on BEAM-983: I'll fix this now, thanks [~jasonkuster] >

[jira] [Created] (BEAM-984) rat plugin fails build because of a missing license.

Amit Sela created BEAM-984: -- Summary: rat plugin fails build because of a missing license. Key: BEAM-984 URL: https://issues.apache.org/jira/browse/BEAM-984 Project: Beam Issue Type: Bug

[jira] [Resolved] (BEAM-891) Flake in Spark metrics library?

[ https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-891. Resolution: Fixed Fix Version/s: 0.4.0-incubating > Flake in Spark metrics library? >

[jira] [Commented] (BEAM-979) ConcurrentModificationException exception after hours of running

[ https://issues.apache.org/jira/browse/BEAM-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666994#comment-15666994 ] Amit Sela commented on BEAM-979: If this is indeed what happens, the exception is just a symptom.. What

[jira] [Commented] (BEAM-918) Let users set STORAGE_LEVEL via SparkPipelineOptions.

[ https://issues.apache.org/jira/browse/BEAM-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666959#comment-15666959 ] Amit Sela commented on BEAM-918: [~jbonofre] you're good to go ;) > Let users set STORAGE_LEVEL via

[jira] [Resolved] (BEAM-762) Use composition over inheritance in spark StreamingEvaluationContext if two contexts are necessary.

[ https://issues.apache.org/jira/browse/BEAM-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-762. Resolution: Fixed Fix Version/s: 0.4.0-incubating > Use composition over inheritance in spark

[jira] [Commented] (BEAM-979) ConcurrentModificationException exception after hours of running

[ https://issues.apache.org/jira/browse/BEAM-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666904#comment-15666904 ] Amit Sela commented on BEAM-979: [~staslev] what pipeline did you execute ? is there any other information

[jira] [Commented] (BEAM-891) Flake in Spark metrics library?

[ https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663349#comment-15663349 ] Amit Sela commented on BEAM-891: Because most tests are batch.. > Flake in Spark metrics library? >

[jira] [Commented] (BEAM-891) Flake in Spark metrics library?

[ https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661544#comment-15661544 ] Amit Sela commented on BEAM-891: It can be overriden by specifying

[jira] [Commented] (BEAM-891) Flake in Spark metrics library?

[ https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661197#comment-15661197 ] Amit Sela commented on BEAM-891: I see your point. How about setting in all tests (that don't test metrics)

  1   2   3   4   >