[jira] [Commented] (BEAM-5519) Spark Streaming Duplicated Encoding/Decoding Effort

[ https://issues.apache.org/jira/browse/BEAM-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634849#comment-16634849 ] Amit Sela commented on BEAM-5519: - [~winkelman.kyle] you bring up a good point. IIRC we did all this mess

[jira] [Commented] (BEAM-1789) window can't not use in spark cluster module

[ https://issues.apache.org/jira/browse/BEAM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974601#comment-15974601 ] Amit Sela commented on BEAM-1789: - Thanks for clearing this for me, now I can pinpoint the issue. Starting

[jira] [Comment Edited] (BEAM-375) HadoopIO and runners-spark conflict with hadoop.version

[ https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967634#comment-15967634 ] Amit Sela edited comment on BEAM-375 at 4/13/17 2:16 PM: - What do you mean by

[jira] [Commented] (BEAM-375) HadoopIO and runners-spark conflict with hadoop.version

[ https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967634#comment-15967634 ] Amit Sela commented on BEAM-375: What do you mean by "blocking" ? > HadoopIO and runners-spark conflict

[jira] [Commented] (BEAM-1789) window can't not use in spark cluster module

[ https://issues.apache.org/jira/browse/BEAM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965633#comment-15965633 ] Amit Sela commented on BEAM-1789: - Sorry, I meant {{stderr}}, {{stdout}} never shows anything in Spark for

[jira] [Commented] (BEAM-1789) window can't not use in spark cluster module

[ https://issues.apache.org/jira/browse/BEAM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965498#comment-15965498 ] Amit Sela commented on BEAM-1789: - I didn't get a chance to run this on cluster, but if I understand

[jira] [Commented] (BEAM-1920) Add Spark 2.x support in Spark runner

[ https://issues.apache.org/jira/browse/BEAM-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962076#comment-15962076 ] Amit Sela commented on BEAM-1920: - Sweet! I'm considering Spark 2.x as default and {{1.6.3}} as the

[jira] [Resolved] (BEAM-1737) Implement a Single-output ParDo as a Multi-output ParDo with a single output

[ https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1737. - Resolution: Fixed Fix Version/s: First stable release > Implement a Single-output ParDo as a

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[ https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957480#comment-15957480 ] Amit Sela commented on BEAM-981: [~iemejia] wold you mind taking a look at this one ? we should be able to

[jira] [Commented] (BEAM-1737) Implement a Single-output ParDo as a Multi-output ParDo with a single output

[ https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956637#comment-15956637 ] Amit Sela commented on BEAM-1737: - Changed the title since the change will actually do just that. >

[jira] [Updated] (BEAM-1737) Implement a Single-output ParDo as a Multi-output ParDo with a single output

[ https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1737: Summary: Implement a Single-output ParDo as a Multi-output ParDo with a single output (was: Interpreting a

[jira] [Assigned] (BEAM-1737) Interpreting a Single-output ParDo as a Multi-output ParDo with a single output causes serialization failures

[ https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1737: --- Assignee: Amit Sela > Interpreting a Single-output ParDo as a Multi-output ParDo with a single >

[jira] [Resolved] (BEAM-1875) Remove Spark runner custom Hadoop and Avro IOs.

[ https://issues.apache.org/jira/browse/BEAM-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1875. - Resolution: Fixed Fix Version/s: First stable release > Remove Spark runner custom Hadoop and Avro

[jira] [Commented] (BEAM-1737) Interpreting a Single-output ParDo as a Multi-output ParDo with a single output causes serialization failures

[ https://issues.apache.org/jira/browse/BEAM-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954833#comment-15954833 ] Amit Sela commented on BEAM-1737: - This is just because you are not allowed to use not Serializables such

[jira] [Created] (BEAM-1875) Remove Spark runner custom Hadoop and Avro IOs.

Amit Sela created BEAM-1875: --- Summary: Remove Spark runner custom Hadoop and Avro IOs. Key: BEAM-1875 URL: https://issues.apache.org/jira/browse/BEAM-1875 Project: Beam Issue Type: Improvement

[jira] [Commented] (BEAM-848) Shuffle input read-values to get maximum parallelism.

[ https://issues.apache.org/jira/browse/BEAM-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953760#comment-15953760 ] Amit Sela commented on BEAM-848: Agree. closed as invalid. If use cases prove the need, we can consider a

[jira] [Closed] (BEAM-848) Shuffle input read-values to get maximum parallelism.

[ https://issues.apache.org/jira/browse/BEAM-848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-848. -- Resolution: Invalid Fix Version/s: First stable release > Shuffle input read-values to get maximum

[jira] [Closed] (BEAM-696) Document: Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-696. -- Resolution: Not A Problem Fix Version/s: Not applicable Resolved by providing better documentation. See

[jira] [Resolved] (BEAM-1827) Fix use of deprecated Spark APIs in the runner.

[ https://issues.apache.org/jira/browse/BEAM-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1827. - Resolution: Fixed Fix Version/s: First stable release > Fix use of deprecated Spark APIs in the

[jira] [Created] (BEAM-1827) Fix use of deprecated Spark APIs in the runner.

Amit Sela created BEAM-1827: --- Summary: Fix use of deprecated Spark APIs in the runner. Key: BEAM-1827 URL: https://issues.apache.org/jira/browse/BEAM-1827 Project: Beam Issue Type: Improvement

[jira] [Resolved] (BEAM-1815) Avoid shuffling twice in GABW

[ https://issues.apache.org/jira/browse/BEAM-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1815. - Resolution: Fixed Fix Version/s: First stable release > Avoid shuffling twice in GABW >

[jira] [Comment Edited] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

[ https://issues.apache.org/jira/browse/BEAM-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943698#comment-15943698 ] Amit Sela edited comment on BEAM-1717 at 3/27/17 5:46 PM: -- [~davor]

[jira] [Commented] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

[ https://issues.apache.org/jira/browse/BEAM-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943698#comment-15943698 ] Amit Sela commented on BEAM-1717: - [~davor] {{beam-sdks-java-core}} is OK now, but now the release fails on

[jira] [Created] (BEAM-1815) Avoid shuffling twice in GABW

Amit Sela created BEAM-1815: --- Summary: Avoid shuffling twice in GABW Key: BEAM-1815 URL: https://issues.apache.org/jira/browse/BEAM-1815 Project: Beam Issue Type: Bug Components:

[jira] [Commented] (BEAM-1802) Spark Runner does not shutdown correctly when executing multiple pipelines in sequence

[ https://issues.apache.org/jira/browse/BEAM-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940422#comment-15940422 ] Amit Sela commented on BEAM-1802: - {{PipelineResult#cancel()}} should do the trick for now > Spark Runner

[jira] [Commented] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

[ https://issues.apache.org/jira/browse/BEAM-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938372#comment-15938372 ] Amit Sela commented on BEAM-1717: - Looks like https://github.com/apache/beam/pull/2261 and the patch I

[jira] [Commented] (BEAM-1789) window can't not use in spark cluster module

[ https://issues.apache.org/jira/browse/BEAM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938115#comment-15938115 ] Amit Sela commented on BEAM-1789: - Please keep in mind that Spark runner support for streaming is still

[jira] [Commented] (BEAM-1789) window can't not use in spark cluster module

[ https://issues.apache.org/jira/browse/BEAM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938032#comment-15938032 ] Amit Sela commented on BEAM-1789: - In local mode all the logging is in one single log, in cluster mode each

[jira] [Commented] (BEAM-1789) window can't not use in spark cluster module

[ https://issues.apache.org/jira/browse/BEAM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937871#comment-15937871 ] Amit Sela commented on BEAM-1789: - [~tianyou] where are you looking at the logs ? in Spark UI ? which

[jira] [Commented] (BEAM-1775) fix issue of start_from_previous_offset in KafkaIO

[ https://issues.apache.org/jira/browse/BEAM-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936210#comment-15936210 ] Amit Sela commented on BEAM-1775: - [~mingmxu] so you want this to be a "fallback" in case runners choose

[jira] [Commented] (BEAM-1765) Remove Aggregators from Spark runner

[ https://issues.apache.org/jira/browse/BEAM-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935311#comment-15935311 ] Amit Sela commented on BEAM-1765: - I'm wondering what's the timeline here ? last I checked direct and Spark

[jira] [Commented] (BEAM-1775) fix issue of start_from_previous_offset in KafkaIO

[ https://issues.apache.org/jira/browse/BEAM-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935305#comment-15935305 ] Amit Sela commented on BEAM-1775: - I think this is might be a Flink runner specific issue (or just a

[jira] [Commented] (BEAM-507) Fill in the documentation/runners/spark portion of the website

[ https://issues.apache.org/jira/browse/BEAM-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933238#comment-15933238 ] Amit Sela commented on BEAM-507: Yup, thanks for the GC ;) > Fill in the documentation/runners/spark

[jira] [Resolved] (BEAM-507) Fill in the documentation/runners/spark portion of the website

[ https://issues.apache.org/jira/browse/BEAM-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-507. Resolution: Fixed Fix Version/s: Not applicable > Fill in the documentation/runners/spark portion of

[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931859#comment-15931859 ] Amit Sela commented on BEAM-1582: - Moved tests that use checkpoint recovery to post-commit. Keeping open

[jira] [Resolved] (BEAM-1752) Tag Spark runner tests that recover from checkpoint.

[ https://issues.apache.org/jira/browse/BEAM-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1752. - Resolution: Fixed Fix Version/s: First stable release > Tag Spark runner tests that recover from

[jira] [Created] (BEAM-1752) Tag Spark runner tests that recover from checkpoint.

Amit Sela created BEAM-1752: --- Summary: Tag Spark runner tests that recover from checkpoint. Key: BEAM-1752 URL: https://issues.apache.org/jira/browse/BEAM-1752 Project: Beam Issue Type: Test

[jira] [Comment Edited] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895861#comment-15895861 ] Amit Sela edited comment on BEAM-1582 at 3/17/17 10:19 PM: --- Could be related to

[jira] [Comment Edited] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895861#comment-15895861 ] Amit Sela edited comment on BEAM-1582 at 3/17/17 10:14 PM: --- Could be related to

[jira] [Commented] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

[ https://issues.apache.org/jira/browse/BEAM-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924819#comment-15924819 ] Amit Sela commented on BEAM-1717: - Once this worked-out, I noticed a duplicate {{test}} jar in

[jira] [Commented] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

[ https://issues.apache.org/jira/browse/BEAM-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924812#comment-15924812 ] Amit Sela commented on BEAM-1717: - Seems that there is more than one issue here. First one is with

[jira] [Created] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

Amit Sela created BEAM-1717: --- Summary: Maven release/deploy tries to uploads some artifacts more than once Key: BEAM-1717 URL: https://issues.apache.org/jira/browse/BEAM-1717 Project: Beam Issue

[jira] [Reopened] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reopened BEAM-1582: - This test keeps flaking so I'll leave it open until we resolve it, or move to PostCommit and accept the fact

[jira] [Resolved] (BEAM-797) A PipelineVisitor that creates a Spark-native pipeline.

[ https://issues.apache.org/jira/browse/BEAM-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-797. Resolution: Fixed Fix Version/s: First stable release > A PipelineVisitor that creates a Spark-native

[jira] [Resolved] (BEAM-1562) Use a "signal" to stop streaming tests as they finish.

[ https://issues.apache.org/jira/browse/BEAM-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1562. - Resolution: Fixed Fix Version/s: First stable release > Use a "signal" to stop streaming tests as

[jira] [Resolved] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1582. - Resolution: Fixed Fix Version/s: First stable release > ResumeFromCheckpointStreamingTest flakes

[jira] [Resolved] (BEAM-1556) Spark executors need to register IO factories

[ https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1556. - Resolution: Fixed Fix Version/s: 0.6.0 > Spark executors need to register IO factories >

[jira] [Resolved] (BEAM-1623) Transform Reshuffle directly in Spark runner

[ https://issues.apache.org/jira/browse/BEAM-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1623. - Resolution: Fixed Fix Version/s: 0.6.0 > Transform Reshuffle directly in Spark runner >

[jira] [Resolved] (BEAM-1626) Remove caching of read MapWithStateDStream.

[ https://issues.apache.org/jira/browse/BEAM-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1626. - Resolution: Fixed Fix Version/s: 0.6.0 > Remove caching of read MapWithStateDStream. >

[jira] [Assigned] (BEAM-1556) Spark executors need to register IO factories

[ https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1556: --- Assignee: Amit Sela (was: Jean-Baptiste Onofré) > Spark executors need to register IO factories >

[jira] [Created] (BEAM-1626) Remove caching of read MapWithStateDStream.

Amit Sela created BEAM-1626: --- Summary: Remove caching of read MapWithStateDStream. Key: BEAM-1626 URL: https://issues.apache.org/jira/browse/BEAM-1626 Project: Beam Issue Type: Bug

[jira] [Resolved] (BEAM-1625) BoundedDataset action() does not materialize RDD

[ https://issues.apache.org/jira/browse/BEAM-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1625. - Resolution: Fixed Fix Version/s: 0.6.0 > BoundedDataset action() does not materialize RDD >

[jira] [Assigned] (BEAM-1608) Add support for Spark cluster metrics to PKB

[ https://issues.apache.org/jira/browse/BEAM-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1608: --- Assignee: (was: Amit Sela) > Add support for Spark cluster metrics to PKB >

[jira] [Commented] (BEAM-1602) Spark Runner support for PerfKit Benchmarker

[ https://issues.apache.org/jira/browse/BEAM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895863#comment-15895863 ] Amit Sela commented on BEAM-1602: - [~jasonkuster] what is Spark missing here ? running an IT against

[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895861#comment-15895861 ] Amit Sela commented on BEAM-1582: - Could be related to SPARK-16480 so that the last {{CheckpointMark}} is

[jira] [Created] (BEAM-1591) Implement Combine optimizations for GABW in streaming.

Amit Sela created BEAM-1591: --- Summary: Implement Combine optimizations for GABW in streaming. Key: BEAM-1591 URL: https://issues.apache.org/jira/browse/BEAM-1591 Project: Beam Issue Type:

[jira] [Updated] (BEAM-673) Data locality for Read.Bounded

[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-673: --- Description: In some distributed filesystems, such as HDFS, we should be able to hint to Spark the preferred

[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[ https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891188#comment-15891188 ] Amit Sela commented on BEAM-1582: - Looks like the flake happens when the entire input is re-read. We inject

[jira] [Created] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

Amit Sela created BEAM-1582: --- Summary: ResumeFromCheckpointStreamingTest flakes with what appears as a second firing. Key: BEAM-1582 URL: https://issues.apache.org/jira/browse/BEAM-1582 Project: Beam

[jira] [Resolved] (BEAM-111) Use SDK implementation of WritableCoder

[ https://issues.apache.org/jira/browse/BEAM-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-111. Resolution: Fixed Fix Version/s: 0.6.0 > Use SDK implementation of WritableCoder >

[jira] [Commented] (BEAM-849) Redesign PipelineResult API

[ https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890953#comment-15890953 ] Amit Sela commented on BEAM-849: A continually growing file is something else, which I agree on, but in that

[jira] [Comment Edited] (BEAM-849) Redesign PipelineResult API

[ https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890885#comment-15890885 ] Amit Sela edited comment on BEAM-849 at 3/1/17 7:35 PM: I disagree with stating that

[jira] [Commented] (BEAM-849) Redesign PipelineResult API

[ https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890885#comment-15890885 ] Amit Sela commented on BEAM-849: I disagree with stating that "Create.of(filename) + ParDo(tail file) +

[jira] [Updated] (BEAM-1562) Use a "signal" to stop streaming tests as they finish.

[ https://issues.apache.org/jira/browse/BEAM-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1562: Description: Streaming tests use a timeout that has to take a large enough buffer to avoid slow runtimes

[jira] [Updated] (BEAM-1562) Use a "signal" to stop streaming tests as they finish.

[ https://issues.apache.org/jira/browse/BEAM-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-1562: Summary: Use a "signal" to stop streaming tests as they finish. (was: Use a "poison pill" to stop streaming

[jira] [Assigned] (BEAM-1562) Use a "poison pill" to stop streaming tests as they finish.

[ https://issues.apache.org/jira/browse/BEAM-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1562: --- Assignee: Amit Sela > Use a "poison pill" to stop streaming tests as they finish. >

[jira] [Resolved] (BEAM-1576) Use UnsupportedSideInputReader in GroupAlsoByWindowEvaluatorFactory.

[ https://issues.apache.org/jira/browse/BEAM-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1576. - Resolution: Fixed Fix Version/s: 0.6.0 > Use UnsupportedSideInputReader in

[jira] [Created] (BEAM-1576) Use UnsupportedSideInputReader in GroupAlsoByWindowEvaluatorFactory.

Amit Sela created BEAM-1576: --- Summary: Use UnsupportedSideInputReader in GroupAlsoByWindowEvaluatorFactory. Key: BEAM-1576 URL: https://issues.apache.org/jira/browse/BEAM-1576 Project: Beam Issue

[jira] [Resolved] (BEAM-920) Support triggers, panes and watermarks.

[ https://issues.apache.org/jira/browse/BEAM-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-920. Resolution: Fixed Fix Version/s: 0.6.0 > Support triggers, panes and watermarks. >

[jira] [Commented] (BEAM-1556) Spark executors need to register IO factories

[ https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888512#comment-15888512 ] Amit Sela commented on BEAM-1556: - Agree, unblock users first. [~jbonofre] I can help with this if you

[jira] [Commented] (BEAM-1556) Spark executors need to register IO factories

[ https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887964#comment-15887964 ] Amit Sela commented on BEAM-1556: - My line of thought about this being in the SDK (or better, the Runner

[jira] [Created] (BEAM-1562) Use a "poison pill" to stop streaming tests as they finish.

Amit Sela created BEAM-1562: --- Summary: Use a "poison pill" to stop streaming tests as they finish. Key: BEAM-1562 URL: https://issues.apache.org/jira/browse/BEAM-1562 Project: Beam Issue Type:

[jira] [Created] (BEAM-1564) Support streaming side-inputs in the Spark runner.

Amit Sela created BEAM-1564: --- Summary: Support streaming side-inputs in the Spark runner. Key: BEAM-1564 URL: https://issues.apache.org/jira/browse/BEAM-1564 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-1563) Flatten Spark runner libraries.

Amit Sela created BEAM-1563: --- Summary: Flatten Spark runner libraries. Key: BEAM-1563 URL: https://issues.apache.org/jira/browse/BEAM-1563 Project: Beam Issue Type: Improvement

[jira] [Commented] (BEAM-1556) Spark executors need to register IO factories

[ https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884602#comment-15884602 ] Amit Sela commented on BEAM-1556: - [~frances] I wonder if this is an issue with other runners as well..

[jira] [Commented] (BEAM-982) Document spark configuration update

[ https://issues.apache.org/jira/browse/BEAM-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883673#comment-15883673 ] Amit Sela commented on BEAM-982: [~jbonofre] {{spark.serializer}} is already hard-coded into the runner, and

[jira] [Closed] (BEAM-229) Add support for additional Spark configuration via PipelineOptions

[ https://issues.apache.org/jira/browse/BEAM-229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-229. -- Resolution: Won't Fix Fix Version/s: Not applicable Spark configurations can be passed as system

[jira] [Closed] (BEAM-18) Add support for new Beam Sink API

[ https://issues.apache.org/jira/browse/BEAM-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-18. - Resolution: Won't Fix Fix Version/s: Not applicable Currently, Beam Sinks are implemented via {{ParDo}}s and

[jira] [Closed] (BEAM-668) Reinstate runner direct translation for TextIO and AvroIO once Beam SDK supports hdfs.

[ https://issues.apache.org/jira/browse/BEAM-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-668. -- Resolution: Won't Fix Fix Version/s: Not applicable It doesn't seem maintainable to directly translate IOs

[jira] [Updated] (BEAM-848) A better shuffle after reading from within mapWithState.

[ https://issues.apache.org/jira/browse/BEAM-848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-848: --- Issue Type: Improvement (was: Bug) > A better shuffle after reading from within mapWithState. >

[jira] [Resolved] (BEAM-1526) Flakes in Spark runner WatermarkTest.testInDoFn

[ https://issues.apache.org/jira/browse/BEAM-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1526. - Resolution: Fixed Fix Version/s: 0.6.0 > Flakes in Spark runner WatermarkTest.testInDoFn >

[jira] [Resolved] (BEAM-1512) Optimize leaf transforms materialization

[ https://issues.apache.org/jira/browse/BEAM-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1512. - Resolution: Fixed Fix Version/s: 0.6.0 > Optimize leaf transforms materialization >

[jira] [Commented] (BEAM-1492) Avoid potential issue in ASM 5.0

[ https://issues.apache.org/jira/browse/BEAM-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872065#comment-15872065 ] Amit Sela commented on BEAM-1492: - [~kenn] Spark and Apex runners both use Kryo 2.x which transitively

[jira] [Resolved] (BEAM-774) Implement Metrics support for Spark runner

[ https://issues.apache.org/jira/browse/BEAM-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-774. Resolution: Fixed Fix Version/s: 0.6.0 > Implement Metrics support for Spark runner >

[jira] [Created] (BEAM-1470) A quite logger for testing

Amit Sela created BEAM-1470: --- Summary: A quite logger for testing Key: BEAM-1470 URL: https://issues.apache.org/jira/browse/BEAM-1470 Project: Beam Issue Type: Improvement Components:

[jira] [Resolved] (BEAM-1437) Spark runner StreamingListeners are not recoverable.

[ https://issues.apache.org/jira/browse/BEAM-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1437. - Resolution: Fixed Fix Version/s: 0.6.0 > Spark runner StreamingListeners are not recoverable. >

[jira] [Created] (BEAM-1444) Flatten of Bounded and Unbounded repeats the union with the RDD for each micro-batch.

Amit Sela created BEAM-1444: --- Summary: Flatten of Bounded and Unbounded repeats the union with the RDD for each micro-batch. Key: BEAM-1444 URL: https://issues.apache.org/jira/browse/BEAM-1444 Project:

[jira] [Created] (BEAM-1437) Spark runner StreamingListeners are not recoverable.

Amit Sela created BEAM-1437: --- Summary: Spark runner StreamingListeners are not recoverable. Key: BEAM-1437 URL: https://issues.apache.org/jira/browse/BEAM-1437 Project: Beam Issue Type: Bug

[jira] [Assigned] (BEAM-1437) Spark runner StreamingListeners are not recoverable.

[ https://issues.apache.org/jira/browse/BEAM-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1437: --- Assignee: (was: Amit Sela) > Spark runner StreamingListeners are not recoverable. >

[jira] [Resolved] (BEAM-1435) PostCommit builds are broken by NoClassDefFoundError in runners

[ https://issues.apache.org/jira/browse/BEAM-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1435. - Resolution: Duplicate Fix Version/s: Not applicable > PostCommit builds are broken by

[jira] [Created] (BEAM-1435) PostCommit builds are broken by NoClassDefFoundError in runners

Amit Sela created BEAM-1435: --- Summary: PostCommit builds are broken by NoClassDefFoundError in runners Key: BEAM-1435 URL: https://issues.apache.org/jira/browse/BEAM-1435 Project: Beam Issue

[jira] [Resolved] (BEAM-1395) SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp

[ https://issues.apache.org/jira/browse/BEAM-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela resolved BEAM-1395. - Resolution: Fixed Fix Version/s: 0.6.0 > SparkGroupAlsoByWindowFn not sorting grouped elements by

[jira] [Closed] (BEAM-1403) Weird excpetion about watermarks in Batch procesing mode

[ https://issues.apache.org/jira/browse/BEAM-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-1403. --- Resolution: Duplicate Fix Version/s: Not applicable > Weird excpetion about watermarks in Batch

[jira] [Assigned] (BEAM-1403) Weird excpetion about watermarks in Batch procesing mode

[ https://issues.apache.org/jira/browse/BEAM-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela reassigned BEAM-1403: --- Assignee: Amit Sela (was: Kenneth Knowles) > Weird excpetion about watermarks in Batch procesing

[jira] [Commented] (BEAM-1396) GABWVOBDoFn expects grouped values to be ordered by their timestamp but there is no such guarantee

[ https://issues.apache.org/jira/browse/BEAM-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853642#comment-15853642 ] Amit Sela commented on BEAM-1396: - It was always called {{SparkGroupAlsoByWindow}}, and it actually

[jira] [Created] (BEAM-1396) GABWVOBDoFn expects grouped values to be ordered by their timestamp but there is no such guarantee

Amit Sela created BEAM-1396: --- Summary: GABWVOBDoFn expects grouped values to be ordered by their timestamp but there is no such guarantee Key: BEAM-1396 URL: https://issues.apache.org/jira/browse/BEAM-1396

[jira] [Created] (BEAM-1395) SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp

Amit Sela created BEAM-1395: --- Summary: SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp Key: BEAM-1395 URL: https://issues.apache.org/jira/browse/BEAM-1395 Project: Beam Issue

[jira] [Created] (BEAM-1304) NPE if trying to get value of Aggregator that does not exist.

Amit Sela created BEAM-1304: --- Summary: NPE if trying to get value of Aggregator that does not exist. Key: BEAM-1304 URL: https://issues.apache.org/jira/browse/BEAM-1304 Project: Beam Issue Type:

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836555#comment-15836555 ] Amit Sela commented on BEAM-302: Oh, got it, thanks! > Add Scio Scala DSL to Beam >

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[ https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836551#comment-15836551 ] Amit Sela commented on BEAM-302: You mean 0.5.0 ? > Add Scio Scala DSL to Beam > --

  1   2   >