[jira] [Commented] (BEAM-68) Support for limiting parallelism of a step

2017-03-09 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903135#comment-15903135 ] Eugene Kirpichov commented on BEAM-68: -- I think this needs a discussion on the Beam dev@ mailing list.

[jira] [Assigned] (BEAM-1422) ParDo should comply with PTransform style guide

2017-03-03 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1422: -- Assignee: Eugene Kirpichov (was: Davor Bonaci) > ParDo should comply with PTransform

[jira] [Commented] (BEAM-1678) Create MemcachedIO

2017-03-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905236#comment-15905236 ] Eugene Kirpichov commented on BEAM-1678: What would this transform / family of transforms do? >

[jira] [Commented] (BEAM-1679) create gRPC IO

2017-03-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905227#comment-15905227 ] Eugene Kirpichov commented on BEAM-1679: Discussion currently happening on the mailing list:

[jira] [Commented] (BEAM-1581) JsonIO

2017-03-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905240#comment-15905240 ] Eugene Kirpichov commented on BEAM-1581: I would recommend to preserve the decomposition and make

[jira] [Commented] (BEAM-1581) JsonIO

2017-03-12 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906648#comment-15906648 ] Eugene Kirpichov commented on BEAM-1581: [~aviemzur] - I think XML source/sink are a bad design in

[jira] [Commented] (BEAM-1678) Create MemcachedIO

2017-03-10 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905722#comment-15905722 ] Eugene Kirpichov commented on BEAM-1678: Note that all IO's are families of PTransforms (usually

[jira] [Created] (BEAM-1712) TestPipeline.run doesn't actually waitUntilFinish

2017-03-13 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1712: -- Summary: TestPipeline.run doesn't actually waitUntilFinish Key: BEAM-1712 URL: https://issues.apache.org/jira/browse/BEAM-1712 Project: Beam Issue Type:

[jira] [Commented] (BEAM-1712) TestPipeline.run doesn't actually waitUntilFinish

2017-03-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923329#comment-15923329 ] Eugene Kirpichov commented on BEAM-1712: OK, seems like that is in fact what happens. So this bug

[jira] [Updated] (BEAM-1712) TestPipeline.run doesn't actually waitUntilFinish

2017-03-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1712: --- Priority: Major (was: Blocker) > TestPipeline.run doesn't actually waitUntilFinish >

[jira] [Commented] (BEAM-1712) TestPipeline.run doesn't actually waitUntilFinish

2017-03-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923326#comment-15923326 ] Eugene Kirpichov commented on BEAM-1712: An optimistic interpretation is probably that Flink runner

[jira] [Commented] (BEAM-1712) TestPipeline.run doesn't actually waitUntilFinish

2017-03-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923351#comment-15923351 ] Eugene Kirpichov commented on BEAM-1712: As an intermediate step to unifying everything under

[jira] [Commented] (BEAM-1712) TestPipeline.run doesn't actually waitUntilFinish

2017-03-13 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923321#comment-15923321 ] Eugene Kirpichov commented on BEAM-1712:

[jira] [Commented] (BEAM-1573) KafkaIO does not allow using Kafka serializers and deserializers

2017-03-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1589#comment-1589 ] Eugene Kirpichov commented on BEAM-1573: Yes, this would be a backward-incompatible change. We are

[jira] [Commented] (BEAM-1573) KafkaIO does not allow using Kafka serializers and deserializers

2017-03-06 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896989#comment-15896989 ] Eugene Kirpichov commented on BEAM-1573: At a conference right now, but quick comment: yes, as

[jira] [Closed] (BEAM-1416) Write transform should comply with PTransform style guide

2017-02-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1416. -- > Write transform should comply with PTransform style guide >

[jira] [Closed] (BEAM-1426) SortValues should comply with PTransform style guide

2017-02-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1426. -- > SortValues should comply with PTransform style guide >

[jira] [Commented] (BEAM-1424) ToString should comply with PTransform style guide

2017-02-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889586#comment-15889586 ] Eugene Kirpichov commented on BEAM-1424: Actual backward-incompatible changes: kv renamed to kvs,

[jira] [Closed] (BEAM-1424) ToString should comply with PTransform style guide

2017-02-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1424. -- > ToString should comply with PTransform style guide >

[jira] [Commented] (BEAM-849) Redesign PipelineResult API

2017-03-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890855#comment-15890855 ] Eugene Kirpichov commented on BEAM-849: --- I disagree that "unbounded pipelines" can't finish

[jira] [Commented] (BEAM-1562) Use a "signal" to stop streaming tests as they finish.

2017-03-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890861#comment-15890861 ] Eugene Kirpichov commented on BEAM-1562: See my comment on BEAM-849: as an alternative, we can use

[jira] [Created] (BEAM-1578) Runners should put PT overrides into a list rather than map

2017-02-28 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1578: -- Summary: Runners should put PT overrides into a list rather than map Key: BEAM-1578 URL: https://issues.apache.org/jira/browse/BEAM-1578 Project: Beam

[jira] [Created] (BEAM-1594) JOB_STATE_DRAINED should be terminal for waitUntilFinish()

2017-03-02 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1594: -- Summary: JOB_STATE_DRAINED should be terminal for waitUntilFinish() Key: BEAM-1594 URL: https://issues.apache.org/jira/browse/BEAM-1594 Project: Beam

[jira] [Assigned] (BEAM-1427) BigQueryIO should comply with PTransform style guide

2017-03-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1427: -- Assignee: Eugene Kirpichov > BigQueryIO should comply with PTransform style guide >

[jira] [Commented] (BEAM-1915) Remove OldDoFn dependency in ApexGroupByKeyOperator

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961586#comment-15961586 ] Eugene Kirpichov commented on BEAM-1915: It happens to also be the last occurrence in the Beam

[jira] [Commented] (BEAM-1915) Remove OldDoFn dependency in ApexGroupByKeyOperator

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961587#comment-15961587 ] Eugene Kirpichov commented on BEAM-1915: cc: [~kenn] > Remove OldDoFn dependency in

[jira] [Commented] (BEAM-1910) test_using_slow_impl very flaky locally

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961590#comment-15961590 ] Eugene Kirpichov commented on BEAM-1910: Got the error again.

[jira] [Commented] (BEAM-1910) test_using_slow_impl very flaky locally

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961592#comment-15961592 ] Eugene Kirpichov commented on BEAM-1910: I guess I also would like a way to run mvn verify but skip

[jira] [Closed] (BEAM-1903) Splittable DoFn should report watermarks via ProcessContext

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1903. -- Resolution: Fixed Fix Version/s: First stable release > Splittable DoFn should report

[jira] [Closed] (BEAM-1904) Remove DoFn.ProcessContinuation

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1904. -- Resolution: Fixed Fix Version/s: First stable release This is effectively done in

[jira] [Created] (BEAM-1913) TFRecordIO should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1913: -- Summary: TFRecordIO should comply with PTransform style guide Key: BEAM-1913 URL: https://issues.apache.org/jira/browse/BEAM-1913 Project: Beam Issue

[jira] [Assigned] (BEAM-1428) KinesisIO should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1428: -- Assignee: (was: Davor Bonaci) > KinesisIO should comply with PTransform style guide

[jira] [Commented] (BEAM-644) Primitive to shift the watermark while assigning timestamps

2017-04-05 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957959#comment-15957959 ] Eugene Kirpichov commented on BEAM-644: --- I don't think there's an SDF-specific issue here: it applies

[jira] [Created] (BEAM-1910) test_using_slow_impl very flaky locally

2017-04-07 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1910: -- Summary: test_using_slow_impl very flaky locally Key: BEAM-1910 URL: https://issues.apache.org/jira/browse/BEAM-1910 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-1581) JSON source and sink

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961285#comment-15961285 ] Eugene Kirpichov commented on BEAM-1581: Yeah, JacksonIO sounds like good idea. It also has fewer

[jira] [Updated] (BEAM-1913) TFRecordIO should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1913: --- Labels: backward-incompatible starter (was: backwards-incompatible starter) > TFRecordIO

[jira] [Assigned] (BEAM-1914) XML IO should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1914: -- Assignee: (was: Davor Bonaci) > XML IO should comply with PTransform style guide >

[jira] [Updated] (BEAM-1914) XML IO should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1914: --- Labels: backward-incompatible starter (was: ) > XML IO should comply with PTransform style

[jira] [Updated] (BEAM-1414) CountingInput should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-1414: --- Labels: backward-incompatible starter (was: backward-incompatible) > CountingInput should

[jira] [Assigned] (BEAM-1414) CountingInput should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1414: -- Assignee: (was: Davor Bonaci) > CountingInput should comply with PTransform style

[jira] [Comment Edited] (BEAM-447) Stop referring to types with Bound/Unbound

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947688#comment-15947688 ] Eugene Kirpichov edited comment on BEAM-447 at 4/7/17 9:46 PM: --- Remaining

[jira] [Created] (BEAM-1914) XML IO should comply with PTransform style guide

2017-04-07 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1914: -- Summary: XML IO should comply with PTransform style guide Key: BEAM-1914 URL: https://issues.apache.org/jira/browse/BEAM-1914 Project: Beam Issue Type:

[jira] [Closed] (BEAM-447) Stop referring to types with Bound/Unbound

2017-04-07 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-447. - Resolution: Duplicate I've filed separate JIRAs for the remaining sub-items. > Stop referring to

[jira] [Created] (BEAM-1979) Outputting to an undeclared side output TupleTag should throw

2017-04-14 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-1979: -- Summary: Outputting to an undeclared side output TupleTag should throw Key: BEAM-1979 URL: https://issues.apache.org/jira/browse/BEAM-1979 Project: Beam

[jira] [Assigned] (BEAM-1913) TFRecordIO should comply with PTransform style guide

2017-04-17 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1913: -- Assignee: Eugene Kirpichov > TFRecordIO should comply with PTransform style guide >

[jira] [Assigned] (BEAM-1914) XML IO should comply with PTransform style guide

2017-04-17 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-1914: -- Assignee: Eugene Kirpichov > XML IO should comply with PTransform style guide >

[jira] [Closed] (BEAM-1913) TFRecordIO should comply with PTransform style guide

2017-04-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1913. -- Resolution: Fixed Fix Version/s: First stable release > TFRecordIO should comply with

[jira] [Commented] (BEAM-1355) HDFS IO should comply with PTransform style guide

2017-04-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973593#comment-15973593 ] Eugene Kirpichov commented on BEAM-1355: Stephen - do you think then, perhaps, that we should

[jira] [Commented] (BEAM-1910) test_using_slow_impl very flaky locally

2017-04-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973669#comment-15973669 ] Eugene Kirpichov commented on BEAM-1910: Vikas, any updates? This is heavily interfering with the

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106185#comment-16106185 ] Eugene Kirpichov commented on BEAM-2671: It's overwhelmingly likely the last one as I mentioned

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-28 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105124#comment-16105124 ] Eugene Kirpichov commented on BEAM-2671: I'm not sure about bumping it to 2.2.0. It looks quite

[jira] [Commented] (BEAM-1190) FileBasedSource should ignore files that matched the glob but don't exist

2017-08-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111508#comment-16111508 ] Eugene Kirpichov commented on BEAM-1190: GCS object listing is strongly consistent now. However the

[jira] [Created] (BEAM-2716) AvroReader should refuse dynamic splits while in the last block

2017-08-02 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2716: -- Summary: AvroReader should refuse dynamic splits while in the last block Key: BEAM-2716 URL: https://issues.apache.org/jira/browse/BEAM-2716 Project: Beam

[jira] [Commented] (BEAM-2716) AvroReader should refuse dynamic splits while in the last block

2017-08-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111671#comment-16111671 ] Eugene Kirpichov commented on BEAM-2716: I think this affects only Avro, other file-based formats

[jira] [Closed] (BEAM-2670) ParDoTest.testPipelineOptionsParameter* validatesRunner tests fail on Spark runner

2017-08-02 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2670. -- Resolution: Fixed > ParDoTest.testPipelineOptionsParameter* validatesRunner tests fail on Spark

[jira] [Created] (BEAM-2712) SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions.

2017-08-01 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2712: -- Summary: SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions. Key: BEAM-2712 URL: https://issues.apache.org/jira/browse/BEAM-2712

[jira] [Assigned] (BEAM-2712) SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions.

2017-08-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2712: -- Assignee: (was: Kenneth Knowles) > SerializablePipelineOptions should not call >

[jira] [Created] (BEAM-2706) Create JdbcIO.readAll()

2017-08-01 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2706: -- Summary: Create JdbcIO.readAll() Key: BEAM-2706 URL: https://issues.apache.org/jira/browse/BEAM-2706 Project: Beam Issue Type: Bug Components:

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

2017-08-01 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109644#comment-16109644 ] Eugene Kirpichov commented on BEAM-1323: Filed https://issues.apache.org/jira/browse/BEAM-2706 for

[jira] [Commented] (BEAM-2684) AmqpIOTest is flaky

2017-08-04 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114002#comment-16114002 ] Eugene Kirpichov commented on BEAM-2684: As an intermediate step, perhaps, add a bunch of logging

[jira] [Assigned] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

2017-08-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2768: -- Assignee: Reuven Lax (was: Kenneth Knowles) > Fix bigquery.WriteTables generating

[jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

2017-08-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127687#comment-16127687 ] Eugene Kirpichov commented on BEAM-2768: Could you tell more about how you're using

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-08-14 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126416#comment-16126416 ] Eugene Kirpichov commented on BEAM-2140: Sorry for the delayed response. The output watermark

[jira] [Closed] (BEAM-2700) BigQueryIO should support using file load jobs when using unbounded collections

2017-08-14 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2700. -- Resolution: Fixed Fix Version/s: 2.2.0 > BigQueryIO should support using file load jobs

[jira] [Closed] (BEAM-2447) Reintroduce DoFn.ProcessContinuation

2017-07-12 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2447. -- Resolution: Fixed Fix Version/s: 2.2.0 > Reintroduce DoFn.ProcessContinuation >

[jira] [Created] (BEAM-2623) Create Watch transform for emitting new elements in a family of growing sets

2017-07-14 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2623: -- Summary: Create Watch transform for emitting new elements in a family of growing sets Key: BEAM-2623 URL: https://issues.apache.org/jira/browse/BEAM-2623

[jira] [Closed] (BEAM-2406) NullPointerException when writing an empty table to BigQuery

2017-07-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2406. -- Resolution: Fixed Fix Version/s: 2.1.0 > NullPointerException when writing an empty table

[jira] [Created] (BEAM-2628) AvroSource.split() sequentially opens every matched file

2017-07-17 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2628: -- Summary: AvroSource.split() sequentially opens every matched file Key: BEAM-2628 URL: https://issues.apache.org/jira/browse/BEAM-2628 Project: Beam

[jira] [Updated] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

2017-07-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov updated BEAM-2640: --- Issue Type: New Feature (was: Bug) > Introduce Create.ofProvider(ValueProvider) >

[jira] [Created] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

2017-07-19 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2640: -- Summary: Introduce Create.ofProvider(ValueProvider) Key: BEAM-2640 URL: https://issues.apache.org/jira/browse/BEAM-2640 Project: Beam Issue Type: Bug

[jira] [Closed] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

2017-07-18 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2532. -- Resolution: Fixed Assignee: Neville Li (was: Chamikara Jayalath) Fix Version/s:

[jira] [Closed] (BEAM-2544) AvroIOTest is flaky

2017-07-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2544. -- Resolution: Fixed Fix Version/s: 2.2.0 > AvroIOTest is flaky > --- > >

[jira] [Closed] (BEAM-2628) AvroSource.split() sequentially opens every matched file

2017-07-20 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2628. -- Resolution: Fixed Fix Version/s: 2.2.0 > AvroSource.split() sequentially opens every

[jira] [Closed] (BEAM-2608) Unclosed BoundedReader in TextIO#ReadTextFn#process()

2017-07-15 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2608. -- Resolution: Fixed Fix Version/s: 2.2.0 > Unclosed BoundedReader in

[jira] [Created] (BEAM-2656) Introduce AvroIO.readAll()

2017-07-21 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2656: -- Summary: Introduce AvroIO.readAll() Key: BEAM-2656 URL: https://issues.apache.org/jira/browse/BEAM-2656 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2680) Improve scalability of the Watch transform

2017-07-25 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2680: -- Summary: Improve scalability of the Watch transform Key: BEAM-2680 URL: https://issues.apache.org/jira/browse/BEAM-2680 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2682) Merge AvroIOTest and AvroIOTransformTest

2017-07-25 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2682: -- Summary: Merge AvroIOTest and AvroIOTransformTest Key: BEAM-2682 URL: https://issues.apache.org/jira/browse/BEAM-2682 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2677) AvroIO.read without specifying a schema

2017-07-24 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2677: -- Summary: AvroIO.read without specifying a schema Key: BEAM-2677 URL: https://issues.apache.org/jira/browse/BEAM-2677 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103911#comment-16103911 ] Eugene Kirpichov commented on BEAM-2671: Actually that test is

[jira] [Assigned] (BEAM-2670) ParDoTest.testPipelineOptionsParameter* validatesRunner tests fail on Spark runner

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov reassigned BEAM-2670: -- Assignee: Eugene Kirpichov (was: Stas Levin) > ParDoTest.testPipelineOptionsParameter*

[jira] [Commented] (BEAM-2670) ParDoTest.testPipelineOptionsParameter* validatesRunner tests fail on Spark runner

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103815#comment-16103815 ] Eugene Kirpichov commented on BEAM-2670: I found the issue and have a fix. >

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103868#comment-16103868 ] Eugene Kirpichov commented on BEAM-2671: CreateStreamTest.testMultiOutputParDo() is failing as

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

2017-07-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104459#comment-16104459 ] Eugene Kirpichov commented on BEAM-2671: I wonder if these tests have something to do with

[jira] [Created] (BEAM-2684) AmqpIOTest is flaky

2017-07-26 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2684: -- Summary: AmqpIOTest is flaky Key: BEAM-2684 URL: https://issues.apache.org/jira/browse/BEAM-2684 Project: Beam Issue Type: Bug Components:

[jira] [Closed] (BEAM-1820) Source.getDefaultOutputCoder() should be @Nullable

2017-07-26 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-1820. -- Resolution: Won't Fix Fix Version/s: Not applicable The consensus on dev@ is that

[jira] [Created] (BEAM-2686) PTransform's should always set a Coder on their output PCollections

2017-07-26 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2686: -- Summary: PTransform's should always set a Coder on their output PCollections Key: BEAM-2686 URL: https://issues.apache.org/jira/browse/BEAM-2686 Project: Beam

[jira] [Commented] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

2017-07-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093588#comment-16093588 ] Eugene Kirpichov commented on BEAM-2640: Coder inference for this is complicated by the fact that

[jira] [Commented] (BEAM-2643) Add TextIO.read_all() to Python SDK

2017-07-19 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093633#comment-16093633 ] Eugene Kirpichov commented on BEAM-2643: See also https://github.com/apache/beam/pull/3598 > Add

[jira] [Created] (BEAM-2641) Improve discoverability of TextIO.readAll() as a replacement of TextIO.read() for large globs

2017-07-19 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2641: -- Summary: Improve discoverability of TextIO.readAll() as a replacement of TextIO.read() for large globs Key: BEAM-2641 URL: https://issues.apache.org/jira/browse/BEAM-2641

[jira] [Created] (BEAM-2607) Enforce that SDF must return stop() after a failed tryClaim() call

2017-07-12 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2607: -- Summary: Enforce that SDF must return stop() after a failed tryClaim() call Key: BEAM-2607 URL: https://issues.apache.org/jira/browse/BEAM-2607 Project: Beam

[jira] [Closed] (BEAM-2511) TextIO should support reading a PCollection of filenames

2017-07-11 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-2511. -- Resolution: Fixed Fix Version/s: 2.2.0 > TextIO should support reading a PCollection of

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068782#comment-16068782 ] Eugene Kirpichov commented on BEAM-2140: Conceptually, watermarks are for PCollections - lower

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-29 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068740#comment-16068740 ] Eugene Kirpichov commented on BEAM-2140: So, to elaborate on what Kenn said. We dug a bit deeper

[jira] [Created] (BEAM-2536) Simplify specifying coders on PCollectionTuple

2017-06-28 Thread Eugene Kirpichov (JIRA)
Eugene Kirpichov created BEAM-2536: -- Summary: Simplify specifying coders on PCollectionTuple Key: BEAM-2536 URL: https://issues.apache.org/jira/browse/BEAM-2536 Project: Beam Issue Type:

[jira] [Commented] (BEAM-2532) BigQueryIO source should avoid expensive JSON schema parsing for every record

2017-06-30 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070428#comment-16070428 ] Eugene Kirpichov commented on BEAM-2532: [~reuvenlax] I think you had some thoughts about how to

[jira] [Comment Edited] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065429#comment-16065429 ] Eugene Kirpichov edited comment on BEAM-2140 at 6/27/17 8:38 PM: - Okay, I

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065429#comment-16065429 ] Eugene Kirpichov commented on BEAM-2140: Okay, I see that I misunderstood what the watermark hold

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065626#comment-16065626 ] Eugene Kirpichov commented on BEAM-2140: _we can't advance the watermark as though it was

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

2017-06-27 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065627#comment-16065627 ] Eugene Kirpichov commented on BEAM-2140: ...Or is the problem that the watermark of the output

[jira] [Commented] (BEAM-673) Data locality for Read.Bounded

2017-04-25 Thread Eugene Kirpichov (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983950#comment-15983950 ] Eugene Kirpichov commented on BEAM-673: --- On second thought, this might be related to SDF: processing

<    1   2   3   4   5   6   7   >