[jira] [Created] (BEAM-1438) The default behavior for the Write transform doesn't work well with the Dataflow streaming runner

Reuven Lax created BEAM-1438: Summary: The default behavior for the Write transform doesn't work well with the Dataflow streaming runner Key: BEAM-1438 URL: https://issues.apache.org/jira/browse/BEAM-1438

[jira] [Created] (BEAM-1402) Make TextIO and AvroIO use best-practice types.

Reuven Lax created BEAM-1402: Summary: Make TextIO and AvroIO use best-practice types. Key: BEAM-1402 URL: https://issues.apache.org/jira/browse/BEAM-1402 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-1401) Sinks in Beam should supported windowed unbounded PCollections

Reuven Lax created BEAM-1401: Summary: Sinks in Beam should supported windowed unbounded PCollections Key: BEAM-1401 URL: https://issues.apache.org/jira/browse/BEAM-1401 Project: Beam Issue

[jira] [Created] (BEAM-1530) BigQueryIO should support value-dependent windows

Reuven Lax created BEAM-1530: Summary: BigQueryIO should support value-dependent windows Key: BEAM-1530 URL: https://issues.apache.org/jira/browse/BEAM-1530 Project: Beam Issue Type: Improvement

[jira] [Created] (BEAM-1750) Refactor BigQueryIO.java into multiple files

Reuven Lax created BEAM-1750: Summary: Refactor BigQueryIO.java into multiple files Key: BEAM-1750 URL: https://issues.apache.org/jira/browse/BEAM-1750 Project: Beam Issue Type: Improvement

[jira] [Created] (BEAM-1873) Javadoc in BigQueryIO doesn't reflect recent changes

Reuven Lax created BEAM-1873: Summary: Javadoc in BigQueryIO doesn't reflect recent changes Key: BEAM-1873 URL: https://issues.apache.org/jira/browse/BEAM-1873 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-1897) Remove Sink

Reuven Lax created BEAM-1897: Summary: Remove Sink Key: BEAM-1897 URL: https://issues.apache.org/jira/browse/BEAM-1897 Project: Beam Issue Type: Bug Components: sdk-java-core

[jira] [Created] (BEAM-2058) BigQuery load job id should be generated at run time, not submission time

Reuven Lax created BEAM-2058: Summary: BigQuery load job id should be generated at run time, not submission time Key: BEAM-2058 URL: https://issues.apache.org/jira/browse/BEAM-2058 Project: Beam

[jira] [Updated] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn

[ https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-1831: - This is not double checked locking - the extra check is only there as a performance optimization. The data

[jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127924#comment-16127924 ] Reuven Lax commented on BEAM-2768: -- The load job for a specific table is the UUID (which is generated when

[jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128378#comment-16128378 ] Reuven Lax commented on BEAM-2768: -- Matti, it would help if you could explain exactly what failures you

[jira] [Created] (BEAM-2772) BigQueryIO doesn't properly set job id when using triggered file loads

Reuven Lax created BEAM-2772: Summary: BigQueryIO doesn't properly set job id when using triggered file loads Key: BEAM-2772 URL: https://issues.apache.org/jira/browse/BEAM-2772 Project: Beam

[jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128321#comment-16128321 ] Reuven Lax commented on BEAM-2768: -- Sorry - this pull request was intended for a different JIRA (it fixes

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

[ https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129000#comment-16129000 ] Reuven Lax commented on BEAM-2671: -- Given how long 2.1.0 has taken to release, I suspect we'll cut 2.2.0

[jira] [Created] (BEAM-2601) FileBasedSink produces incorrect shards when writing to multiple destinations

Reuven Lax created BEAM-2601: Summary: FileBasedSink produces incorrect shards when writing to multiple destinations Key: BEAM-2601 URL: https://issues.apache.org/jira/browse/BEAM-2601 Project: Beam

[jira] [Created] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

Reuven Lax created BEAM-2624: Summary: File-based sinks should produce a PCollection of written filenames Key: BEAM-2624 URL: https://issues.apache.org/jira/browse/BEAM-2624 Project: Beam Issue

[jira] [Updated] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[ https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2624: - Component/s: (was: sdk-java-gcp) > File-based sinks should produce a PCollection of written filenames

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[ https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072945#comment-16072945 ] Reuven Lax commented on BEAM-2353: -- n/m - just saw this is bumped to 2.2. In that case all of these

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[ https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072582#comment-16072582 ] Reuven Lax commented on BEAM-2353: -- In that case we should take https://github.com/apache/beam/pull/3356

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[ https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072737#comment-16072737 ] Reuven Lax commented on BEAM-2353: -- +JB > FileNamePolicy context parameters allow backwards

[jira] [Commented] (BEAM-1458) Checkpoint support in Beam

[ https://issues.apache.org/jira/browse/BEAM-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1603#comment-1603 ] Reuven Lax commented on BEAM-1458: -- That was my plan. On Thu, Jun 22, 2017 at 8:22 PM, Kenneth Knowles

[jira] [Created] (BEAM-2052) Windowed file sinks should support dynamic sharding

Reuven Lax created BEAM-2052: Summary: Windowed file sinks should support dynamic sharding Key: BEAM-2052 URL: https://issues.apache.org/jira/browse/BEAM-2052 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-2108) Integration tests for PubsubIO

[ https://issues.apache.org/jira/browse/BEAM-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2108: - Correct. We have some basic ones for Dataflow that we should be able to port to Beam. On Thu, Apr 27, 2017 at

[jira] [Commented] (BEAM-1831) Checking of containment in createdTables may have race condition in StreamingWriteFn

[ https://issues.apache.org/jira/browse/BEAM-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989139#comment-15989139 ] Reuven Lax commented on BEAM-1831: -- Some more details - createdTables is ConcurrentHashMap, and is

[jira] [Commented] (BEAM-2108) Integration tests for PubsubIO

[ https://issues.apache.org/jira/browse/BEAM-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988271#comment-15988271 ] Reuven Lax commented on BEAM-2108: -- The plan was to integrate the fake PubSub client for use by the

[jira] [Created] (BEAM-2700) BigQueryIO should support using file load jobs when using unbounded collections

Reuven Lax created BEAM-2700: Summary: BigQueryIO should support using file load jobs when using unbounded collections Key: BEAM-2700 URL: https://issues.apache.org/jira/browse/BEAM-2700 Project: Beam

[jira] [Resolved] (BEAM-2772) BigQueryIO doesn't properly set job id when using triggered file loads

[ https://issues.apache.org/jira/browse/BEAM-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2772. -- Resolution: Fixed Fix Version/s: 2.2.0 > BigQueryIO doesn't properly set job id when using

[jira] [Created] (BEAM-2154) Writing to large numbers of BigQuery tables causes out-of-memory

Reuven Lax created BEAM-2154: Summary: Writing to large numbers of BigQuery tables causes out-of-memory Key: BEAM-2154 URL: https://issues.apache.org/jira/browse/BEAM-2154 Project: Beam Issue

[jira] [Commented] (BEAM-404) PubsubIO should have a mode that supports maintaining message attributes.

[ https://issues.apache.org/jira/browse/BEAM-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002214#comment-16002214 ] Reuven Lax commented on BEAM-404: - This has already been fixed in Dataflow 2.0 > PubsubIO should have a

[jira] [Created] (BEAM-2305) Dinstinct transform produces unexpected output when triggered

Reuven Lax created BEAM-2305: Summary: Dinstinct transform produces unexpected output when triggered Key: BEAM-2305 URL: https://issues.apache.org/jira/browse/BEAM-2305 Project: Beam Issue

[jira] [Created] (BEAM-2304) State API cannot recognize State superclases

Reuven Lax created BEAM-2304: Summary: State API cannot recognize State superclases Key: BEAM-2304 URL: https://issues.apache.org/jira/browse/BEAM-2304 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors

Reuven Lax created BEAM-2302: Summary: WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors Key: BEAM-2302 URL: https://issues.apache.org/jira/browse/BEAM-2302

[jira] [Commented] (BEAM-1438) The default behavior for the Write transform doesn't work well with the Dataflow streaming runner

[ https://issues.apache.org/jira/browse/BEAM-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056813#comment-16056813 ] Reuven Lax commented on BEAM-1438: -- I believe so > The default behavior for the Write transform

[jira] [Commented] (BEAM-1458) Checkpoint support in Beam

[ https://issues.apache.org/jira/browse/BEAM-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058773#comment-16058773 ] Reuven Lax commented on BEAM-1458: -- Correct - I'm working on a doc for snapshot/update support for Beam.

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[ https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060978#comment-16060978 ] Reuven Lax commented on BEAM-2353: -- The dynamic FileBasedSink PR is also a breaking change to

[jira] [Commented] (BEAM-2337) BigQuery IO improvements

[ https://issues.apache.org/jira/browse/BEAM-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018191#comment-16018191 ] Reuven Lax commented on BEAM-2337: -- Currently mostly in javadoc. However it appears that the Python BQ

[jira] [Commented] (BEAM-2305) Dinstinct transform produces unexpected output when triggered

[ https://issues.apache.org/jira/browse/BEAM-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012901#comment-16012901 ] Reuven Lax commented on BEAM-2305: -- On second thought, I think this is working as intended though

[jira] [Updated] (BEAM-2370) BigQuery Insert with Partition Decorator throwing error

[ https://issues.apache.org/jira/browse/BEAM-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2370: - I believe this is a know issue. For now if you pass in null for the table description (new

[jira] [Commented] (BEAM-2451) writeProtos() should allow a user to specify message attributes

[ https://issues.apache.org/jira/browse/BEAM-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053016#comment-16053016 ] Reuven Lax commented on BEAM-2451: -- The intention of the withTimestampAttribute/withIdAttribute was for

[jira] [Created] (BEAM-2210) PubsubIO.readPubsubMessagesWithoutAttributes is awkward

Reuven Lax created BEAM-2210: Summary: PubsubIO.readPubsubMessagesWithoutAttributes is awkward Key: BEAM-2210 URL: https://issues.apache.org/jira/browse/BEAM-2210 Project: Beam Issue Type: Bug

[jira] [Assigned] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[ https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax reassigned BEAM-2498: Assignee: Kenneth Knowles > Dataflow runner should shade Runner/Fn API protos >

[jira] [Assigned] (BEAM-2516) User reports 4 minutes to process 1 million line CSV in DirectRunner

[ https://issues.apache.org/jira/browse/BEAM-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax reassigned BEAM-2516: Assignee: Kenneth Knowles > User reports 4 minutes to process 1 million line CSV in DirectRunner >

[jira] [Commented] (BEAM-2761) Write to empty BigQuery partition fails with "No schema specified on job or table." despite having provided schema

[ https://issues.apache.org/jira/browse/BEAM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162362#comment-16162362 ] Reuven Lax commented on BEAM-2761: -- I've been trying to write a unit test that reproduces this. So far no

[jira] [Commented] (BEAM-2761) Write to empty BigQuery partition fails with "No schema specified on job or table." despite having provided schema

[ https://issues.apache.org/jira/browse/BEAM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162363#comment-16162363 ] Reuven Lax commented on BEAM-2761: -- Also a note - Create.empty(TableRowJsonCoder.of()) is a simpler way of

[jira] [Updated] (BEAM-2761) Write to empty BigQuery partition fails with "No schema specified on job or table." despite having provided schema

[ https://issues.apache.org/jira/browse/BEAM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2761: - Fix Version/s: 2.2.0 > Write to empty BigQuery partition fails with "No schema specified on job or >

[jira] [Closed] (BEAM-190) Dead-letter drop for bad BigQuery records

[ https://issues.apache.org/jira/browse/BEAM-190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax closed BEAM-190. --- Resolution: Fixed Fix Version/s: 2.1.0 > Dead-letter drop for bad BigQuery records >

[jira] [Commented] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[ https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162332#comment-16162332 ] Reuven Lax commented on BEAM-2498: -- [~kenn] This is currently on to 2.2.0 list. Does this need to go in

[jira] [Commented] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

[ https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172623#comment-16172623 ] Reuven Lax commented on BEAM-2271: -- Can this bug be closed now? > Release guide or pom.xml needs update

[jira] [Commented] (BEAM-2576) Move non-core transform payloads out of Runner API proto

[ https://issues.apache.org/jira/browse/BEAM-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172624#comment-16172624 ] Reuven Lax commented on BEAM-2576: -- Moving to 2.3.0 > Move non-core transform payloads out of Runner API

[jira] [Updated] (BEAM-2576) Move non-core transform payloads out of Runner API proto

[ https://issues.apache.org/jira/browse/BEAM-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2576: - Fix Version/s: (was: 2.2.0) 2.3.0 > Move non-core transform payloads out of Runner

[jira] [Updated] (BEAM-2603) Add Meter in beam metrics

[ https://issues.apache.org/jira/browse/BEAM-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2603: - Fix Version/s: (was: 2.2.0) 2.3.0 > Add Meter in beam metrics >

[jira] [Updated] (BEAM-2604) Delegate beam metrics to runners

[ https://issues.apache.org/jira/browse/BEAM-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2604: - Fix Version/s: (was: 2.2.0) 2.3.0 > Delegate beam metrics to runners >

[jira] [Commented] (BEAM-1868) CreateStreamTest testMultiOutputParDo is flaky on the Spark runner

[ https://issues.apache.org/jira/browse/BEAM-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172620#comment-16172620 ] Reuven Lax commented on BEAM-1868: -- This still hasn't been worked on AFAICT, so bumping to 2.3.0. >

[jira] [Updated] (BEAM-1868) CreateStreamTest testMultiOutputParDo is flaky on the Spark runner

[ https://issues.apache.org/jira/browse/BEAM-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-1868: - Fix Version/s: (was: 2.2.0) 2.3.0 > CreateStreamTest testMultiOutputParDo is flaky

[jira] [Updated] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[ https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2345: - Fix Version/s: (was: 2.2.0) 2.3.0 > Version configuration of plugins / dependencies

[jira] [Commented] (BEAM-2956) DataflowRunner incorrectly reports the user agent for the Dataflow distribution

[ https://issues.apache.org/jira/browse/BEAM-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172626#comment-16172626 ] Reuven Lax commented on BEAM-2956: -- Can this be closed? > DataflowRunner incorrectly reports the user

[jira] [Updated] (BEAM-2299) Beam repo build fails in Windows OS

[ https://issues.apache.org/jira/browse/BEAM-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2299: - Fix Version/s: (was: 2.2.0) 2.3.0 > Beam repo build fails in Windows OS >

[jira] [Updated] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[ https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2523: - Fix Version/s: (was: 2.2.0) 2.3.0 > GCP IO exposes protobuf on its API surface,

[jira] [Commented] (BEAM-2299) Beam repo build fails in Windows OS

[ https://issues.apache.org/jira/browse/BEAM-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172173#comment-16172173 ] Reuven Lax commented on BEAM-2299: -- bumping this to 2.3.0 > Beam repo build fails in Windows OS >

[jira] [Commented] (BEAM-2298) Java WordCount doesn't work in Window OS for glob expressions or file: prefixed paths

[ https://issues.apache.org/jira/browse/BEAM-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172175#comment-16172175 ] Reuven Lax commented on BEAM-2298: -- is this fixed? > Java WordCount doesn't work in Window OS for glob

[jira] [Resolved] (BEAM-2829) Add ability to set job labels in DataflowPipelineOptions

[ https://issues.apache.org/jira/browse/BEAM-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2829. -- Resolution: Fixed > Add ability to set job labels in DataflowPipelineOptions >

[jira] [Closed] (BEAM-2829) Add ability to set job labels in DataflowPipelineOptions

[ https://issues.apache.org/jira/browse/BEAM-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax closed BEAM-2829. > Add ability to set job labels in DataflowPipelineOptions >

[jira] [Updated] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[ https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2273: - Fix Version/s: (was: 2.2.0) 2.3.0 > mvn clean doesn't fully clean up archetypes. >

[jira] [Commented] (BEAM-2984) Job submission too large with embedded Beam protos

[ https://issues.apache.org/jira/browse/BEAM-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176953#comment-16176953 ] Reuven Lax commented on BEAM-2984: -- Is this a 2.2.0 blocker? > Job submission too large with embedded

[jira] [Updated] (BEAM-2377) Cross compile flink runner to scala 2.11

[ https://issues.apache.org/jira/browse/BEAM-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2377: - Fix Version/s: (was: 2.2.0) 2.3.0 > Cross compile flink runner to scala 2.11 >

[jira] [Commented] (BEAM-2298) Java WordCount doesn't work in Window OS for glob expressions or file: prefixed paths

[ https://issues.apache.org/jira/browse/BEAM-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176954#comment-16176954 ] Reuven Lax commented on BEAM-2298: -- I am resolving this for now. Please reopen if you believe the issue

[jira] [Resolved] (BEAM-2298) Java WordCount doesn't work in Window OS for glob expressions or file: prefixed paths

[ https://issues.apache.org/jira/browse/BEAM-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2298. -- Resolution: Fixed > Java WordCount doesn't work in Window OS for glob expressions or file: > prefixed

[jira] [Resolved] (BEAM-2956) DataflowRunner incorrectly reports the user agent for the Dataflow distribution

[ https://issues.apache.org/jira/browse/BEAM-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2956. -- Resolution: Fixed > DataflowRunner incorrectly reports the user agent for the Dataflow > distribution >

[jira] [Closed] (BEAM-2870) BQ Partitioned Table Write Fails When Destination has Partition Decorator

[ https://issues.apache.org/jira/browse/BEAM-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax closed BEAM-2870. Resolution: Fixed > BQ Partitioned Table Write Fails When Destination has Partition Decorator >

[jira] [Closed] (BEAM-2834) NullPointerException @ BigQueryServicesImpl.java:759

[ https://issues.apache.org/jira/browse/BEAM-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax closed BEAM-2834. Resolution: Fixed > NullPointerException @ BigQueryServicesImpl.java:759 >

[jira] [Commented] (BEAM-2870) BQ Partitioned Table Write Fails When Destination has Partition Decorator

[ https://issues.apache.org/jira/browse/BEAM-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172081#comment-16172081 ] Reuven Lax commented on BEAM-2870: -- I think the problem is that we need to strip the partition decorator

[jira] [Commented] (BEAM-2870) BQ Partitioned Table Write Fails When Destination has Partition Decorator

[ https://issues.apache.org/jira/browse/BEAM-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172151#comment-16172151 ] Reuven Lax commented on BEAM-2870: -- Can you try out this PR and let us know if it fixes the issue you are

[jira] [Updated] (BEAM-2865) Implement FileIO.write()

[ https://issues.apache.org/jira/browse/BEAM-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2865: - Fix Version/s: (was: 2.2.0) 2.3.0 > Implement FileIO.write() >

[jira] [Commented] (BEAM-2604) Delegate beam metrics to runners

[ https://issues.apache.org/jira/browse/BEAM-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172172#comment-16172172 ] Reuven Lax commented on BEAM-2604: -- Does this need to block the 2.2.0 cut? > Delegate beam metrics to

[jira] [Updated] (BEAM-2994) Refactor TikaIO

[ https://issues.apache.org/jira/browse/BEAM-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2994: - Fix Version/s: (was: 2.2.0) 2.3.0 > Refactor TikaIO > --- > >

[jira] [Assigned] (BEAM-3039) DatastoreIO.Write fails multiple mutations of same entity

[ https://issues.apache.org/jira/browse/BEAM-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax reassigned BEAM-3039: Assignee: Chamikara Jayalath (was: Reuven Lax) > DatastoreIO.Write fails multiple mutations of

[jira] [Commented] (BEAM-3029) BigTable integration tests failing on Dataflow: UserAgent must not be empty

[ https://issues.apache.org/jira/browse/BEAM-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205375#comment-16205375 ] Reuven Lax commented on BEAM-3029: -- [~chamikara] any idea what's causing this? > BigTable integration

[jira] [Assigned] (BEAM-3029) BigTable integration tests failing on Dataflow: UserAgent must not be empty

[ https://issues.apache.org/jira/browse/BEAM-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax reassigned BEAM-3029: Assignee: Chamikara Jayalath (was: Daniel Oliveira) > BigTable integration tests failing on

[jira] [Commented] (BEAM-2843) Log the reason for failure for BigQuery jobs

[ https://issues.apache.org/jira/browse/BEAM-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154798#comment-16154798 ] Reuven Lax commented on BEAM-2843: -- I believe the failure reason is logged inside BigQueryServicesImpl. >

[jira] [Created] (BEAM-2823) Beam Windows MavenInstall tests failing

Reuven Lax created BEAM-2823: Summary: Beam Windows MavenInstall tests failing Key: BEAM-2823 URL: https://issues.apache.org/jira/browse/BEAM-2823 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2858) temp file garbage collection in BigQuery sink should be in a separate DoFn

[ https://issues.apache.org/jira/browse/BEAM-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157856#comment-16157856 ] Reuven Lax commented on BEAM-2858: -- I asked the BigQuery team, and they said the load job should fail. How

[jira] [Updated] (BEAM-2952) How to use KV.OrderByKey

[ https://issues.apache.org/jira/browse/BEAM-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2952: - Fix Version/s: (was: 2.1.0) > How to use KV.OrderByKey > > >

[jira] [Commented] (BEAM-2952) How to use KV.OrderByKey

[ https://issues.apache.org/jira/browse/BEAM-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164081#comment-16164081 ] Reuven Lax commented on BEAM-2952: -- I'm not sure how you want to use it. A PCollection is not ordered, so

[jira] [Commented] (BEAM-2872) AvroIO.TypedWrite#to() method produces a compilation error

[ https://issues.apache.org/jira/browse/BEAM-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160416#comment-16160416 ] Reuven Lax commented on BEAM-2872: -- I can't reproduce this. This code is returning a raw type, so Java

[jira] [Commented] (BEAM-2761) Write to empty BigQuery partition fails with "No schema specified on job or table." despite having provided schema

[ https://issues.apache.org/jira/browse/BEAM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163509#comment-16163509 ] Reuven Lax commented on BEAM-2761: -- Hi, I ran the precise job listed in the bug using the latest Beam

[jira] [Resolved] (BEAM-2761) Write to empty BigQuery partition fails with "No schema specified on job or table." despite having provided schema

[ https://issues.apache.org/jira/browse/BEAM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2761. -- Resolution: Cannot Reproduce > Write to empty BigQuery partition fails with "No schema specified on job

[jira] [Commented] (BEAM-2858) temp file garbage collection in BigQuery sink should be in a separate DoFn

[ https://issues.apache.org/jira/browse/BEAM-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163529#comment-16163529 ] Reuven Lax commented on BEAM-2858: -- I just reproduced this and verified it does not cause data loss. The

[jira] [Commented] (BEAM-2840) BigQueryIO write is slow/fail with a bounded source

[ https://issues.apache.org/jira/browse/BEAM-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156373#comment-16156373 ] Reuven Lax commented on BEAM-2840: -- Interesting - I tested this with 20TB of data being written to

[jira] [Created] (BEAM-2858) temp file garbage collection in BigQuery sink should be in a separate DoFn

Reuven Lax created BEAM-2858: Summary: temp file garbage collection in BigQuery sink should be in a separate DoFn Key: BEAM-2858 URL: https://issues.apache.org/jira/browse/BEAM-2858 Project: Beam

[jira] [Closed] (BEAM-2858) temp file garbage collection in BigQuery sink should be in a separate DoFn

[ https://issues.apache.org/jira/browse/BEAM-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax closed BEAM-2858. Resolution: Fixed > temp file garbage collection in BigQuery sink should be in a separate DoFn >

[jira] [Resolved] (BEAM-2992) Remove codepaths for reading unsplit BigQuery sources

[ https://issues.apache.org/jira/browse/BEAM-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2992. -- Resolution: Fixed > Remove codepaths for reading unsplit BigQuery sources >

[jira] [Closed] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors

[ https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax closed BEAM-2302. > WriteFiles with runner-determined sharding and large numbers of windows > causes OOM errors >

[jira] [Resolved] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors

[ https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2302. -- Resolution: Fixed Fix Version/s: 2.1.0 > WriteFiles with runner-determined sharding and large

[jira] [Updated] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[ https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-2624: - Fix Version/s: 2.2.0 > File-based sinks should produce a PCollection of written filenames >

[jira] [Commented] (BEAM-2834) NullPointerException @ BigQueryServicesImpl.java:759

[ https://issues.apache.org/jira/browse/BEAM-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151017#comment-16151017 ] Reuven Lax commented on BEAM-2834: -- Thanks for the bug report! This does appear to be a bug. It won't

[jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers

[ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145446#comment-16145446 ] Reuven Lax commented on BEAM-2768: -- Which runner are you using? I can't see anything wrong with the code,

[jira] [Commented] (BEAM-3011) Pin Runner harness container image in Python SDK

[ https://issues.apache.org/jira/browse/BEAM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195614#comment-16195614 ] Reuven Lax commented on BEAM-3011: -- Any update on this issue? This is blocking 2.2.0 and has been

[jira] [Resolved] (BEAM-2954) update shade configurations in extension/sql

[ https://issues.apache.org/jira/browse/BEAM-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax resolved BEAM-2954. -- Resolution: Fixed > update shade configurations in extension/sql >

[jira] [Updated] (BEAM-3243) multiple anonymous DoFn lead to conflicting names

[ https://issues.apache.org/jira/browse/BEAM-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuven Lax updated BEAM-3243: - Fix Version/s: (was: 2.2.0) 2.3.0 > multiple anonymous DoFn lead to conflicting

[jira] [Commented] (BEAM-3200) Streaming Pipeline throws RuntimeException when using DynamicDestinations and Method.FILE_LOADS

[ https://issues.apache.org/jira/browse/BEAM-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257862#comment-16257862 ] Reuven Lax commented on BEAM-3200: -- We shuffle with Destination as they key before calling WriteTables.

  1   2   >