date:20191014



 [ 
https://issues.apache.org/jira/browse/BEAM-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-8391:
---
Status: Open  (was: Triage Needed)

> `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write 
> methods
> -
>
> Key: BEAM-8391
> URL: https://issues.apache.org/jira/browse/BEAM-8391
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Tatu Saloranta
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> `AwsModule` class uses old `typeSerializer.writeTypePrefixForObject()` method 
> in `serializeWithType()`, deprecated in Jackson 2.9. While this still works 
> (and should work for 2.x), it makes sense to use replacement that is fully 
> supported.
> I will provide a patch for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-8391) `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write methods



 [ 
https://issues.apache.org/jira/browse/BEAM-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-8391:
--

Assignee: Tatu Saloranta

> `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write 
> methods
> -
>
> Key: BEAM-8391
> URL: https://issues.apache.org/jira/browse/BEAM-8391
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Tatu Saloranta
>Assignee: Tatu Saloranta
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> `AwsModule` class uses old `typeSerializer.writeTypePrefixForObject()` method 
> in `serializeWithType()`, deprecated in Jackson 2.9. While this still works 
> (and should work for 2.x), it makes sense to use replacement that is fully 
> supported.
> I will provide a patch for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8385) Add an option to run KafkaIOIT with 10GB dataset



 [ 
https://issues.apache.org/jira/browse/BEAM-8385?focusedWorklogId=327641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327641
 ]

ASF GitHub Bot logged work on BEAM-8385:


Author: ASF GitHub Bot
Created on: 14/Oct/19 07:16
Start Date: 14/Oct/19 07:16
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on pull request #9770: [BEAM-8385] 
Add a KafkaIOIT hash for 100M records
URL: https://github.com/apache/beam/pull/9770#discussion_r334348492
 
 

 ##
 File path: .test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml
 ##
 @@ -34,5 +34,5 @@ data:
   sleep 20
 done
 echo "Applying runtime configuration using confluentinc/cp-kafka:5.0.1"
-kafka-topics --zookeeper zookeeper:2181 --create --if-not-exists --force 
--topic apache-beam-load-test --partitions 3 --replication-factor 2
+kafka-topics --zookeeper zookeeper:2181 --create --if-not-exists --force 
--topic apache-beam-load-test --partitions 1 --replication-factor 1
 
 Review comment:
   The increase of number of partitions introduces a possibility that not all 
messages will be read from the topic and the test will hang. Keeping it at 1 
ensures that the tests finish
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327641)
Time Spent: 40m  (was: 0.5h)

> Add an option to run KafkaIOIT with 10GB dataset
> 
>
> Key: BEAM-8385
> URL: https://issues.apache.org/jira/browse/BEAM-8385
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8344) Add infer schema support in ParquetIO and refactor ParquetTableProvider



 [ 
https://issues.apache.org/jira/browse/BEAM-8344?focusedWorklogId=327643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327643
 ]

ASF GitHub Bot logged work on BEAM-8344:


Author: ASF GitHub Bot
Created on: 14/Oct/19 07:37
Start Date: 14/Oct/19 07:37
Worklog Time Spent: 10m 
  Work Description: bmv126 commented on pull request #9721: [BEAM-8344] Add 
inferSchema support in ParquetIO and refactor ParquetTableProvider
URL: https://github.com/apache/beam/pull/9721#discussion_r334354220
 
 

 ##
 File path: 
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
 ##
 @@ -122,15 +122,34 @@
* pattern).
*/
   public static Read read(Schema schema) {
-return new AutoValue_ParquetIO_Read.Builder().setSchema(schema).build();
+return new AutoValue_ParquetIO_Read.Builder()
+.setSchema(schema)
+.setInferBeamSchema(false)
+.build();
   }
 
   /**
* Like {@link #read(Schema)}, but reads each file in a {@link PCollection} 
of {@link
* org.apache.beam.sdk.io.FileIO.ReadableFile}, which allows more flexible 
usage.
*/
   public static ReadFiles readFiles(Schema schema) {
-return new 
AutoValue_ParquetIO_ReadFiles.Builder().setSchema(schema).build();
+return new AutoValue_ParquetIO_ReadFiles.Builder()
+.setSchema(schema)
+.setInferBeamSchema(false)
+.build();
+  }
+
+  private static  PCollection setBeamSchema(
+  PCollection pc, Class clazz, @Nullable Schema schema) {
+org.apache.beam.sdk.schemas.Schema beamSchema =
+org.apache.beam.sdk.schemas.utils.AvroUtils.getSchema(clazz, schema);
+if (beamSchema != null) {
 
 Review comment:
   @amaliujia I have done the modification. Can you have a look
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327643)
Time Spent: 40m  (was: 0.5h)

> Add infer schema support in ParquetIO and refactor ParquetTableProvider
> ---
>
> Key: BEAM-8344
> URL: https://issues.apache.org/jira/browse/BEAM-8344
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql, io-java-parquet
>Reporter: Vishwas
>Assignee: Vishwas
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Add support for inferring Beam Schema in ParquetIO.
> Refactor ParquetTable code to use Convert.rows().
> Remove unnecessary java class GenericRecordReadConverter.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8385) Add an option to run KafkaIOIT with 10GB dataset



 [ 
https://issues.apache.org/jira/browse/BEAM-8385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-8385:
---
Status: Open  (was: Triage Needed)

> Add an option to run KafkaIOIT with 10GB dataset
> 
>
> Key: BEAM-8385
> URL: https://issues.apache.org/jira/browse/BEAM-8385
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-8388:
---
Status: Open  (was: Triage Needed)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 1h 40m
>  Remaining Estimate: 22h 20m
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-7328) Update Avro to version 1.9.0 in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-7328:
--

Assignee: (was: Ismaël Mejía)

> Update Avro to version 1.9.0 in Java SDK
> 
>
> Key: BEAM-7328
> URL: https://issues.apache.org/jira/browse/BEAM-7328
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Priority: Minor
>
> Avro 1.9.0 has nice improvements like a reduced size (1MB less) , multiple 
> dependencies are not needed anymore (Guava, paranamer, etc) as well as 
> cleanups in its APIs to not expose and be tight to Jackson so a worth upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-7328) Update Avro to version 1.9.0 in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-7328.

Fix Version/s: Not applicable
   Resolution: Duplicate

> Update Avro to version 1.9.0 in Java SDK
> 
>
> Key: BEAM-7328
> URL: https://issues.apache.org/jira/browse/BEAM-7328
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Avro 1.9.0 has nice improvements like a reduced size (1MB less) , multiple 
> dependencies are not needed anymore (Guava, paranamer, etc) as well as 
> cleanups in its APIs to not expose and be tight to Jackson so a worth upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=327674&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327674
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 14/Oct/19 08:44
Start Date: 14/Oct/19 08:44
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9779: [BEAM-8388] Updating 
avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779#issuecomment-541557410
 
 
   [BEAM-7328](https://issues.apache.org/jira/browse/BEAM-7328) (the previous 
issue to migrate to Avro 1.9.x) ended up being 'stalled' for two reasons still 
worth to consider in this PR:
   
   1. Potential compatibility issues for Beam Java users given that Avro is 
transitively 'leaked' as part of Core SDK public API. This is the same issue 
for any 'transitive' dependency we offer on Beam (e.g. Jackson), but there may 
be some minor issues for users because of Avro API changes, remember Avro minor 
versions are not backwards compatible code-wise (but they are format-wise). 
Anyway migrating is good for multiple other reasons like security fixes, but we 
probably should warn or discuss on the consequences of this on the ML.
   
   2. Issues on target runners where Avro 1.9.x has not been downstreamed. More 
concretely Spark runner core module depends still on Avro 1.8.x (AFAIK Spark 
runner does not use Avro at all, but maybe Spark core does) so we should 
probably as part of this PR guarantee that all the Spark related tests pass 
too).
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327674)
Remaining Estimate: 22h 10m  (was: 22h 20m)
Time Spent: 1h 50m  (was: 1h 40m)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 1h 50m
>  Remaining Estimate: 22h 10m
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=327675&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327675
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 14/Oct/19 08:46
Start Date: 14/Oct/19 08:46
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9779: [BEAM-8388] Updating 
avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779#issuecomment-541557410
 
 
   [BEAM-7328](https://issues.apache.org/jira/browse/BEAM-7328) (the previous 
issue to migrate to Avro 1.9.x) ended up being 'stalled' for two reasons still 
worth to consider in this PR:
   
   1. Potential compatibility issues for Beam Java users given that Avro is 
transitively 'leaked' as part of Core SDK public API. This is the same issue 
for any 'transitive' dependency we offer on Beam (e.g. Jackson), but there may 
be some minor issues for users because of Avro API changes, remember Avro minor 
versions are not backwards compatible code-wise (but they 'should' be 
format-wise). Anyway migrating is good for multiple other reasons like security 
fixes, but we probably should warn or discuss on the consequences of this on 
the ML.
   
   2. Runtime issues on target runners where Avro 1.9.x has not been 
downstreamed. More concretely Spark runner core module depends still on Avro 
1.8.x (AFAIK Spark runner does not use Avro at all, but maybe Spark core does) 
so we should probably as part of this PR guarantee that all the Spark related 
tests pass too).
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327675)
Remaining Estimate: 22h  (was: 22h 10m)
Time Spent: 2h  (was: 1h 50m)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 2h
>  Remaining Estimate: 22h
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



[ 
https://issues.apache.org/jira/browse/BEAM-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950836#comment-16950836
 ] 

Kamil Wasilewski commented on BEAM-8368:


Thanks [~wesm]. We don't import `pyarrow.flight`, so we should be safe with 
0.15.1 release

 

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8281) Resize file-based IOITs



 [ 
https://issues.apache.org/jira/browse/BEAM-8281?focusedWorklogId=327687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327687
 ]

ASF GitHub Bot logged work on BEAM-8281:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:20
Start Date: 14/Oct/19 09:20
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9638: [BEAM-8281] Resize 
IOITs datasets
URL: https://github.com/apache/beam/pull/9638#issuecomment-541572843
 
 
   I added smaller test configurations for fast testing after receiving a 
suggestion on a dev@ thread.
   @lgajowy WDYT?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327687)
Time Spent: 40m  (was: 0.5h)

> Resize file-based IOITs 
> 
>
> Key: BEAM-8281
> URL: https://issues.apache.org/jira/browse/BEAM-8281
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Resize the IOITs to use known amounts of data saved on GCS so that it's 
> possible to report dataset size and measure throughput as part of the test 
> metrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8392) Update pyarrow version bounds

Kamil Wasilewski created BEAM-8392:
--

 Summary: Update pyarrow version bounds
 Key: BEAM-8392
 URL: https://issues.apache.org/jira/browse/BEAM-8392
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Kamil Wasilewski


arrow is 
[[considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]
 a 0.15.1 release which is likely to solve the problem described here: 
[https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]

The task is to bump up the lower bound for pyarrow in setup.py as soon as 
0.15.1 is released.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8391) `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write methods



 [ 
https://issues.apache.org/jira/browse/BEAM-8391?focusedWorklogId=327697&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327697
 ]

ASF GitHub Bot logged work on BEAM-8391:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:35
Start Date: 14/Oct/19 09:35
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #9783: [BEAM-8391] 
Update type id write methods in `AwsModule`
URL: https://github.com/apache/beam/pull/9783
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327697)
Time Spent: 1h  (was: 50m)

> `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write 
> methods
> -
>
> Key: BEAM-8391
> URL: https://issues.apache.org/jira/browse/BEAM-8391
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Tatu Saloranta
>Assignee: Tatu Saloranta
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> `AwsModule` class uses old `typeSerializer.writeTypePrefixForObject()` method 
> in `serializeWithType()`, deprecated in Jackson 2.9. While this still works 
> (and should work for 2.x), it makes sense to use replacement that is fully 
> supported.
> I will provide a patch for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8391) `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write methods



 [ 
https://issues.apache.org/jira/browse/BEAM-8391?focusedWorklogId=327696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327696
 ]

ASF GitHub Bot logged work on BEAM-8391:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:35
Start Date: 14/Oct/19 09:35
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9783: [BEAM-8391] Update 
type id write methods in `AwsModule`
URL: https://github.com/apache/beam/pull/9783#issuecomment-541578827
 
 
   Merged, thanks again. I just saw you are a jackson maintainer catching up 
downstream, makes sense now :). Never mind about the update suggestion if you 
don't have the time.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327696)
Time Spent: 50m  (was: 40m)

> `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write 
> methods
> -
>
> Key: BEAM-8391
> URL: https://issues.apache.org/jira/browse/BEAM-8391
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Tatu Saloranta
>Assignee: Tatu Saloranta
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> `AwsModule` class uses old `typeSerializer.writeTypePrefixForObject()` method 
> in `serializeWithType()`, deprecated in Jackson 2.9. While this still works 
> (and should work for 2.x), it makes sense to use replacement that is fully 
> supported.
> I will provide a patch for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-8391) `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write methods



 [ 
https://issues.apache.org/jira/browse/BEAM-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-8391.

Fix Version/s: 2.17.0
   Resolution: Fixed

> `AwsModule` of sdks/java/io/amazon-web-services uses deprecated type id write 
> methods
> -
>
> Key: BEAM-8391
> URL: https://issues.apache.org/jira/browse/BEAM-8391
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Tatu Saloranta
>Assignee: Tatu Saloranta
>Priority: Major
> Fix For: 2.17.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> `AwsModule` class uses old `typeSerializer.writeTypePrefixForObject()` method 
> in `serializeWithType()`, deprecated in Jackson 2.9. While this still works 
> (and should work for 2.x), it makes sense to use replacement that is fully 
> supported.
> I will provide a patch for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8392) Update pyarrow version bounds



 [ 
https://issues.apache.org/jira/browse/BEAM-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Wasilewski updated BEAM-8392:
---
Description: 
arrow is 
[considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]]
 a 0.15.1 release which is likely to solve the problem described here: 
[https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]

The task is to bump up the lower bound for pyarrow in setup.py as soon as 
0.15.1 is released.

  was:
arrow is 
[[considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]
 a 0.15.1 release which is likely to solve the problem described here: 
[https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]

The task is to bump up the lower bound for pyarrow in setup.py as soon as 
0.15.1 is released.


> Update pyarrow version bounds
> -
>
> Key: BEAM-8392
> URL: https://issues.apache.org/jira/browse/BEAM-8392
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Kamil Wasilewski
>Priority: Major
>
> arrow is 
> [considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]]
>  a 0.15.1 release which is likely to solve the problem described here: 
> [https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]
> The task is to bump up the lower bound for pyarrow in setup.py as soon as 
> 0.15.1 is released.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8392) Update pyarrow version bounds



 [ 
https://issues.apache.org/jira/browse/BEAM-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Wasilewski updated BEAM-8392:
---
Description: 
arrow is 
[considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]
 a 0.15.1 release which is likely to solve the problem described here: 
[https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]

The task is to bump up the lower bound for pyarrow in setup.py as soon as 
0.15.1 is released.

  was:
arrow is 
[considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]]
 a 0.15.1 release which is likely to solve the problem described here: 
[https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]

The task is to bump up the lower bound for pyarrow in setup.py as soon as 
0.15.1 is released.


> Update pyarrow version bounds
> -
>
> Key: BEAM-8392
> URL: https://issues.apache.org/jira/browse/BEAM-8392
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Kamil Wasilewski
>Priority: Major
>
> arrow is 
> [considering|https://lists.apache.org/thread.html/462464941d79e67bfef7d73fc900ea5fe0200e8a926253c6eb0285aa@%3Cdev.arrow.apache.org%3E]
>  a 0.15.1 release which is likely to solve the problem described here: 
> [https://issues.apache.org/jira/browse/BEAM-8368|https://github.com/apache/beam/pull/9768]
> The task is to bump up the lower bound for pyarrow in setup.py as soon as 
> 0.15.1 is released.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=327702&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327702
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:45
Start Date: 14/Oct/19 09:45
Worklog Time Spent: 10m 
  Work Description: RyanSkraba commented on pull request #9779: [BEAM-8388] 
Updating avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779#discussion_r334400709
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
 ##
 @@ -75,8 +75,8 @@
   static {
 // This works around a bug in the Avro library (AVRO-1891) around 
SpecificRecord's handling
 // of DateTime types.
-SpecificData.get().addLogicalTypeConversion(new 
TimeConversions.TimestampConversion());
-GenericData.get().addLogicalTypeConversion(new 
TimeConversions.TimestampConversion());
+SpecificData.get().addLogicalTypeConversion(new 
JodaTimeConversions.TimestampConversion());
+GenericData.get().addLogicalTypeConversion(new 
JodaTimeConversions.TimestampConversion());
 
 Review comment:
   Hello!  Just for information, Avro 1.9.x supports `joda-time`  and 
`java.time`, with the strong recommendation that users start preferring the 
`java.time` classes.  Avro 1.10.x removes built-in support for joda. 
   
   In any case, good news: AVRO-1891 has been fixed in Avro 1.9.x, so these 
lines can probably be removed!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327702)
Remaining Estimate: 21h 50m  (was: 22h)
Time Spent: 2h 10m  (was: 2h)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 2h 10m
>  Remaining Estimate: 21h 50m
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



 [ 
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=327704&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327704
 ]

ASF GitHub Bot logged work on BEAM-8368:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:54
Start Date: 14/Oct/19 09:54
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on pull request #9768: [BEAM-8368] 
Avoid libprotobuf-generated exception when importing apache_beam
URL: https://github.com/apache/beam/pull/9768#discussion_r334404318
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -119,7 +119,9 @@ def get_version():
 'oauth2client>=2.0.1,<4',
 'protobuf>=3.5.0.post1,<4',
 # [BEAM-6287] pyarrow is not supported on Windows for Python 2
-('pyarrow>=0.11.1,<0.15.0; python_version >= "3.0" or '
+# [BEAM-8368] pyarrow>=0.14.0 triggers an exception when importing
+# apache_beam on macOS 10.15
 
 Review comment:
   Done, jira is filled.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327704)
Time Spent: 0.5h  (was: 20m)

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata



 [ 
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=327706&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327706
 ]

ASF GitHub Bot logged work on BEAM-8374:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:54
Start Date: 14/Oct/19 09:54
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #9758: [BEAM-8374] 
Fixes bug in SnsIO PublishResultCoder
URL: https://github.com/apache/beam/pull/9758#discussion_r334403378
 
 

 ##
 File path: 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java
 ##
 @@ -42,12 +50,27 @@ static PublishResultCoder of() {
   public void encode(PublishResult value, OutputStream outStream)
   throws CoderException, IOException {
 StringUtf8Coder.of().encode(value.getMessageId(), outStream);
+StringUtf8Coder.of().encode(value.getSdkResponseMetadata().getRequestId(), 
outStream);
+MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())
+.encode(value.getSdkHttpMetadata().getHttpHeaders(), outStream);
+VarInt.encode(value.getSdkHttpMetadata().getHttpStatusCode(), outStream);
 
 Review comment:
   nit: I will prefer the HTTP status code to be first just for consistency on 
the way HTTP works.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327706)
Time Spent: 1h  (was: 50m)

> PublishResult returned by SnsIO is missing sdkResponseMetadata and 
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field 
> so the PublishResult returned by Beam returns null for 
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible 
> to check the HTTP status for errors, which is necessary since this is not 
> handled in SnsIO.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



 [ 
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=327705&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327705
 ]

ASF GitHub Bot logged work on BEAM-8368:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:54
Start Date: 14/Oct/19 09:54
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on pull request #9768: [BEAM-8368] 
Avoid libprotobuf-generated exception when importing apache_beam
URL: https://github.com/apache/beam/pull/9768#discussion_r334404455
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -119,7 +119,9 @@ def get_version():
 'oauth2client>=2.0.1,<4',
 'protobuf>=3.5.0.post1,<4',
 # [BEAM-6287] pyarrow is not supported on Windows for Python 2
-('pyarrow>=0.11.1,<0.15.0; python_version >= "3.0" or '
+# [BEAM-8368] pyarrow>=0.14.0 triggers an exception when importing
+# apache_beam on macOS 10.15
 
 Review comment:
   https://issues.apache.org/jira/browse/BEAM-8392
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327705)
Time Spent: 40m  (was: 0.5h)

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata



 [ 
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=327707&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327707
 ]

ASF GitHub Bot logged work on BEAM-8374:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:54
Start Date: 14/Oct/19 09:54
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #9758: [BEAM-8374] 
Fixes bug in SnsIO PublishResultCoder
URL: https://github.com/apache/beam/pull/9758#discussion_r334403338
 
 

 ##
 File path: 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java
 ##
 @@ -42,12 +50,27 @@ static PublishResultCoder of() {
   public void encode(PublishResult value, OutputStream outStream)
   throws CoderException, IOException {
 StringUtf8Coder.of().encode(value.getMessageId(), outStream);
+StringUtf8Coder.of().encode(value.getSdkResponseMetadata().getRequestId(), 
outStream);
 
 Review comment:
   Since `SdkResponseMetadata` and `SdkHttpMetadata` seem to be used by 
multiple parts of AWS SDK I was wondering if we could create specific Coders 
(and tests) for both objects and just reuse them here. I know this could add 
some extra chores to this PR but we may benefit from them eventually in other 
IOs..
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327707)
Time Spent: 1h  (was: 50m)

> PublishResult returned by SnsIO is missing sdkResponseMetadata and 
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field 
> so the PublishResult returned by Beam returns null for 
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible 
> to check the HTTP status for errors, which is necessary since this is not 
> handled in SnsIO.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



 [ 
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=327708&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327708
 ]

ASF GitHub Bot logged work on BEAM-8368:


Author: ASF GitHub Bot
Created on: 14/Oct/19 09:56
Start Date: 14/Oct/19 09:56
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9768: [BEAM-8368] Avoid 
libprotobuf-generated exception when importing apache_beam
URL: https://github.com/apache/beam/pull/9768#issuecomment-541587102
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327708)
Time Spent: 50m  (was: 40m)

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata



 [ 
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=327710&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327710
 ]

ASF GitHub Bot logged work on BEAM-8374:


Author: ASF GitHub Bot
Created on: 14/Oct/19 10:03
Start Date: 14/Oct/19 10:03
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #9758: [BEAM-8374] 
Fixes bug in SnsIO PublishResultCoder
URL: https://github.com/apache/beam/pull/9758#discussion_r334403378
 
 

 ##
 File path: 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java
 ##
 @@ -42,12 +50,27 @@ static PublishResultCoder of() {
   public void encode(PublishResult value, OutputStream outStream)
   throws CoderException, IOException {
 StringUtf8Coder.of().encode(value.getMessageId(), outStream);
+StringUtf8Coder.of().encode(value.getSdkResponseMetadata().getRequestId(), 
outStream);
+MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())
+.encode(value.getSdkHttpMetadata().getHttpHeaders(), outStream);
+VarInt.encode(value.getSdkHttpMetadata().getHttpStatusCode(), outStream);
 
 Review comment:
   nit: I'd prefer the HTTP status code to be first just for consistency on the 
way HTTP works. But that's pretty minor.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327710)
Time Spent: 1h 10m  (was: 1h)

> PublishResult returned by SnsIO is missing sdkResponseMetadata and 
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field 
> so the PublishResult returned by Beam returns null for 
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible 
> to check the HTTP status for errors, which is necessary since this is not 
> handled in SnsIO.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=327713&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327713
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 14/Oct/19 10:09
Start Date: 14/Oct/19 10:09
Worklog Time Spent: 10m 
  Work Description: RyanSkraba commented on issue #9779: [BEAM-8388] 
Updating avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779#issuecomment-541592666
 
 
   @iemejia 
   
   > 2\. Spark runner core module depends still on Avro 1.8.x
   
   Except where it still depends on 1.7.x  underneath it all... (See [Spark 
2.3.3](https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11/2.3.3))
   
   I think 1.7 and 1.8 are API compatible-ish enough, but it hasn't been 
exhaustively tested.  I ran into a problem with ParquetIO using a 1.8.x Avro 
API that required shading for older versions of spark (documented here: 
https://beam.apache.org/documentation/io/built-in/parquet/)
   
   It might be worth checking this sooner rather than later, in case bumping 
Avro implies dropping some versions for Spark support!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327713)
Remaining Estimate: 21h 40m  (was: 21h 50m)
Time Spent: 2h 20m  (was: 2h 10m)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 2h 20m
>  Remaining Estimate: 21h 40m
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata



 [ 
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=327714&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327714
 ]

ASF GitHub Bot logged work on BEAM-8374:


Author: ASF GitHub Bot
Created on: 14/Oct/19 10:13
Start Date: 14/Oct/19 10:13
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9758: [BEAM-8374] Fixes bug 
in SnsIO PublishResultCoder
URL: https://github.com/apache/beam/pull/9758#issuecomment-541594305
 
 
   One extra issue that I have not really thought about is the size difference 
that we will have by adding the full headers/sdk metadata to all answers, 
mmm... now I am hesitating about maybe having a `V2` like approach for the 
cases where we care about the full metadata (akin to FileIO).
   What is the use case that is pushing into this approach?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327714)
Time Spent: 1h 20m  (was: 1h 10m)

> PublishResult returned by SnsIO is missing sdkResponseMetadata and 
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field 
> so the PublishResult returned by Beam returns null for 
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible 
> to check the HTTP status for errors, which is necessary since this is not 
> handled in SnsIO.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=327724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327724
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 10:27
Start Date: 14/Oct/19 10:27
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541600585
 
 
   Run Python 2 PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327724)
Time Spent: 40m  (was: 0.5h)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=327742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327742
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 11:17
Start Date: 14/Oct/19 11:17
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541619151
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327742)
Time Spent: 50m  (was: 40m)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=327771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327771
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 11:51
Start Date: 14/Oct/19 11:51
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541631306
 
 
   Run Python 37 PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327771)
Time Spent: 1h 10m  (was: 1h)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=327770&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327770
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 11:51
Start Date: 14/Oct/19 11:51
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541631306
 
 
   Run Python 37 PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327770)
Time Spent: 1h  (was: 50m)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8281) Resize file-based IOITs



 [ 
https://issues.apache.org/jira/browse/BEAM-8281?focusedWorklogId=32&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-32
 ]

ASF GitHub Bot logged work on BEAM-8281:


Author: ASF GitHub Bot
Created on: 14/Oct/19 12:02
Start Date: 14/Oct/19 12:02
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on pull request #9638: [BEAM-8281] 
Resize IOITs datasets
URL: https://github.com/apache/beam/pull/9638#discussion_r334439693
 
 

 ##
 File path: 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
 ##
 @@ -45,15 +43,53 @@ public static String appendTimestampSuffix(String text) {
 return String.format("%s_%s", text, new Date().getTime());
   }
 
-  public static String getExpectedHashForLineCount(int lineCount) {
-Map expectedHashes =
-ImmutableMap.of(
-1000, "8604c70b43405ef9803cb49b77235ea2",
-100_000, "4c8bb3b99dcc59459b20fefba400d446",
-1_000_000, "9796db06e7a7960f974d5a91164afff1",
-100_000_000, "6ce05f456e2fdc846ded2abd0ec1de95");
+  public static IOTestConfig getTestConfigurationForConfigName(ConfigName 
name) {
+EnumMap configMap = new 
EnumMap<>(ConfigName.class);
+configMap.put(
+ConfigName.TEXT_1000_GZIP,
+new IOTestConfig("8604c70b43405ef9803cb49b77235ea2", 2564, 1000, 
Compression.GZIP));
+configMap.put(
+ConfigName.TEXT_1000_UNCOMPRESSED,
+new IOTestConfig("8604c70b43405ef9803cb49b77235ea2", 2564, 1000, 
Compression.GZIP));
 
 Review comment:
   I think there's an error - Compression.GZIP -> Compression.UNCOMPRESSED.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 32)
Time Spent: 50m  (was: 40m)

> Resize file-based IOITs 
> 
>
> Key: BEAM-8281
> URL: https://issues.apache.org/jira/browse/BEAM-8281
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Resize the IOITs to use known amounts of data saved on GCS so that it's 
> possible to report dataset size and measure throughput as part of the test 
> metrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8281) Resize file-based IOITs



 [ 
https://issues.apache.org/jira/browse/BEAM-8281?focusedWorklogId=327776&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327776
 ]

ASF GitHub Bot logged work on BEAM-8281:


Author: ASF GitHub Bot
Created on: 14/Oct/19 12:02
Start Date: 14/Oct/19 12:02
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on pull request #9638: [BEAM-8281] 
Resize IOITs datasets
URL: https://github.com/apache/beam/pull/9638#discussion_r334443378
 
 

 ##
 File path: 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOTestPipelineOptions.java
 ##
 @@ -48,4 +42,10 @@
   boolean getReportGcsPerformanceMetrics();
 
   void setReportGcsPerformanceMetrics(boolean performanceMetrics);
+
+  @Description("Test configuration name")
+  @Validation.Required
+  String getTestConfigName();
+
+  void setTestConfigName(String testConfig);
 
 Review comment:
   I think this can't be required. In general, IOITs should allow to:  
- be configured according to a specific "scenario" (this is what you're 
doing in this PR and this is good)
- be configured however user likes, according to some arbitrary scenario 
that isn't hardcoded
   
   Once we're forcing the config to be passed to the test, we loose the 2 
bullet from above. 
   
   On the other hand, I know that wasn't the case different test sizes because 
the hash was hardcoded. Maybe we should allow users passing their custom 
hashcode using pipeline options and specify fixed scenarios elsewhere (in code 
that invokes the tests, rather than implements them).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327776)
Time Spent: 50m  (was: 40m)

> Resize file-based IOITs 
> 
>
> Key: BEAM-8281
> URL: https://issues.apache.org/jira/browse/BEAM-8281
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Resize the IOITs to use known amounts of data saved on GCS so that it's 
> possible to report dataset size and measure throughput as part of the test 
> metrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-7370) Beam Dependency Update Request: Sphinx

[
https://issues.apache.org/jira/browse/BEAM-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7370:

Description:

- 2019-05-20 16:38:07.937770 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-17 12:32:27.855338 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-24 12:02:59.052884 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-01 12:04:13.113613 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-08 12:03:15.091005 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-15 12:03:09.406918 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-22 12:03:31.157859 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:05:13.023604 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-05 12:03:03.242767 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-12 12:04:01.647619 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-19 12:02:52.342008 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.2.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:02:44.918642 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.2.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:03:28.319578 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.2.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.or

[jira] [Updated] (BEAM-7369) Beam Dependency Update Request: mock

[
https://issues.apache.org/jira/browse/BEAM-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7369:

Description:

- 2019-05-20 16:37:42.331317 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-17 12:32:01.319061 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-24 12:02:34.500201 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-01 12:03:46.078139 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-08 12:02:33.351570 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-15 12:02:47.637435 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-22 12:03:10.835914 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:04:50.359555 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-05 12:02:38.989136 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-12 12:03:38.282286 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-19 12:02:30.405376 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:02:24.272597 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:03:07.834005 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/

[jira] [Work logged] (BEAM-8281) Resize file-based IOITs



 [ 
https://issues.apache.org/jira/browse/BEAM-8281?focusedWorklogId=327778&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327778
 ]

ASF GitHub Bot logged work on BEAM-8281:


Author: ASF GitHub Bot
Created on: 14/Oct/19 12:02
Start Date: 14/Oct/19 12:02
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on pull request #9638: [BEAM-8281] 
Resize IOITs datasets
URL: https://github.com/apache/beam/pull/9638#discussion_r334416029
 
 

 ##
 File path: 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
 ##
 @@ -45,15 +43,53 @@ public static String appendTimestampSuffix(String text) {
 return String.format("%s_%s", text, new Date().getTime());
   }
 
-  public static String getExpectedHashForLineCount(int lineCount) {
-Map expectedHashes =
-ImmutableMap.of(
-1000, "8604c70b43405ef9803cb49b77235ea2",
-100_000, "4c8bb3b99dcc59459b20fefba400d446",
-1_000_000, "9796db06e7a7960f974d5a91164afff1",
-100_000_000, "6ce05f456e2fdc846ded2abd0ec1de95");
+  public static IOTestConfig getTestConfigurationForConfigName(ConfigName 
name) {
 
 Review comment:
   I think this method should be stored in a place common for all IOITs, not 
only file-based ones. Perhaps we should move it to `IOTestConfig` class.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327778)
Time Spent: 50m  (was: 40m)

> Resize file-based IOITs 
> 
>
> Key: BEAM-8281
> URL: https://issues.apache.org/jira/browse/BEAM-8281
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Resize the IOITs to use known amounts of data saved on GCS so that it's 
> possible to report dataset size and measure throughput as part of the test 
> metrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-6089) Beam Dependency Update Request: oauth2client

[
https://issues.apache.org/jira/browse/BEAM-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-6089:

Description:

- 2018-11-19 12:11:53.801885 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:10:31.359164 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:11:18.194090 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:13:40.021791 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:12:09.88 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:20:14.935936 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:23:14.664558 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-14 12:12:14.081917 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-21 12:18:38.928775 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-28 12:10:22.371989 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-04 12:11:00.277439 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-11 12:11:25.422782 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-18 12:21:52.210498 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

[jira] [Updated] (BEAM-6645) Beam Dependency Update Request: com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin

[
https://issues.apache.org/jira/browse/BEAM-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-6645:

Description:

- 2019-02-11 12:12:17.710870 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.20.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-18 12:22:38.110512 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.20.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-25 12:12:13.220144 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.20.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-03-04 12:13:34.975335 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-03-11 12:13:20.550078 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-03-25 04:18:13.860760 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-01 12:09:02.686241 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-08 12:11:15.303458 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-15 12:30:47.279374 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-22 12:08:36.647085 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 16:38:23.480410 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|ht

[jira] [Updated] (BEAM-6951) Beam Dependency Update Request: com.github.spotbugs:spotbugs-annotations

[
https://issues.apache.org/jira/browse/BEAM-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-6951:

Description:

- 2019-04-01 12:15:05.460427 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-08 12:15:37.305259 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-15 12:35:52.817108 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-22 12:13:25.261372 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 16:39:18.034675 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 16:54:09.180503 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 17:37:40.326607 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-27 12:06:48.236836 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-03 12:04:13.166255 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-10 12:04:08.241217 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-17 12:33:32.545061 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-24 12:04:02.498856 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current ver

[jira] [Updated] (BEAM-7792) Beam Dependency Update Request: com.github.spotbugs:spotbugs

[
https://issues.apache.org/jira/browse/BEAM-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7792:

Description:

- 2019-07-22 12:15:41.216793 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:06:22.055418 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-05 12:03:56.940731 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-12 12:05:04.611973 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-19 12:03:13.035336 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:02:56.981076 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:03:52.780365 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-09 12:02:54.192199 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-16 12:02:39.812475 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-23 12:03:12.396932 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta4

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-30 12:04:13.122038 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta4

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-07 12:03:31.042633 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta4

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]

[jira] [Updated] (BEAM-5750) Beam Dependency Update Request: javax.servlet:javax.servlet-api

[
https://issues.apache.org/jira/browse/BEAM-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5750:

Description:

- 2018-10-15 12:13:14.253682 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:20.677351 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:03.954722 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:14:50.209888 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:14:50.401239 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:31.652290 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:29.396363 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:14:52.265627 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:21.331309 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:42.365110 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:01.296226 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:13.634593 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify

[jira] [Updated] (BEAM-8002) Beam Dependency Update Request: Gradle:

[
https://issues.apache.org/jira/browse/BEAM-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8002:

Description:

- 2019-08-19 12:04:19.192688 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:04:17.802062 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:04:53.057461 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-09 12:03:53.298241 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-16 12:03:47.037326 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-23 12:04:12.360299 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-30 12:05:19.680040 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-07 12:04:48.065365 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-14 12:04:37.533903 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-08-19 12:04:19.192688 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:04:17.802062 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:04:53.057461 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-09 12:03:53.298241 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://

[jira] [Updated] (BEAM-5748) Beam Dependency Update Request: org.conscrypt:conscrypt-openjdk

[
https://issues.apache.org/jira/browse/BEAM-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5748:

Description:

- 2018-10-15 12:11:29.496210 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:11:12.117107 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:13:29.155766 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:12:12.225263 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:12:03.483941 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:12:49.618981 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:11:54.540783 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:12:15.486522 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:14:35.731081 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:15:07.608265 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:21:22.979404 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:24:30.102231 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify

[jira] [Updated] (BEAM-5752) Beam Dependency Update Request: org.eclipse.jetty:jetty-server

[
https://issues.apache.org/jira/browse/BEAM-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5752:

Description:

- 2018-10-15 12:13:17.575546 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:26.908573 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:10.104039 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:14:56.891103 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:14:57.086704 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:38.454968 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:38.329092 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:15:03.778648 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:33.026807 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:54.249911 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:13.071360 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:25.376755 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

[jira] [Updated] (BEAM-5753) Beam Dependency Update Request: org.eclipse.jetty:jetty-servlet

[
https://issues.apache.org/jira/browse/BEAM-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5753:

Description:

- 2018-10-15 12:13:19.256622 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:32.240729 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:15.388249 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:15:02.810121 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:15:02.741564 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:44.462009 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:43.766710 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:15:09.151091 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:38.451204 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:59.925429 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:18.611955 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:30.751791 -

Please consider upgrading the dependency
org.eclipse.jetty:je

[jira] [Commented] (BEAM-7847) Generate Python SDK docs using Python 3

2019-10-14 Thread yoshiki obata (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950999#comment-16950999
 ] 

yoshiki obata commented on BEAM-7847:
-

Thank you, [~udim] and [~tvalentyn].
I'll prepare for PR and message to the mailing list.

> Generate Python SDK docs using Python 3 
> 
>
> Key: BEAM-7847
> URL: https://issues.apache.org/jira/browse/BEAM-7847
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Major
>
> Currently scripts/generate_pydoc.sh script fails on Python 3 with 
> "RuntimeError: empty_like method already has a docstring" errors:
> {noformat}
> pip install -e .[gcp,test]
> pip install Sphinx==1.6.5
> pip install sphinx_rtd_theme==0.2.4
> ./scripts/generate_pydoc.sh
> /home/valentyn/projects/beam/beam/beam/sdks/python/target/docs/source/apache_beam.testing.benchmarks.nexmark.queries.query0.rst:4:
>  WARNING: autodoc: failed to import module 
> 'apache_beam.testing.benchmarks.nexmark.queries.query0'; the following 
> exception was raised:
> Traceback (most recent call last):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/sphinx/ext/autodoc.py",
>  line 658, in import_object
> __import__(self.modname)
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/__init__.py", 
> line 98, in 
> from apache_beam import io
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/__init__.py",
>  line 22, in 
> from apache_beam.io.avroio import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/avroio.py",
>  line 61, in 
> from apache_beam.io import filebasedsink
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/filebasedsink.py",
>  line 34, in 
> from apache_beam.io import iobase
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/iobase.py",
>  line 50, in 
> from apache_beam.transforms import core
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/__init__.py",
>  line 29, in 
> from apache_beam.transforms.util import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 228, in 
> class _BatchSizeEstimator(object):
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 359, in _BatchSizeEstimator
> import numpy as np
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/__init__.py", 
> line 142, in 
> from . import core
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/__init__.py",
>  line 17, in 
> from . import multiarray
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/multiarray.py",
>  line 78, in 
> def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 203, in decorator
> docs_from_dispatcher=docs_from_dispatcher)(implementation)
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 159, in decorator
> add_docstring(implementation, dispatcher.__doc__)
> RuntimeError: empty_like method already has a docstring
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8393) Java BigQueryIO clustering support breaks on multiple partitions

2019-10-14 Thread Jeff Klukas (Jira)

Jeff Klukas created BEAM-8393:
-

 Summary: Java BigQueryIO clustering support breaks on multiple 
partitions
 Key: BEAM-8393
 URL: https://issues.apache.org/jira/browse/BEAM-8393
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Affects Versions: 2.15.0, 2.16.0
Reporter: Jeff Klukas
Assignee: Jeff Klukas


Support for writing to clustered tables in BigQuery was added in 2.15, which 
involved adding a new optional clustering field to TableDestination. Clustering 
support is working for most cases, but fails with errors about incompatible 
partitioning specifications for any data that is handled by the 
MultiplePartitions branch of BigQueryIO logic.

There is a case in that code path where we provide a modified TableDestination 
and neglect to copy the clustering definition, so the final load job does not 
include any clustering columns.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8393) Java BigQueryIO clustering support breaks on multiple partitions



 [ 
https://issues.apache.org/jira/browse/BEAM-8393?focusedWorklogId=327817&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327817
 ]

ASF GitHub Bot logged work on BEAM-8393:


Author: ASF GitHub Bot
Created on: 14/Oct/19 13:57
Start Date: 14/Oct/19 13:57
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9784: [BEAM-8393] 
Fix Java BigQueryIO clustering support for multiple partitions
URL: https://github.com/apache/beam/pull/9784
 
 
   Support for writing to clustered tables in BigQuery was added in 2.15,
   which involved adding a new optional clustering field to TableDestination.
   Clustering support is working for most cases, but fails with errors about
   incompatible partitioning specifications for any data that is handled by the
   MultiplePartitions branch of BigQueryIO logic.
   
   This PR fixes the bug by passing through the clustering specification in
   MatchTableDynamicDestinations.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python3

[jira] [Work logged] (BEAM-8393) Java BigQueryIO clustering support breaks on multiple partitions



 [ 
https://issues.apache.org/jira/browse/BEAM-8393?focusedWorklogId=327819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327819
 ]

ASF GitHub Bot logged work on BEAM-8393:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:02
Start Date: 14/Oct/19 14:02
Worklog Time Spent: 10m 
  Work Description: jklukas commented on issue #9784: [BEAM-8393] Fix Java 
BigQueryIO clustering support for multiple partitions
URL: https://github.com/apache/beam/pull/9784#issuecomment-541696374
 
 
   I have not added any additional test case here as I feel the bug and fix are 
both simple and obvious. Introducing a test case would involve potentially 
significant new test code to demonstrate the change; there is an existing case 
in `BigQueryIOWriteTest` that checks that a large collection of files is 
correctly separated into multiple partitions, but it doesn't actually call 
BigQueryIO to load a large number of files.
   
   Demonstrating that this change is working correctly would require actually 
running a load job into BigQuery over a large enough number of files to trigger 
multiple partitions. The effort and compute resource to run the test seem 
unnecessary for the low complexity of the change.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327819)
Time Spent: 20m  (was: 10m)

> Java BigQueryIO clustering support breaks on multiple partitions
> 
>
> Key: BEAM-8393
> URL: https://issues.apache.org/jira/browse/BEAM-8393
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Jeff Klukas
>Assignee: Jeff Klukas
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Support for writing to clustered tables in BigQuery was added in 2.15, which 
> involved adding a new optional clustering field to TableDestination. 
> Clustering support is working for most cases, but fails with errors about 
> incompatible partitioning specifications for any data that is handled by the 
> MultiplePartitions branch of BigQueryIO logic.
> There is a case in that code path where we provide a modified 
> TableDestination and neglect to copy the clustering definition, so the final 
> load job does not include any clustering columns.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8393) Java BigQueryIO clustering support breaks on multiple partitions



 [ 
https://issues.apache.org/jira/browse/BEAM-8393?focusedWorklogId=327824&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327824
 ]

ASF GitHub Bot logged work on BEAM-8393:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:04
Start Date: 14/Oct/19 14:04
Worklog Time Spent: 10m 
  Work Description: jklukas commented on issue #9784: [BEAM-8393] Fix Java 
BigQueryIO clustering support for multiple partitions
URL: https://github.com/apache/beam/pull/9784#issuecomment-541697628
 
 
   R: @chamikaramj @juancho088
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327824)
Time Spent: 0.5h  (was: 20m)

> Java BigQueryIO clustering support breaks on multiple partitions
> 
>
> Key: BEAM-8393
> URL: https://issues.apache.org/jira/browse/BEAM-8393
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Jeff Klukas
>Assignee: Jeff Klukas
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Support for writing to clustered tables in BigQuery was added in 2.15, which 
> involved adding a new optional clustering field to TableDestination. 
> Clustering support is working for most cases, but fails with errors about 
> incompatible partitioning specifications for any data that is handled by the 
> MultiplePartitions branch of BigQueryIO logic.
> There is a case in that code path where we provide a modified 
> TableDestination and neglect to copy the clustering definition, so the final 
> load job does not include any clustering columns.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=327839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327839
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:22
Start Date: 14/Oct/19 14:22
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-541708239
 
 
   Run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327839)
Time Spent: 7h 10m  (was: 7h)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=327844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327844
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:36
Start Date: 14/Oct/19 14:36
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-541717049
 
 
   Run BigQueryIO Batch Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327844)
Time Spent: 7h 20m  (was: 7h 10m)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=327845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327845
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:37
Start Date: 14/Oct/19 14:37
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-541717215
 
 
   Run BigQueryIO Stream Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327845)
Time Spent: 7.5h  (was: 7h 20m)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-6007) Create ClassLoadingStrategy with Java 11 compatible way



 [ 
https://issues.apache.org/jira/browse/BEAM-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy updated BEAM-6007:

Parent: BEAM-2530
Issue Type: Sub-task  (was: Improvement)

> Create ClassLoadingStrategy with Java 11 compatible way
> ---
>
> Key: BEAM-6007
> URL: https://issues.apache.org/jira/browse/BEAM-6007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Keisuke Kondo
>Assignee: Keisuke Kondo
>Priority: Minor
>
> Since sun.misc.Unsafe API was deprecated in Java 11,
>  
> {code:java}
> ClassLoadingStrategy.Default.INJECTION{code}
> is not work with Java 11.
> Author of byte-buddy library shared way to make it compatible with Java 11 on 
> his blog post. (and way to keep compatibility with java 9 and prior version)
> [http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-6007) Create ClassLoadingStrategy with Java 11 compatible way



[ 
https://issues.apache.org/jira/browse/BEAM-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951039#comment-16951039
 ] 

Lukasz Gajowy commented on BEAM-6007:
-

[~klonika99] will you be working on this? Did you have any progress that you 
could share?

> Create ClassLoadingStrategy with Java 11 compatible way
> ---
>
> Key: BEAM-6007
> URL: https://issues.apache.org/jira/browse/BEAM-6007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Keisuke Kondo
>Assignee: Keisuke Kondo
>Priority: Minor
>
> Since sun.misc.Unsafe API was deprecated in Java 11,
>  
> {code:java}
> ClassLoadingStrategy.Default.INJECTION{code}
> is not work with Java 11.
> Author of byte-buddy library shared way to make it compatible with Java 11 on 
> his blog post. (and way to keep compatibility with java 9 and prior version)
> [http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=327859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327859
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:42
Start Date: 14/Oct/19 14:42
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-541720456
 
 
   Run BigQueryIO Streaming Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327859)
Time Spent: 7h 40m  (was: 7.5h)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8106) Support Java 11 container



 [ 
https://issues.apache.org/jira/browse/BEAM-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy updated BEAM-8106:

Parent: BEAM-2530
Issue Type: Sub-task  (was: Task)

> Support Java 11 container
> -
>
> Key: BEAM-8106
> URL: https://issues.apache.org/jira/browse/BEAM-8106
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Hannah Jiang
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms



 [ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy resolved BEAM-3060.
-
Fix Version/s: Not applicable
   Resolution: Done

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8319) Errorprone 0.0.13 fails during JDK11 build



 [ 
https://issues.apache.org/jira/browse/BEAM-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy updated BEAM-8319:

Parent: BEAM-2530
Issue Type: Sub-task  (was: New Feature)

> Errorprone 0.0.13 fails during JDK11 build
> --
>
> Key: BEAM-8319
> URL: https://issues.apache.org/jira/browse/BEAM-8319
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> I'm using openjdk 1.11.02. After switching version to;
> {code:java}
> javaVersion = 11 {code}
> in BeamModule Plugin and running
> {code:java}
> ./gradlew clean build -p sdks/java/code -xtest {code}
> building fails. I was able to run errorprone after upgrading it but had 
> problems with conflicting guava version. See more here: 
> https://issues.apache.org/jira/browse/BEAM-5085
>  
> Stacktrace:
> {code:java}
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':model:pipeline:compileJava'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.accept(ExecuteActionsTaskExecuter.java:121)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.accept(ExecuteActionsTaskExecuter.java:117)
> at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:184)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ResolveIncrementalChangesTaskExecuter.execute(ResolveIncrementalChangesTaskExecuter.java:84)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.FinishSnapshotTaskInputsBuildOperationTaskExecuter.execute(FinishSnapshotTaskInputsBuildOperationTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:102)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.execute(ResolveBeforeExecutionStateTaskExecuter.java:74)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:109)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67)
> at 
> org.gradle.api.internal.tasks.execution.StartSnapshotTaskInputsBuildOperationTaskExecuter.execute(StartSnapshotTaskInputsBuildOperationTaskExecuter.java:52)
> at 
> org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93)
> at 
> org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:94)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:63)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
> at 
> org.gradle.internal.operations.DefaultBuildOperationEx

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=327874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327874
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 14/Oct/19 14:57
Start Date: 14/Oct/19 14:57
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-541729442
 
 
   Run BigQueryIO Streaming Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327874)
Time Spent: 7h 50m  (was: 7h 40m)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=327879&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327879
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 15:05
Start Date: 14/Oct/19 15:05
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541734038
 
 
   Run Python 2 PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327879)
Time Spent: 1h 20m  (was: 1h 10m)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=327890&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327890
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 14/Oct/19 15:22
Start Date: 14/Oct/19 15:22
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-541744007
 
 
   I fixed the quote escaping logic. WDYT, @lgajowy ?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327890)
Time Spent: 8h  (was: 7h 50m)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 8h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read



 [ 
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=327892&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327892
 ]

ASF GitHub Bot logged work on BEAM-8382:


Author: ASF GitHub Bot
Created on: 14/Oct/19 15:26
Start Date: 14/Oct/19 15:26
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9765: [BEAM-8382] 
Add polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-541746270
 
 
   @jfarr I guess it would not complicate things too much. My idea is the 
following - when we fetch new records in `ShardReadersPool.readLoop()` with 
`shardRecordsIterator.readNextBatch()` then it will finally end up in calling 
`SimplifiedKinesisClient.getRecords()` where we execute 
`AmazonKinesis.getRecords()` and wrap exceptions, if any. So, it looks like 
that we can re-throw `KMSThrottlingException` and catch it in 
`ShardReadersPool.readLoop()`. In case of this exception we can write warning 
in logs about that, wait with progressive timeout and recall 
`shardRecordsIterator.readNextBatch()` again. In this case no additional knobs 
well be needed.
   Wdyt?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327892)
Time Spent: 40m  (was: 0.5h)

> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to 
> ReadProvisionedThroughputExceeded on the order of hundreds of times per 
> second, regardless of the actual Kinesis throughput. This is because the 
> ShardReadersPool readLoop() method is polling getRecords() as fast as 
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds) 
> between calls to getRecords to avoid exceeding the limit on getRecords 
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read



 [ 
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=327914&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327914
 ]

ASF GitHub Bot logged work on BEAM-8382:


Author: ASF GitHub Bot
Created on: 14/Oct/19 15:38
Start Date: 14/Oct/19 15:38
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9765: [BEAM-8382] 
Add polling interval to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-541746270
 
 
   @jfarr I guess it would not complicate things too much. My idea is the 
following - when we fetch new records in `ShardReadersPool.readLoop()` with 
`shardRecordsIterator.readNextBatch()` then it will finally end up in calling 
`SimplifiedKinesisClient.getRecords()` where we execute 
`AmazonKinesis.getRecords()` and wrap exceptions, if any. So, it looks like 
that we can re-throw `KMSThrottlingException` and catch it in 
`ShardReadersPool.readLoop()`. In case of this exception we can write warning 
in logs about that, wait with progressive timeout (starting from 1 second) and 
recall `shardRecordsIterator.readNextBatch()` again. In this case no additional 
knobs well be needed.
   I'm a bit sceptical about user-defined config option in this case since we 
usually never know which value will be optimal and sufficient.
   Wdyt?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327914)
Time Spent: 50m  (was: 40m)

> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to 
> ReadProvisionedThroughputExceeded on the order of hundreds of times per 
> second, regardless of the actual Kinesis throughput. This is because the 
> ShardReadersPool readLoop() method is polling getRecords() as fast as 
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds) 
> between calls to getRecords to avoid exceeding the limit on getRecords 
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids



 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=327928&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327928
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 14/Oct/19 15:59
Start Date: 14/Oct/19 15:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #8915: [BEAM-4046] Remove 
old project name mappings.
URL: https://github.com/apache/beam/pull/8915#issuecomment-541764699
 
 
   I rebased this on top of master.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327928)
Time Spent: 42.5h  (was: 42h 20m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 42.5h
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=327931&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327931
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 14/Oct/19 16:03
Start Date: 14/Oct/19 16:03
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #9779: [BEAM-8388] 
Updating avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779#issuecomment-541767115
 
 
   Run Spark ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327931)
Remaining Estimate: 21.5h  (was: 21h 40m)
Time Spent: 2.5h  (was: 2h 20m)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 2.5h
>  Remaining Estimate: 21.5h
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



 [ 
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=327937&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327937
 ]

ASF GitHub Bot logged work on BEAM-8368:


Author: ASF GitHub Bot
Created on: 14/Oct/19 16:07
Start Date: 14/Oct/19 16:07
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #9768: [BEAM-8368] 
Avoid libprotobuf-generated exception when importing apache_beam
URL: https://github.com/apache/beam/pull/9768#issuecomment-541769468
 
 
   @aaltay should we go ahead and merge this prior to the 2.17.0 cut? We can 
update it if arrow 0.15.1 is out in time.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327937)
Time Spent: 1h  (was: 50m)

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8387) Remove sdk-worker-parallelism option from JobServerDriver

2019-10-14 Thread Thomas Weise (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-8387:
---
Status: Open  (was: Triage Needed)

> Remove sdk-worker-parallelism option from JobServerDriver
> -
>
> Key: BEAM-8387
> URL: https://issues.apache.org/jira/browse/BEAM-8387
> Project: Beam
>  Issue Type: Task
>  Components: runner-core
>Affects Versions: 2.16.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>
> The option was added when it wasn't possible to specify it as pipeline 
> option, which is no longer the case. The pipeline option has a value of 0, 
> which means that the runner should pick a suitable value. But this is then 
> overridden in FlinkJobInvoker with 1 (because that's the default in the  
> JobServerDriver config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8387) Remove sdk-worker-parallelism option from JobServerDriver



 [ 
https://issues.apache.org/jira/browse/BEAM-8387?focusedWorklogId=327946&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327946
 ]

ASF GitHub Bot logged work on BEAM-8387:


Author: ASF GitHub Bot
Created on: 14/Oct/19 16:22
Start Date: 14/Oct/19 16:22
Worklog Time Spent: 10m 
  Work Description: tweise commented on pull request #9785: [BEAM-8387] 
Remove sdk-worker-parallelism option from JobServerDriver
URL: https://github.com/apache/beam/pull/9785
 
 
   This option was added when it wasn't possible to specify the worker count as 
pipeline option, which is no longer the case. The pipeline option has a special 
value of 0, which means that the runner should pick a suitable value. 
Unfortunately this was overridden in FlinkJobInvoker with 1 (because that's the 
default in the  JobServerDriver config). There is no need to have this option 
in the job server.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/bea

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=327963&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327963
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 16:40
Start Date: 14/Oct/19 16:40
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on pull request #9764: [BEAM-8365] 
[WIP] Project push-down for TestTableProvider
URL: https://github.com/apache/beam/pull/9764#discussion_r334568107
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
 ##
 @@ -150,12 +165,40 @@ public BeamTableStatistics 
getTableStatistics(PipelineOptions options) {
   return begin.apply(Create.of(tableWithRows.rows).withCoder(rowCoder()));
 }
 
+@Override
+public PCollection buildIOReader(
+PBegin begin, BeamSqlTableFilter filters, List fieldNames) {
+  PCollection withAllFields = buildIOReader(begin);
+  if (options == PushDownOptions.NONE
+  || (fieldNames.isEmpty() && filters instanceof DefaultTableFilter)) {
+return withAllFields;
+  }
 
 Review comment:
   Some portion of this `if` statement can be moved to `BeamIOSourceRel`, more 
specifically the `Transform#expand` method.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327963)
Time Spent: 2.5h  (was: 2h 20m)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=327980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327980
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:07
Start Date: 14/Oct/19 17:07
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on pull request #9764: [BEAM-8365] 
[WIP] Project push-down for TestTableProvider
URL: https://github.com/apache/beam/pull/9764#discussion_r334568107
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
 ##
 @@ -150,12 +165,40 @@ public BeamTableStatistics 
getTableStatistics(PipelineOptions options) {
   return begin.apply(Create.of(tableWithRows.rows).withCoder(rowCoder()));
 }
 
+@Override
+public PCollection buildIOReader(
+PBegin begin, BeamSqlTableFilter filters, List fieldNames) {
+  PCollection withAllFields = buildIOReader(begin);
+  if (options == PushDownOptions.NONE
+  || (fieldNames.isEmpty() && filters instanceof DefaultTableFilter)) {
+return withAllFields;
+  }
 
 Review comment:
   Some portion of this `if` statement can be moved to `BeamIOSourceRel`, more 
specifically into the `Transform#expand` method.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327980)
Time Spent: 2h 40m  (was: 2.5h)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=327981&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327981
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:08
Start Date: 14/Oct/19 17:08
Worklog Time Spent: 10m 
  Work Description: apilloud commented on pull request #9743: [BEAM-8365] 
Project push-down for test table provider
URL: https://github.com/apache/beam/pull/9743
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327981)
Time Spent: 2h 50m  (was: 2h 40m)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)



 [ 
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=327982&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327982
 ]

ASF GitHub Bot logged work on BEAM-7389:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:09
Start Date: 14/Oct/19 17:09
Worklog Time Spent: 10m 
  Work Description: davidcavazos commented on pull request #9786: 
[BEAM-7389] Remove old element-wise snippets directory
URL: https://github.com/apache/beam/pull/9786
 
 
   I thought I included all the old `element-wise` files deleted in #9692 but 
apparently I missed to include some files. I'm removing them here.
   
   R: @aaltay 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [x] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds

[jira] [Work logged] (BEAM-3713) Consider moving away from nose to nose2 or pytest.



 [ 
https://issues.apache.org/jira/browse/BEAM-3713?focusedWorklogId=327986&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327986
 ]

ASF GitHub Bot logged work on BEAM-3713:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:22
Start Date: 14/Oct/19 17:22
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9756: [BEAM-3713] Add pytest 
for unit tests
URL: https://github.com/apache/beam/pull/9756#issuecomment-541807541
 
 
   @chadrik PTAL
   CC/R: @markflyhigh, @chamikaramj 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327986)
Time Spent: 8.5h  (was: 8h 20m)

> Consider moving away from nose to nose2 or pytest.
> --
>
> Key: BEAM-3713
> URL: https://issues.apache.org/jira/browse/BEAM-3713
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Per 
> [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,]
>  , nose is in maintenance mode.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=327987&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327987
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:27
Start Date: 14/Oct/19 17:27
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #9764: [BEAM-8365] [WIP] 
Project push-down for TestTableProvider
URL: https://github.com/apache/beam/pull/9764#issuecomment-541810496
 
 
   Run JavaBeamZetaSQL PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327987)
Time Spent: 3h  (was: 2h 50m)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=327988&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327988
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:28
Start Date: 14/Oct/19 17:28
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #9764: [BEAM-8365] [WIP] 
Project push-down for TestTableProvider
URL: https://github.com/apache/beam/pull/9764#issuecomment-541810496
 
 
   Run JavaBeamZetaSQL PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327988)
Time Spent: 3h 10m  (was: 3h)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3342) Create a Cloud Bigtable IO connector for Python



 [ 
https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=327994&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327994
 ]

ASF GitHub Bot logged work on BEAM-3342:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:30
Start Date: 14/Oct/19 17:30
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #8457: [BEAM-3342] 
Create a Cloud Bigtable IO connector for Python
URL: https://github.com/apache/beam/pull/8457#issuecomment-541811674
 
 
   Any updates here ?
   
   Thanks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 327994)
Time Spent: 40h 50m  (was: 40h 40m)

> Create a Cloud Bigtable IO connector for Python
> ---
>
> Key: BEAM-3342
> URL: https://issues.apache.org/jira/browse/BEAM-3342
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>Priority: Major
>  Time Spent: 40h 50m
>  Remaining Estimate: 0h
>
> I would like to create a Cloud Bigtable python connector.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

2019-10-14 Thread Kenneth Knowles (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951180#comment-16951180
 ] 

Kenneth Knowles commented on BEAM-2535:
---

[~rohdesam]

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the timer callback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3713) Consider moving away from nose to nose2 or pytest.



 [ 
https://issues.apache.org/jira/browse/BEAM-3713?focusedWorklogId=328007&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328007
 ]

ASF GitHub Bot logged work on BEAM-3713:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:38
Start Date: 14/Oct/19 17:38
Worklog Time Spent: 10m 
  Work Description: chadrik commented on pull request #9756: [BEAM-3713] 
Add pytest for unit tests
URL: https://github.com/apache/beam/pull/9756#discussion_r334590246
 
 

 ##
 File path: sdks/python/conftest.py
 ##
 @@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Pytest configuration and custom hooks."""
+
+from __future__ import absolute_import
+
+import sys
+
+# See pytest.ini for main collection rules.
+collect_ignore_glob = []
+if sys.version_info < (3,):
+  collect_ignore_glob.append('*_py3.py')
+for minor in [5, 6, 7, 8, 9]:
+  if sys.version_info < (3, minor):
+collect_ignore_glob.append('*_py3%d.py' % minor)
 
 Review comment:
   How about something like this:
   
   ```python
   MAX_SUPPORTED_PYTHON_VERSION = (3, 8)
   
   # See pytest.ini for main collection rules.
   collect_ignore_glob = []
   if sys.version_info < (3,):
 collect_ignore_glob.append('*_py3.py')
   else:
 for minor in range(sys.version_info.minor +1, 
MAX_SUPPORTED_PYTHON_VERSION[1] + 1):
   collect_ignore_glob.append('*_py3%d.py' % minor)
   ```
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328007)
Time Spent: 8h 40m  (was: 8.5h)

> Consider moving away from nose to nose2 or pytest.
> --
>
> Key: BEAM-3713
> URL: https://issues.apache.org/jira/browse/BEAM-3713
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> Per 
> [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,]
>  , nose is in maintenance mode.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8394) Support for Multiple DataSources in JdbcIO

2019-10-14 Thread Rahul Patwari (Jira)

Rahul Patwari created BEAM-8394:
---

 Summary: Support for Multiple DataSources in JdbcIO
 Key: BEAM-8394
 URL: https://issues.apache.org/jira/browse/BEAM-8394
 Project: Beam
  Issue Type: Task
  Components: io-java-jdbc
Reporter: Rahul Patwari
Assignee: Rahul Patwari


When a Pipeline consists of multiple databases, user should be able to provide 
DataSourceConfiguration for each database and  
DataSourceProviderFromDataSourceConfiguration should return the respective 
DataSources for each database when apply() method is called.

 

Also add withDataSourceConfiguration() method in JdbcIO.ReadRows class



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=328008&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328008
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:46
Start Date: 14/Oct/19 17:46
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #9764: [BEAM-8365] [WIP] 
Project push-down for TestTableProvider
URL: https://github.com/apache/beam/pull/9764#issuecomment-541820040
 
 
   Run JavaBeamZetaSQL PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328008)
Time Spent: 3h 20m  (was: 3h 10m)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8365) Add project push-down capability to IO APIs



 [ 
https://issues.apache.org/jira/browse/BEAM-8365?focusedWorklogId=328010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328010
 ]

ASF GitHub Bot logged work on BEAM-8365:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:46
Start Date: 14/Oct/19 17:46
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #9764: [BEAM-8365] [WIP] 
Project push-down for TestTableProvider
URL: https://github.com/apache/beam/pull/9764#issuecomment-541820040
 
 
   Run JavaBeamZetaSQL PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328010)
Time Spent: 3.5h  (was: 3h 20m)

> Add project push-down capability to IO APIs
> ---
>
> Key: BEAM-8365
> URL: https://issues.apache.org/jira/browse/BEAM-8365
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> * InMemoryTable should implement a following method:
> {code:java}
> public PCollection buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List fieldNames);{code}
> Which should return a `PCollection` with fields specified in `fieldNames` 
> list.
>  * Create a rule to push fields used by a Calc (in projects and in a 
> condition) down into TestTable IO.
>  * Updating that same Calc  (from previous step) to have a proper input and 
> output schemes, remove unused fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8394) Support for Multiple DataSources in JdbcIO



 [ 
https://issues.apache.org/jira/browse/BEAM-8394?focusedWorklogId=328014&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328014
 ]

ASF GitHub Bot logged work on BEAM-8394:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:48
Start Date: 14/Oct/19 17:48
Worklog Time Spent: 10m 
  Work Description: rahul8383 commented on pull request #9787: [BEAM-8394]: 
Support for Multiple DataSources in JdbcIO
URL: https://github.com/apache/beam/pull/9787
 
 
   Made changes in JdbcIO.java to support:
   When a Pipeline consists of multiple databases, the user should be able to 
provide DataSourceConfiguration for each database and 
DataSourceProviderFromDataSourceConfiguration should return the respective 
DataSources for each database when apply() method is called.
   
   Also added withDataSourceConfiguration() method in JdbcIO.ReadRows class
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCom

[jira] [Work logged] (BEAM-3342) Create a Cloud Bigtable IO connector for Python



 [ 
https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=328016&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328016
 ]

ASF GitHub Bot logged work on BEAM-3342:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:49
Start Date: 14/Oct/19 17:49
Worklog Time Spent: 10m 
  Work Description: mf2199 commented on issue #8457: [BEAM-3342] Create a 
Cloud Bigtable IO connector for Python
URL: https://github.com/apache/beam/pull/8457#issuecomment-541821885
 
 
   Hi Chamikara,
   
   Just forwarded you some details. It's close to 9pm now here in
   Saint-Petersburg, I’ll get back to it tomorrow.
   
   Max.
   
   
   
   
   пн, 14 окт. 2019 г., 20:30 Chamikara Jayalath :
   
   > Any updates here ?
   >
   > Thanks.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328016)
Time Spent: 41h  (was: 40h 50m)

> Create a Cloud Bigtable IO connector for Python
> ---
>
> Key: BEAM-3342
> URL: https://issues.apache.org/jira/browse/BEAM-3342
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>Priority: Major
>  Time Spent: 41h
>  Remaining Estimate: 0h
>
> I would like to create a Cloud Bigtable python connector.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=328024&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328024
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:56
Start Date: 14/Oct/19 17:56
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541825637
 
 
   Run PythonLint PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328024)
Time Spent: 1.5h  (was: 1h 20m)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=328026&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328026
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:57
Start Date: 14/Oct/19 17:57
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541826086
 
 
   Run Python 3 PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328026)
Time Spent: 1h 50m  (was: 1h 40m)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8395) JavaBeamZetaSQL PreCommit is flaky

Kirill Kozlov created BEAM-8395:
---

 Summary: JavaBeamZetaSQL PreCommit is flaky
 Key: BEAM-8395
 URL: https://issues.apache.org/jira/browse/BEAM-8395
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql-zetasql
Affects Versions: 2.15.0
Reporter: Kirill Kozlov


Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
*10:28:56* org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED*10:28:56* java.lang.NoSuchMethodError at 
ZetaSQLDialectSpecTest.java:2553*10:28:56* 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED*10:28:56* 
java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-1440) Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-1440?focusedWorklogId=328025&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328025
 ]

ASF GitHub Bot logged work on BEAM-1440:


Author: ASF GitHub Bot
Created on: 14/Oct/19 17:57
Start Date: 14/Oct/19 17:57
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9772: [BEAM-1440] Create a 
BigQuery source that implements iobase.BoundedSource for Python
URL: https://github.com/apache/beam/pull/9772#issuecomment-541826086
 
 
   Run Python 3 PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328025)
Time Spent: 1h 40m  (was: 1.5h)

> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK
> --
>
> Key: BEAM-1440
> URL: https://issues.apache.org/jira/browse/BEAM-1440
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Currently we have a BigQuery native source for Python SDK [1].
> This can only be used by Dataflow runner.
> We should  implement a Beam BigQuery source that implements 
> iobase.BoundedSource [2] interface so that other runners that try to use 
> Python SDK can read from BigQuery as well. Java SDK already has a Beam 
> BigQuery source [3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py
> [2] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L70
> [3] 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1189



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8395) JavaBeamZetaSQL PreCommit is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Kozlov updated BEAM-8395:

Description: 
Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
{code:java}
10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED
10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553

10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED
10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError{code}
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]

  was:
Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*

 
{code:java}
10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED
10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553

10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED
10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError{code}
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

 

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]


> JavaBeamZetaSQL PreCommit is flaky
> --
>
> Key: BEAM-8395
> URL: https://issues.apache.org/jira/browse/BEAM-8395
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql-zetasql
>Affects Versions: 2.15.0
>Reporter: Kirill Kozlov
>Priority: Major
>
> Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
> {code:java}
> 10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
> testExceptAll FAILED
> 10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553
> 10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
> testZetaSQLStructFieldAccessInTumble FAILED
> 10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272
> ... More of java.lang.NoClassDefFoundError{code}
> Jenkins that failed: 
> [https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]
> Jenkins that passed: 
> [https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8395) JavaBeamZetaSQL PreCommit is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Kozlov updated BEAM-8395:

Description: 
Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*

 
{code:java}
10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED
10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553

10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED
10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError{code}
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

 

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]

  was:
Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
*10:28:56* org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED*10:28:56* java.lang.NoSuchMethodError at 
ZetaSQLDialectSpecTest.java:2553*10:28:56* 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED*10:28:56* 
java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]


> JavaBeamZetaSQL PreCommit is flaky
> --
>
> Key: BEAM-8395
> URL: https://issues.apache.org/jira/browse/BEAM-8395
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql-zetasql
>Affects Versions: 2.15.0
>Reporter: Kirill Kozlov
>Priority: Major
>
> Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
>  
> {code:java}
> 10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
> testExceptAll FAILED
> 10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553
> 10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
> testZetaSQLStructFieldAccessInTumble FAILED
> 10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272
> ... More of java.lang.NoClassDefFoundError{code}
> Jenkins that failed: 
> [https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]
>  
> Jenkins that passed: 
> [https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



 [ 
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=328040&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328040
 ]

ASF GitHub Bot logged work on BEAM-8368:


Author: ASF GitHub Bot
Created on: 14/Oct/19 18:27
Start Date: 14/Oct/19 18:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #9768: [BEAM-8368] 
Avoid libprotobuf-generated exception when importing apache_beam
URL: https://github.com/apache/beam/pull/9768
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328040)
Time Spent: 1h 20m  (was: 1h 10m)

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8368) [Python] libprotobuf-generated exception when importing apache_beam



 [ 
https://issues.apache.org/jira/browse/BEAM-8368?focusedWorklogId=328039&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328039
 ]

ASF GitHub Bot logged work on BEAM-8368:


Author: ASF GitHub Bot
Created on: 14/Oct/19 18:27
Start Date: 14/Oct/19 18:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #9768: [BEAM-8368] Avoid 
libprotobuf-generated exception when importing apache_beam
URL: https://github.com/apache/beam/pull/9768#issuecomment-541841831
 
 
   > @aaltay should we go ahead and merge this prior to the 2.17.0 cut? We can 
update it if arrow 0.15.1 is out in time.
   
   Yes, thank you. Is arrow 0.15.1 fixing this issue?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328039)
Time Spent: 1h 10m  (was: 1h)

> [Python] libprotobuf-generated exception when importing apache_beam
> ---
>
> Key: BEAM-8368
> URL: https://issues.apache.org/jira/browse/BEAM-8368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.15.0, 2.17.0
>Reporter: Ubaier Bhat
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.17.0
>
> Attachments: error_log.txt
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Unable to import apache_beam after upgrading to macos 10.15 (Catalina). 
> Cleared all the pipenvs and but can't get it working again.
> {code}
> import apache_beam as beam
> /Users/***/.local/share/virtualenvs/beam-etl-ims6DitU/lib/python3.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: Some syntactic constructs of Python 3 are not yet fully 
> supported by Apache Beam.
>   'Some syntactic constructs of Python 3 are not yet fully supported by '
> [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already 
> exists in database: 
> [libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> libc++abi.dylib: terminating with uncaught exception of type 
> google::protobuf::FatalException: CHECK failed: 
> GeneratedDatabase()->Add(encoded_file_descriptor, size): 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)



 [ 
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=328043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328043
 ]

ASF GitHub Bot logged work on BEAM-7389:


Author: ASF GitHub Bot
Created on: 14/Oct/19 18:33
Start Date: 14/Oct/19 18:33
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #9786: [BEAM-7389] 
Remove old element-wise snippets directory
URL: https://github.com/apache/beam/pull/9786
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328043)
Time Spent: 65h 20m  (was: 65h 10m)

> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
>  Time Spent: 65h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8394) Support for Multiple DataSources in JdbcIO



 [ 
https://issues.apache.org/jira/browse/BEAM-8394?focusedWorklogId=328049&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328049
 ]

ASF GitHub Bot logged work on BEAM-8394:


Author: ASF GitHub Bot
Created on: 14/Oct/19 18:38
Start Date: 14/Oct/19 18:38
Worklog Time Spent: 10m 
  Work Description: rahul8383 commented on issue #9787: [BEAM-8394]: 
Support for Multiple DataSources in JdbcIO
URL: https://github.com/apache/beam/pull/9787#issuecomment-541847998
 
 
   R: @jbonofre 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328049)
Time Spent: 20m  (was: 10m)

> Support for Multiple DataSources in JdbcIO
> --
>
> Key: BEAM-8394
> URL: https://issues.apache.org/jira/browse/BEAM-8394
> Project: Beam
>  Issue Type: Task
>  Components: io-java-jdbc
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When a Pipeline consists of multiple databases, user should be able to 
> provide DataSourceConfiguration for each database and  
> DataSourceProviderFromDataSourceConfiguration should return the respective 
> DataSources for each database when apply() method is called.
>  
> Also add withDataSourceConfiguration() method in JdbcIO.ReadRows class



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8395) JavaBeamZetaSQL PreCommit is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Kozlov updated BEAM-8395:

Description: 
Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
{code:java}
10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED
10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553

10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED
10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError{code}
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]

 

Stack trace:
{code:java}
java.lang.NoSuchMethodError: 
com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom([Ljava/lang/String;[Lcom/google/protobuf/Descriptors$FileDescriptor;)Lcom/google/protobuf/Descriptors$FileDescriptor;Close
 stack trace

at 
com.google.zetasql.functions.ZetaSQLDateTime.(ZetaSQLDateTime.java:508)

at 
com.google.zetasql.functions.ZetaSQLDateTime$DateTimestampPart.getDescriptor(ZetaSQLDateTime.java:460)

at 
com.google.zetasql.SimpleCatalog.processGetBuiltinFunctionsResponse(SimpleCatalog.java:377)

at com.google.zetasql.SimpleCatalog.addZetaSQLFunctions(SimpleCatalog.java:365)

at 
org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.addBuiltinFunctionsToCatalog(SqlAnalyzer.java:152)

at 
org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.createPopulatedCatalog(SqlAnalyzer.java:136)

at 
org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.analyze(SqlAnalyzer.java:90)

at 
org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer$Builder.analyze(SqlAnalyzer.java:281)

at 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:136)

at 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:92)

at 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:87)

at 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:66)

at 
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest.testExceptAll(ZetaSQLDialectSpecTest.java:2553){code}

  was:
Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
{code:java}
10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testExceptAll FAILED
10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553

10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
testZetaSQLStructFieldAccessInTumble FAILED
10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272

... More of java.lang.NoClassDefFoundError{code}
Jenkins that failed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]

Jenkins that passed: 
[https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]


> JavaBeamZetaSQL PreCommit is flaky
> --
>
> Key: BEAM-8395
> URL: https://issues.apache.org/jira/browse/BEAM-8395
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql-zetasql
>Affects Versions: 2.15.0
>Reporter: Kirill Kozlov
>Priority: Major
>
> Occasionally fails on task: *Task :sdks:java:extensions:sql:zetasql:test*
> {code:java}
> 10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
> testExceptAll FAILED
> 10:28:56 java.lang.NoSuchMethodError at ZetaSQLDialectSpecTest.java:2553
> 10:28:56 org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLDialectSpecTest > 
> testZetaSQLStructFieldAccessInTumble FAILED
> 10:28:56 java.lang.NoClassDefFoundError at ZetaSQLDialectSpecTest.java:1272
> ... More of java.lang.NoClassDefFoundError{code}
> Jenkins that failed: 
> [https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/8/]
> Jenkins that passed: 
> [https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/9/]
>  
> Stack trace:
> {code:java}
> java.lang.NoSuchMethodError: 
> com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom([Ljava/lang/String;[Lcom/google/protobuf/Descriptors$FileDescriptor;)Lcom/google/protobuf/Descriptors$FileDescriptor;Close
>  stack trace
> at 
> com.google.zetasql.functions.ZetaSQLDateTime.(ZetaSQLDateTime.java:508)
> at 
> com.google.zetasql.functions.ZetaSQLDateTime$DateTimestampPart.getDescriptor(ZetaSQLDateTime.java:460)
> at 
> com.google.zetasql.SimpleCatalog.processGetBuiltinFunctionsResponse(SimpleCatalog.java:377)
> at 
> com.google.zetasql.SimpleCatalog.addZetaSQLFunctions(SimpleCatalog.java:365)
> at 
> org.apache.beam.sdk.extensi

[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)



 [ 
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=328058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328058
 ]

ASF GitHub Bot logged work on BEAM-7389:


Author: ASF GitHub Bot
Created on: 14/Oct/19 19:01
Start Date: 14/Oct/19 19:01
Worklog Time Spent: 10m 
  Work Description: davidcavazos commented on pull request #9788: 
[BEAM-7389] Add links to docs for easier navigation
URL: https://github.com/apache/beam/pull/9788
 
 
   Add links to docs for easier navigation. If a user lands on the GitHub code 
pages, have a way to redirect them to the docs for more information.
   
   R: @aaltay 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [x] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds

[jira] [Work logged] (BEAM-8394) Support for Multiple DataSources in JdbcIO



 [ 
https://issues.apache.org/jira/browse/BEAM-8394?focusedWorklogId=328060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328060
 ]

ASF GitHub Bot logged work on BEAM-8394:


Author: ASF GitHub Bot
Created on: 14/Oct/19 19:05
Start Date: 14/Oct/19 19:05
Worklog Time Spent: 10m 
  Work Description: rahul8383 commented on issue #9787: [BEAM-8394]: 
Support for Multiple DataSources in JdbcIO
URL: https://github.com/apache/beam/pull/9787#issuecomment-541862241
 
 
   R: @iemejia 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 328060)
Time Spent: 0.5h  (was: 20m)

> Support for Multiple DataSources in JdbcIO
> --
>
> Key: BEAM-8394
> URL: https://issues.apache.org/jira/browse/BEAM-8394
> Project: Beam
>  Issue Type: Task
>  Components: io-java-jdbc
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When a Pipeline consists of multiple databases, user should be able to 
> provide DataSourceConfiguration for each database and  
> DataSourceProviderFromDataSourceConfiguration should return the respective 
> DataSources for each database when apply() method is called.
>  
> Also add withDataSourceConfiguration() method in JdbcIO.ReadRows class



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)