date:20200120



 [ 
https://issues.apache.org/jira/browse/BEAM-7427?focusedWorklogId=374836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374836
 ]

ASF GitHub Bot logged work on BEAM-7427:


Author: ASF GitHub Bot
Created on: 21/Jan/20 06:49
Start Date: 21/Jan/20 06:49
Worklog Time Spent: 10m 
  Work Description: jbonofre commented on issue #8757: [BEAM-7427] Fix 
JmsCheckpointMark Avro Encoding
URL: https://github.com/apache/beam/pull/8757#issuecomment-576542230
 
 
   I'm evaluating the following options:
   0. Add a test with AvroCoder on checkpoint mark
   1. use JmsRecord in the checkpoint instead of Message to ack. The issue is 
how to "load" Message from JmsRecord.
   2. use Session to ack instead of message. The Session stores the unack 
messages, so the checkpoint mark could delegate to the session.
   
   I'm evaluating the different options. I keep you posted soon.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374836)
Time Spent: 3h 10m  (was: 3h)

> JmsCheckpointMark Avro Serialization issue with UnboundedSource
> ---
>
> Key: BEAM-7427
> URL: https://issues.apache.org/jira/browse/BEAM-7427
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-jms
>Affects Versions: 2.12.0
> Environment: Message Broker : solace
> JMS Client (Over AMQP) : "org.apache.qpid:qpid-jms-client:0.42.0
>Reporter: Mourad
>Assignee: Mourad
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> I get the following exception when reading from unbounded JMS Source:
>   
> {code:java}
> Caused by: org.apache.avro.SchemaParseException: Illegal character in: this$0
> at org.apache.avro.Schema.validateName(Schema.java:1151)
> at org.apache.avro.Schema.access$200(Schema.java:81)
> at org.apache.avro.Schema$Field.(Schema.java:403)
> at org.apache.avro.Schema$Field.(Schema.java:396)
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:622)
> at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:740)
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:604)
> at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:218)
> at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:215)
> at 
> avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at 
> avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at 
> avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at 
> avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> {code}
>  
> The exception is thrown by Avro when introspecting {{JmsCheckpointMark}} to 
> generate schema.
> JmsIO config :
>  
> {code:java}
> PCollection messages = pipeline.apply("read messages from the 
> events broker", JmsIO.readMessage() 
> .withConnectionFactory(jmsConnectionFactory) .withTopic(options.getTopic()) 
> .withMessageMapper(new DFAMessageMapper()) 
> .withCoder(AvroCoder.of(DFAMessage.class)));
> {code}
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core

2020-01-20 Thread Tomo Suzuki (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019845#comment-17019845
 ] 

Tomo Suzuki commented on BEAM-9144:
---

[~atdixon] That's an interesting error. Is it possible to find the JAR file 
containing the class file which has reference to the problematic 
"org.apache.beam.vendor.grpc.v1p21p0" package?

As I couldn't find the package in Beam's master branch. I suspect that an old 
Beam JAR file sneaked into your class path from some dependency. Revealing who 
(a class file in a JAR file) contains the reference helps to verify that.

Once the problematic JAR file is identified, you can run "mvn dependency:tree" 
or "./gradlew dependencies" to identify the dependency tree.


> Beam's own Avro TimeConversion class in beam-sdk-java-core 
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> From Aaron's comment in 
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
>  .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues 
> resolved in 1.9.x) so downgrading Avro is not possible for us.
>  Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to 
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are 
> not available in 1.9.x.
> {quote}
> The Java class is 
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
>  It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of 
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide 
> flexibility in runtime Avro versions (as it did until Beam 2.16).
> h2. Difference of the TimeConversion Classes
> Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method. 
> Details below:
> Avro 1.8's TimeConversions.TimestampConversion:
> {code:java}
>   public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
>   return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
>   return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType 
> type) {
>   return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
>   return timestamp.getMillis();
> }
>   }
> {code}
> Avro 1.9's JodaTimeConversions.TimestampConversion:
> {code:java}
>   public static class TimestampConversion extends Conversion {
> @Override
> public Class getConvertedType() {
>   return DateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
>   return "timestamp-millis";
> }
> @Override
> public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType 
> type) {
>   return new DateTime(millisFromEpoch, DateTimeZone.UTC);
> }
> @Override
> public Long toLong(DateTime timestamp, Schema schema, LogicalType type) {
>   return timestamp.getMillis();
> }
> @Override
> public Schema getRecommendedSchema() {
>   return 
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7746) Add type hints to python code



 [ 
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=374799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374799
 ]

ASF GitHub Bot logged work on BEAM-7746:


Author: ASF GitHub Bot
Created on: 21/Jan/20 04:12
Start Date: 21/Jan/20 04:12
Worklog Time Spent: 10m 
  Work Description: chadrik commented on issue #10594: [BEAM-7746] Stop 
automatically creating staticmethods in register_urn
URL: https://github.com/apache/beam/pull/10594#issuecomment-576510338
 
 
   Looks like the code changed a bit since I originally wrote this change.
   Will fix these and look for anything else I may have missed.
   
   
   On Fri, Jan 17, 2020 at 4:15 PM Udi Meiri  wrote:
   
   > *@udim* commented on this pull request.
   > --
   >
   > In sdks/python/apache_beam/coders/coders.py
   > :
   >
   > > @@ -1014,6 +1015,7 @@ def to_runner_api_parameter(self, context):
   >  else:
   >return super(TupleCoder, self).to_runner_api_parameter(context)
   >
   > +  @staticmethod
   >@Coder.register_urn(common_urns.coders.KV.urn, None)
   >def from_runner_api_parameter(unused_payload, components, 
unused_context):
   >  return TupleCoder(components)
   >
   > Should ParamWindowedValueCoder.from_runner_api_parameter also get the
   > @staticmethod decorator?
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374799)
Time Spent: 54h  (was: 53h 50m)

> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 54h
>  Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484 
> type hints so that I can clearly see what types are required, get completion 
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here:  [https://github.com/apache/beam/pull/9056]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8684) Beam Dependency Update Request: com.google.apis:google-api-services-bigquery



 [ 
https://issues.apache.org/jira/browse/BEAM-8684?focusedWorklogId=374794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374794
 ]

ASF GitHub Bot logged work on BEAM-8684:


Author: ASF GitHub Bot
Created on: 21/Jan/20 03:49
Start Date: 21/Jan/20 03:49
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10631: [BEAM-8684] Google 
proto and google-services library upgrades
URL: https://github.com/apache/beam/pull/10631#issuecomment-576506609
 
 
   com.google.cloud:google-cloud-bigquery:1.102.0 needs google-cloud-core 1.91.0
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374794)
Time Spent: 3h  (was: 2h 50m)

> Beam Dependency Update Request: com.google.apis:google-api-services-bigquery
> 
>
> Key: BEAM-8684
> URL: https://issues.apache.org/jira/browse/BEAM-8684
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:07.113511 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:04:31.460554 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:09:35.656962 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:08:45.501496 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:08:46.385699 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:04:15.902178 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:07:51.708019 
> -
> Please consider upgrading the dependency 
> com.google.apis:google-api-services-bigquery. 
> The current version is v2-rev20181221-1.28.0. The latest version is 
> v2-rev20190917-1.30.3 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:07:45.384750 
>

[jira] [Work logged] (BEAM-9120) Deprecate onSuccessMatcher, onCreateMatcher



 [ 
https://issues.apache.org/jira/browse/BEAM-9120?focusedWorklogId=374763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374763
 ]

ASF GitHub Bot logged work on BEAM-9120:


Author: ASF GitHub Bot
Created on: 21/Jan/20 00:53
Start Date: 21/Jan/20 00:53
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #10587: [BEAM-9120] 
Make WordCountsMatcher extend TypeSafeMatcher>
URL: https://github.com/apache/beam/pull/10587#issuecomment-576473738
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374763)
Time Spent: 1h 10m  (was: 1h)

> Deprecate onSuccessMatcher, onCreateMatcher
> ---
>
> Key: BEAM-9120
> URL: https://issues.apache.org/jira/browse/BEAM-9120
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Instead of creating matchers on PipelineResult we should just make assertions 
> on real matchers after waiting for the pipeline to finish.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source



 [ 
https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=374730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374730
 ]

ASF GitHub Bot logged work on BEAM-9072:


Author: ASF GitHub Bot
Created on: 20/Jan/20 22:13
Start Date: 20/Jan/20 22:13
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10440: [BEAM-9072] [SQL] 
DataStoreV1 IO connector
URL: https://github.com/apache/beam/pull/10440#issuecomment-576445504
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374730)
Time Spent: 5h  (was: 4h 50m)

> [SQL] Add support for Datastore source
> --
>
> Key: BEAM-9072
> URL: https://issues.apache.org/jira/browse/BEAM-9072
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> * Create a Datastore table and table provider
>  * Conversion between Datastore and Beam data types
>  * Implement buildIOReader
>  * Implement buildIOWrite
>  * Implement getTableStatistics
> Doc: 
> [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9160) Update AWS SDK to support Pod Level Identity

2020-01-20 Thread Mohamed Nouh (Jira)

Mohamed Nouh created BEAM-9160:
--

 Summary: Update AWS SDK to support Pod Level Identity
 Key: BEAM-9160
 URL: https://issues.apache.org/jira/browse/BEAM-9160
 Project: Beam
  Issue Type: Improvement
  Components: dependencies
Affects Versions: 2.17.0
Reporter: Mohamed Nouh


Many organizations have started leveraging pod level identity in Kubernetes. 
The current version of the AWS SDK packaged with Bean 2.17.0 is out of date and 
doesn't provide native support to pod level identity access management.

 

It is recommended that we introduce support to access AWS resources such as S3 
using pod level identity. 

Current Version of the AWS Java SDK in Beam:

def aws_java_sdk_version = "1.11.519"

Proposed AWS Java SDK Version:


 com.amazonaws
 aws-java-sdk
 1.11.710




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9046) Kafka connector for Python throws ClassCastException when reading KafkaRecord

2020-01-20 Thread Maximilian Michels (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019752#comment-17019752
 ] 

Maximilian Michels commented on BEAM-9046:
--

Are you running this against a Beam 2.13.0 Expansion / Job server? At this 
point your setup should be identical to mine :)

> Kafka connector for Python throws ClassCastException when reading KafkaRecord
> -
>
> Key: BEAM-9046
> URL: https://issues.apache.org/jira/browse/BEAM-9046
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-kafka
>Affects Versions: 2.16.0
>Reporter: Berkay Öztürk
>Priority: Major
>  Labels: KafkaIO, Python
> Fix For: Not applicable
>
>
>  I'm trying to read the data streaming from Apache Kafka using the Python SDK 
> for Apache Beam with the Flink runner. After running Kafka 2.4.0 and Flink 
> 1.8.3, I follow these steps:
>  * Compile and run Beam 2.16 with Flink 1.8 runner.
> {code:java}
> git clone --single-branch --branch release-2.16.0 
> https://github.com/apache/beam.git beam-2.16.0
> cd beam-2.16.0
> nohup ./gradlew :runners:flink:1.8:job-server:runShadow 
> -PflinkMasterUrl=localhost:8081 &
> {code}
>  * Run the Python pipeline.
> {code:python}
> from apache_beam import Pipeline
> from apache_beam.io.external.kafka import ReadFromKafka
> from apache_beam.options.pipeline_options import PipelineOptions
> if __name__ == '__main__':
> with Pipeline(options=PipelineOptions([
> '--runner=FlinkRunner',
> '--flink_version=1.8',
> '--flink_master_url=localhost:8081',
> '--environment_type=LOOPBACK',
> '--streaming'
> ])) as pipeline:
> (
> pipeline
> | 'read' >> ReadFromKafka({'bootstrap.servers': 
> 'localhost:9092'}, ['test'])  # [BEAM-3788] ???
> )
> result = pipeline.run()
> result.wait_until_finish()
> {code}
>  * Publish some data to Kafka.
> {code:java}
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> >{"hello":"world!"}
> {code}
> The Python script throws this error:
> {code:java}
> [flink-runner-job-invoker] ERROR 
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Error 
> during job invocation BeamApp-USER-somejob. 
> org.apache.flink.client.program.ProgramInvocationException: Job failed. 
> (JobID: xxx)
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:483)
> at 
> org.apache.beam.runners.flink.FlinkExecutionEnvironments$BeamFlinkRemoteStreamEnvironment.executeRemotely(FlinkExecutionEnvironments.java:360)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:310)
> at 
> org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator$StreamingTranslationContext.execute(FlinkStreamingPortablePipelineTranslator.java:173)
> at 
> org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:104)
> at 
> org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:80)
> at 
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:78)
> at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
> at 
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
> ... 13 more
> Caused by: java.lang.ClassCastException: 
> org.apache.beam.sdk.io.kafka.KafkaRecord cannot be cast to [B
> at 
> org.apache.beam.sdk.coders.ByteArrayCoder.encode(ByteArrayCoder.java:41)
> at 
> org.apache.beam.sdk.coders.LengthPrefixCoder.encode(LengthPrefixCoder.java:56)
> at 
>

[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.



 [ 
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=374717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374717
 ]

ASF GitHub Bot logged work on BEAM-9061:


Author: ASF GitHub Bot
Created on: 20/Jan/20 20:19
Start Date: 20/Jan/20 20:19
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on pull request #10619: [BEAM-9061] 
Add version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10619
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374717)
Time Spent: 3.5h  (was: 3h 20m)

> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
> 
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> {noformat}
> 08:35:26   File "apache_beam/runners/portability/fn_api_runner_test.py", line 
> 38, in 
> 08:35:26 import hamcrest  # pylint: disable=ungrouped-imports
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
>  line 2, in 
> 08:35:26 from hamcrest.library import *
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
>  line 7, in 
> 08:35:26 from hamcrest.library.object import *
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
>  line 4, in 
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
>  line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.



 [ 
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=374716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374716
 ]

ASF GitHub Bot logged work on BEAM-9061:


Author: ASF GitHub Bot
Created on: 20/Jan/20 20:19
Start Date: 20/Jan/20 20:19
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #10619: [BEAM-9061] Add 
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10619#issuecomment-576418311
 
 
   Thanks Valentyn! I'm going to merge it. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374716)
Time Spent: 3h 20m  (was: 3h 10m)

> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
> 
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> {noformat}
> 08:35:26   File "apache_beam/runners/portability/fn_api_runner_test.py", line 
> 38, in 
> 08:35:26 import hamcrest  # pylint: disable=ungrouped-imports
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
>  line 2, in 
> 08:35:26 from hamcrest.library import *
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
>  line 7, in 
> 08:35:26 from hamcrest.library.object import *
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
>  line 4, in 
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
>  line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness

2020-01-20 Thread Boyuan Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019717#comment-17019717
 ] 

Boyuan Zhang commented on BEAM-8618:


Thanks for your update!

> Tear down unused DoFns periodically in Python SDK harness
> -
>
> Key: BEAM-8618
> URL: https://issues.apache.org/jira/browse/BEAM-8618
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
>
> Per the discussion in the ML, detail can be found [1],  the teardown of DoFns 
> should be supported in the portability framework. It happens at two places:
> 1) Upon the control service termination
> 2) Tear down the unused DoFns periodically
> The aim of this JIRA is to add support for tear down the unused DoFns 
> periodically in Python SDK harness.
> [1] 
> https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9123) HadoopResourceId returns wrong directory name



 [ 
https://issues.apache.org/jira/browse/BEAM-9123?focusedWorklogId=374712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374712
 ]

ASF GitHub Bot logged work on BEAM-9123:


Author: ASF GitHub Bot
Created on: 20/Jan/20 19:58
Start Date: 20/Jan/20 19:58
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on pull request #10626: 
[release-2.19.0][BEAM-9123] HadoopResourceId returns wrong directoryName bugfix
URL: https://github.com/apache/beam/pull/10626
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374712)
Time Spent: 3h 20m  (was: 3h 10m)

> HadoopResourceId returns wrong directory name
> -
>
> Key: BEAM-9123
> URL: https://issues.apache.org/jira/browse/BEAM-9123
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop-file-system
>Affects Versions: 2.17.0
>Reporter: Marek Simunek
>Assignee: Marek Simunek
>Priority: Major
> Fix For: 2.18.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
>  _HadoopResourceId_ returns for directory uri wrong fileName() which doesnt 
> conform _ResourceId_ interface javadoc
> {color:#629755}Returns the name of the file or directory..{color}
> String {color:#ffc66d}getFilename{color}(){color:#cc7832}; {color}
>  
> {code:java}
> URI hdfsClusterBaseUri = new URI(configuration.get("fs.defaultFS") + "/");
> HadoopResourceId hadoopResourceId =
> new HadoopResourceId(hdfsClusterBaseUri.resolve("/dirA/file1"));{code}
> Expected behavior: 
> _hadoopResourceId.getCurrentDirectory().getFilename()_ returns *dirA*
> Actual behavior:
>  _hadoopResourceId.getCurrentDirectory().getFilename()_ returns *""*
> The problem is that uri for folder is ending with '*/*' and  
> [getFileName()|https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java#L68]
>  is using hadoop Path.getName() which returns everything after last slash.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9123) HadoopResourceId returns wrong directory name



 [ 
https://issues.apache.org/jira/browse/BEAM-9123?focusedWorklogId=374711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374711
 ]

ASF GitHub Bot logged work on BEAM-9123:


Author: ASF GitHub Bot
Created on: 20/Jan/20 19:57
Start Date: 20/Jan/20 19:57
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #10626: 
[release-2.19.0][BEAM-9123] HadoopResourceId returns wrong directoryName bugfix
URL: https://github.com/apache/beam/pull/10626#issuecomment-576411898
 
 
   Java_Examples_Dataflow fails for tests misconfiguration: 
https://github.com/apache/beam/pull/10635
   
   I'm going to merge this commit into release branch. Thanks for your help!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374711)
Time Spent: 3h 10m  (was: 3h)

> HadoopResourceId returns wrong directory name
> -
>
> Key: BEAM-9123
> URL: https://issues.apache.org/jira/browse/BEAM-9123
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop-file-system
>Affects Versions: 2.17.0
>Reporter: Marek Simunek
>Assignee: Marek Simunek
>Priority: Major
> Fix For: 2.18.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
>  _HadoopResourceId_ returns for directory uri wrong fileName() which doesnt 
> conform _ResourceId_ interface javadoc
> {color:#629755}Returns the name of the file or directory..{color}
> String {color:#ffc66d}getFilename{color}(){color:#cc7832}; {color}
>  
> {code:java}
> URI hdfsClusterBaseUri = new URI(configuration.get("fs.defaultFS") + "/");
> HadoopResourceId hadoopResourceId =
> new HadoopResourceId(hdfsClusterBaseUri.resolve("/dirA/file1"));{code}
> Expected behavior: 
> _hadoopResourceId.getCurrentDirectory().getFilename()_ returns *dirA*
> Actual behavior:
>  _hadoopResourceId.getCurrentDirectory().getFilename()_ returns *""*
> The problem is that uri for folder is ending with '*/*' and  
> [getFileName()|https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java#L68]
>  is using hadoop Path.getName() which returns everything after last slash.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7699) Remove duplicated null checks in Row.equals



 [ 
https://issues.apache.org/jira/browse/BEAM-7699?focusedWorklogId=374692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374692
 ]

ASF GitHub Bot logged work on BEAM-7699:


Author: ASF GitHub Bot
Created on: 20/Jan/20 18:34
Start Date: 20/Jan/20 18:34
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #9009: [BEAM-7699] 
Propagate null check/comparison to deepEquals
URL: https://github.com/apache/beam/pull/9009#issuecomment-576389078
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374692)
Time Spent: 2h 20m  (was: 2h 10m)

> Remove duplicated null checks in Row.equals
> ---
>
> Key: BEAM-7699
> URL: https://issues.apache.org/jira/browse/BEAM-7699
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Naheon Kim
>Priority: Trivial
>  Labels: easyfix, starter
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> After [BEAM-5866|https://issues.apache.org/jira/browse/BEAM-5866] and 
> [BEAM-7125|https://issues.apache.org/jira/browse/BEAM-7125] Row has 
> duplicated null value checks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (BEAM-9046) Kafka connector for Python throws ClassCastException when reading KafkaRecord



[ 
https://issues.apache.org/jira/browse/BEAM-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019670#comment-17019670
 ] 

Berkay Öztürk edited comment on BEAM-9046 at 1/20/20 6:18 PM:
--

[~mxm] I think I'm getting there. I have built the Docker containers and 
downgraded to apache_beam 2.13.0:

 
{code:java}
Traceback (most recent call last):
  File "test.py", line 27, in 
p.run()
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 419, in run
return self.runner.run_pipeline(self, self._options)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/portability/portable_runner.py", line 186, in 
run_pipeline
portable_options))
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 650, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/pipeline_context.py", line 84, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in to_runner_api
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in 
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/pipeline_context.py", line 84, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in to_runner_api
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in 
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/pipeline_context.py", line 84, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 856, in to_runner_api
return self.transform.to_runner_api_transform(context, self.full_label)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/transforms/external.py", line 180, in 
to_runner_api_transform
id, context.coders._id_to_proto[id], proto))
RuntimeError: Re-used coder id: ref_Coder_TupleCoder_1
spec {
  spec {
urn: "beam:coder:kv:v1"
  }
  environment_id: "ref_Environment_default_environment_1"
}
component_coder_ids: "ref_Coder_BytesCoder_2"
component_coder_ids: "ref_Coder_BytesCoder_2"spec {
  spec {
urn: "beam:coder:kv:v1"
  }
}
component_coder_ids: "ref_Coder_StrUtf8Coder_2"
component_coder_ids: "ref_Coder_StrUtf8Coder_2"{code}
Code (Taken from your Beam Summit NA 2019 presentation):

 
{code:java}
pipeline_options = PipelineOptions(['--job_name=Cross-Language-Demo', 
'--runner=PortableRunner', '--job_endpoint=localhost:8099', '--parallelism=1'])

p = beam.Pipeline(options=pipeline_options)

def to_upper(kv):
k, v = kv
return str(k).upper(), str(v).upper()

(p
| ReadFromKafka(consumer_config={'bootstrap.servers': 'localhost:9092', 
'auto.offset.reset': 'latest'}, topics=['test'])
| beam.Map(lambda kv: to_upper(kv)).with_output_types(KV[str, str])
| WriteToKafka(producer_config={'bootstrap.servers': 'localhost:9092'}, 
topic='test-out')
)

p.run()
{code}
 

 


was (Author: berkayozturk):
[~mxm] I think I'm getting there. I have built

[jira] [Work logged] (BEAM-2535) Allow explicit output time independent of firing specification for all timers



 [ 
https://issues.apache.org/jira/browse/BEAM-2535?focusedWorklogId=374686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374686
 ]

ASF GitHub Bot logged work on BEAM-2535:


Author: ASF GitHub Bot
Created on: 20/Jan/20 18:17
Start Date: 20/Jan/20 18:17
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10627: [BEAM-2535] 
Support outputTimestamp and watermark holds in processing timers.
URL: https://github.com/apache/beam/pull/10627#issuecomment-576384782
 
 
   R: @reuvenlax 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374686)
Time Spent: 9h 50m  (was: 9h 40m)

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the timer callback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9046) Kafka connector for Python throws ClassCastException when reading KafkaRecord



[ 
https://issues.apache.org/jira/browse/BEAM-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019670#comment-17019670
 ] 

Berkay Öztürk commented on BEAM-9046:
-

[~mxm] I think I'm getting there. I have built the Docker containers and 
downgraded to apache_beam 2.13.0:

 
{code:java}
Traceback (most recent call last):
  File "test.py", line 27, in 
p.run()
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 419, in run
return self.runner.run_pipeline(self, self._options)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/portability/portable_runner.py", line 186, in 
run_pipeline
portable_options))
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 650, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/pipeline_context.py", line 84, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in to_runner_api
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in 
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/pipeline_context.py", line 84, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in to_runner_api
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 871, in 
for part in self.parts],
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/runners/pipeline_context.py", line 84, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/pipeline.py", line 856, in to_runner_api
return self.transform.to_runner_api_transform(context, self.full_label)
  File "/home/USER/beam213/lib/python3.7/site-packages/apache   

  _beam/transforms/external.py", line 180, in 
to_runner_api_transform
id, context.coders._id_to_proto[id], proto))
RuntimeError: Re-used coder id: ref_Coder_TupleCoder_1
spec {
  spec {
urn: "beam:coder:kv:v1"
  }
  environment_id: "ref_Environment_default_environment_1"
}
component_coder_ids: "ref_Coder_BytesCoder_2"
component_coder_ids: "ref_Coder_BytesCoder_2"spec {
  spec {
urn: "beam:coder:kv:v1"
  }
}
component_coder_ids: "ref_Coder_StrUtf8Coder_2"
component_coder_ids: "ref_Coder_StrUtf8Coder_2"{code}
Code (Taken from your Beam Summit NA 2019 presentation):

 
{code:java}
pipeline_options = PipelineOptions(['--job_name=Cross-Language-Demo', 
'--runner=PortableRunner', '--job_endpoint=localhost:8099', '--parallelism=1', 
'--kafka_bootstrap=localhost:9092'])

p = beam.Pipeline(options=pipeline_options)

def to_upper(kv):
k, v = kv
return str(k).upper(), str(v).upper()

(p
| ReadFromKafka(consumer_config={'bootstrap.servers': 'localhost:9092', 
'auto.offset.reset': 'latest'}, topics=['test'])
| beam.Map(lambda kv: to_upper(kv)).with_output_types(KV[str, str])
| WriteToKafka(producer_config={'bootstrap.servers': 'localhost:9092'}, 
topic='test-out')
)

p.run()
{code}
 

 

> Kafka connector for Python throws ClassCastException when reading KafkaRecord
>

[jira] [Updated] (BEAM-9008) Add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9008:
---
Summary: Add readAll() method to CassandraIO  (was: add readAll() method to 
CassandraIO)

> Add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374664
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:40
Start Date: 20/Jan/20 16:40
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10546: [BEAM-9008] add 
CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#issuecomment-576352218
 
 
   Btw, don't hesitate to ask questions or discuss here if you have any doubts 
before starting.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374664)
Time Spent: 2h 10m  (was: 2h)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374654
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368623996
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
 ##
 @@ -608,121 +470,32 @@ static double getRingFraction(List 
tokenRanges) {
   return ringFraction;
 }
 
-/**
- * Check if the current partitioner is the Murmur3 (default in Cassandra 
version newer than 2).
- */
-@VisibleForTesting
-static boolean isMurmur3Partitioner(Cluster cluster) {
-  return MURMUR3PARTITIONER.equals(cluster.getMetadata().getPartitioner());
-}
-
 /** Measure distance between two tokens. */
 @VisibleForTesting
 static BigInteger distance(BigInteger left, BigInteger right) {
   return (right.compareTo(left) > 0)
   ? right.subtract(left)
   : 
right.subtract(left).add(SplitGenerator.getRangeSize(MURMUR3PARTITIONER));
 }
+  }
 
-/**
- * Represent a token range in Cassandra instance, wrapping the partition 
count, size and token
- * range.
- */
-@VisibleForTesting
-static class TokenRange {
-  private final long partitionCount;
-  private final long meanPartitionSize;
-  private final BigInteger rangeStart;
-  private final BigInteger rangeEnd;
-
-  TokenRange(
-  long partitionCount, long meanPartitionSize, BigInteger rangeStart, 
BigInteger rangeEnd) {
-this.partitionCount = partitionCount;
-this.meanPartitionSize = meanPartitionSize;
-this.rangeStart = rangeStart;
-this.rangeEnd = rangeEnd;
-  }
-}
-
-private class CassandraReader extends BoundedSource.BoundedReader {
-  private final CassandraIO.CassandraSource source;
-  private Cluster cluster;
-  private Session session;
-  private Iterator iterator;
-  private T current;
-
-  CassandraReader(CassandraSource source) {
-this.source = source;
-  }
-
-  @Override
-  public boolean start() {
-LOG.debug("Starting Cassandra reader");
-cluster =
-getCluster(
-source.spec.hosts(),
-source.spec.port(),
-source.spec.username(),
-source.spec.password(),
-source.spec.localDc(),
-source.spec.consistencyLevel());
-session = cluster.connect(source.spec.keyspace().get());
-LOG.debug("Queries: " + source.splitQueries);
-List futures = new ArrayList<>();
-for (String query : source.splitQueries) {
-  futures.add(session.executeAsync(query));
-}
-
-final Mapper mapper = getMapper(session, source.spec.entity());
-
-for (ResultSetFuture result : futures) {
-  if (iterator == null) {
-iterator = mapper.map(result.getUninterruptibly());
-  } else {
-iterator = Iterators.concat(iterator, 
mapper.map(result.getUninterruptibly()));
-  }
-}
-
-return advance();
-  }
-
-  @Override
-  public boolean advance() {
-if (iterator.hasNext()) {
-  current = iterator.next();
-  return true;
-}
-current = null;
-return false;
-  }
-
-  @Override
-  public void close() {
-LOG.debug("Closing Cassandra reader");
-if (session != null) {
-  session.close();
-}
-if (cluster != null) {
-  cluster.close();
-}
-  }
-
-  @Override
-  public T getCurrent() throws NoSuchElementException {
-if (current == null) {
-  throw new NoSuchElementException();
-}
-return current;
-  }
-
-  @Override
-  public CassandraIO.CassandraSource getCurrentSource() {
-return source;
-  }
-
-  private Mapper getMapper(Session session, Class enitity) {
-return source.spec.mapperFactoryFn().apply(session);
-  }
+  /**
+   * Represent a token range in Cassandra instance, wrapping the partition 
count, size and token
+   * range.
+   */
+  @VisibleForTesting
+  static class TokenRange {
 
 Review comment:
   We don't need this one anymore I think we can remove this class now as well 
as: `CassandraIO.getRingFraction` and `CassandraIOTest.testRingFraction`. Can 
you remove these too please.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374662
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368626298
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/RingRange.java
 ##
 @@ -17,10 +17,11 @@
  */
 package org.apache.beam.sdk.io.cassandra;
 
+import java.io.Serializable;
 import java.math.BigInteger;
 
 /** Models a Cassandra token range. */
-final class RingRange {
+final class RingRange implements Serializable {
 
 Review comment:
   Actually this should probably be done not for this class but for the 
`CassandraQuery` object
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374662)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374647
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r367997934
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
 ##
 @@ -1146,4 +918,297 @@ private void waitForFuturesToFinish() throws 
ExecutionException, InterruptedExce
   }
 }
   }
+
+  /**
+   * A {@link PTransform} to read data from Apache Cassandra. See {@link 
CassandraIO} for more
+   * information on usage and configuration.
+   */
+  @AutoValue
+  public abstract static class ReadAll
+  extends PTransform, PCollection> {
+@Nullable
+abstract ValueProvider> hosts();
+
+@Nullable
+abstract ValueProvider query();
+
+@Nullable
+abstract ValueProvider port();
+
+@Nullable
+abstract ValueProvider keyspace();
+
+@Nullable
+abstract ValueProvider table();
+
+@Nullable
+abstract Class entity();
+
+@Nullable
+abstract Coder coder();
+
+@Nullable
+abstract ValueProvider username();
+
+@Nullable
+abstract ValueProvider password();
+
+@Nullable
+abstract ValueProvider localDc();
+
+@Nullable
+abstract ValueProvider consistencyLevel();
+
+@Nullable
+abstract ValueProvider splitCount();
+
+@Nullable
+abstract SerializableFunction mapperFactoryFn();
+
+@Nullable
+abstract SerializableFunction groupingFn();
+
+abstract Builder builder();
+
+/** Specify the hosts of the Apache Cassandra instances. */
+public ReadAll withHosts(List hosts) {
+  checkArgument(hosts != null, "hosts can not be null");
+  checkArgument(!hosts.isEmpty(), "hosts can not be empty");
+  return withHosts(ValueProvider.StaticValueProvider.of(hosts));
+}
+
+/** Specify the hosts of the Apache Cassandra instances. */
+public ReadAll withHosts(ValueProvider> hosts) {
+  return builder().setHosts(hosts).build();
+}
+
+/** Specify the port number of the Apache Cassandra instances. */
+public ReadAll withPort(int port) {
+  checkArgument(port > 0, "port must be > 0, but was: %s", port);
+  return withPort(ValueProvider.StaticValueProvider.of(port));
+}
+
+/** Specify the port number of the Apache Cassandra instances. */
+public ReadAll withPort(ValueProvider port) {
+  return builder().setPort(port).build();
+}
+
+/** Specify the Cassandra keyspace where to read data. */
+public ReadAll withKeyspace(String keyspace) {
+  checkArgument(keyspace != null, "keyspace can not be null");
+  return withKeyspace(ValueProvider.StaticValueProvider.of(keyspace));
+}
+
+/** Specify the Cassandra keyspace where to read data. */
+public ReadAll withKeyspace(ValueProvider keyspace) {
+  return builder().setKeyspace(keyspace).build();
+}
+
+/** Specify the Cassandra table where to read data. */
+public ReadAll withTable(String table) {
+  checkArgument(table != null, "table can not be null");
+  return withTable(ValueProvider.StaticValueProvider.of(table));
+}
+
+/** Specify the Cassandra table where to read data. */
+public ReadAll withTable(ValueProvider table) {
+  return builder().setTable(table).build();
+}
+
+/** Specify the query to read data. */
+public ReadAll withQuery(String query) {
+  checkArgument(query != null && query.length() > 0, "query cannot be 
null");
+  return withQuery(ValueProvider.StaticValueProvider.of(query));
+}
+
+/** Specify the query to read data. */
+public ReadAll withQuery(ValueProvider query) {
+  return builder().setQuery(query).build();
+}
+
+/**
+ * Specify the entity class (annotated POJO). The {@link CassandraIO} will 
read the data and
+ * convert the data as entity instances. The {@link PCollection} resulting 
from the read will
+ * contains entity elements.
+ */
+public ReadAll withEntity(Class entity) {
+  checkArgument(entity != null, "entity can not be null");
+  return builder().setEntity(entity).build();
+}
+
+/** Specify the {@link Coder} used to serialize the entity in the {@link 
PCollection}. */
+public ReadAll withCoder(Coder coder) {
+  checkArgument(coder != null, "coder can not be null");
+  return builder().setCoder(coder).build();
+}
+
+/** Specify the username for authentication. */
+public ReadAll withUsername(String username) {
+  checkArgument(username != null, "username can not be null");
+

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374648
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r367999012
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/QueryFn.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ColumnMetadata;
+import com.datastax.driver.core.PreparedStatement;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.Token;
+import java.util.Iterator;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class QueryFn extends DoFn, T> {
+
+  private static final Logger LOG = LoggerFactory.getLogger(CassandraIO.class);
 
 Review comment:
   Replace `CassandraIO.class` with the new `ReadFn.class`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374648)
Time Spent: 1h 20m  (was: 1h 10m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374653=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374653
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368626645
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
 ##
 @@ -387,6 +420,7 @@ public Mapper apply(Session input) {
 
 @Override
 public Iterator map(ResultSet resultSet) {
+
 
 Review comment:
   nit: remove space
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374653)
Time Spent: 1h 40m  (was: 1.5h)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374659
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368627136
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
 ##
 @@ -466,40 +500,9 @@ public void testCustomMapperImplDelete() {
 assertEquals(1, counter.intValue());
   }
 
-  @Test
-  public void testSplit() throws Exception {
-PipelineOptions options = PipelineOptionsFactory.create();
 
 Review comment:
   Once we have the SplitFn we should probably test this, actually we had a lot 
of historic issues with Splitting on this connector so please pay attention to 
ensure we don't introduce regressions.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374659)
Time Spent: 1h 50m  (was: 1h 40m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374652
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368013640
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
 ##
 @@ -732,6 +505,35 @@ public T getCurrent() throws NoSuchElementException {
 DELETE
   }
 
+  /**
+   * Check if the current partitioner is the Murmur3 (default in Cassandra 
version newer than 2).
+   */
+  @VisibleForTesting
+  static boolean isMurmur3Partitioner(Cluster cluster) {
+return MURMUR3PARTITIONER.equals(cluster.getMetadata().getPartitioner());
+  }
+
+  static String generateRangeQuery(CassandraConfig spec, String partitionKey) {
 
 Review comment:
   Can you move this one to `QueryFn` since that's the only place where it is 
used.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374652)
Time Spent: 1h 40m  (was: 1.5h)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374661
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368622445
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
 ##
 @@ -325,7 +359,58 @@ private CassandraIO() {}
   checkArgument(entity() != null, "withEntity() is required");
   checkArgument(coder() != null, "withCoder() is required");
 
-  return input.apply(org.apache.beam.sdk.io.Read.from(new 
CassandraSource<>(this, null)));
+  try (Cluster cluster =
+  getCluster(hosts(), port(), username(), password(), localDc(), 
consistencyLevel())) {
+if (isMurmur3Partitioner(cluster)) {
+  LOG.info("Murmur3Partitioner detected, splitting");
+
+  List tokens =
 
 Review comment:
   For maintenance coherence in DoFn based IOs we follow a pattern where the 
first part Creates the required object to query (the one I mentioned above), 
the second splits the query into multiples queries for each partition via a 
`SplitFn extends DoFn` and then with the result of such splits we do a read 
equivalent to current `QueryFn`. Can you refactor this segment to behave like 
that (with the query object I mentioned above).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374661)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374656
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368626515
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
 ##
 @@ -312,6 +298,52 @@ public void testRead() throws Exception {
 pipeline.run();
   }
 
+  @Test
+  public void testReadAll() {
+
 
 Review comment:
   nit: remove space
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374656)
Time Spent: 1h 50m  (was: 1h 40m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374657
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r367999517
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/QueryFn.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ColumnMetadata;
+import com.datastax.driver.core.PreparedStatement;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.Token;
+import java.util.Iterator;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class QueryFn extends DoFn, T> {
+
+  private static final Logger LOG = LoggerFactory.getLogger(CassandraIO.class);
+
+  private final CassandraConfig read;
+
+  private transient Cluster cluster;
+
+  private transient Session session;
+
+  private String partitionKey;
+
+  public QueryFn(CassandraConfig read) {
+this.read = read;
+  }
+
+  @Setup
+  public void setup() {
+this.cluster =
+CassandraIO.getCluster(
+read.hosts(),
+read.port(),
+read.username(),
+read.password(),
+read.localDc(),
+read.consistencyLevel());
+this.session = this.cluster.connect(read.keyspace().get());
+this.partitionKey =
+
cluster.getMetadata().getKeyspace(read.keyspace().get()).getTable(read.table().get())
+.getPartitionKey().stream()
+.map(ColumnMetadata::getName)
+.collect(Collectors.joining(","));
+  }
+
+  public void teardown() {
+this.session.close();
+this.cluster.close();
+  }
+
+  @ProcessElement
+  public void processElement(@Element Iterable tokens, 
OutputReceiver receiver) {
+
 
 Review comment:
   nit: remove space
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374657)
Time Spent: 1h 50m  (was: 1h 40m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374658
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368619172
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraConfig.java
 ##
 @@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Session;
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+
+@AutoValue
+abstract class CassandraConfig implements Serializable {
+  @Nullable
+  abstract ValueProvider> hosts();
+
+  @Nullable
+  abstract ValueProvider query();
 
 Review comment:
   We should divide this object into two: One that has the parameters we need 
to query elements and other that has just the connection part. The goal is that 
the query part will be sufficient to do a read and to be splitted if necessary, 
with this we might have a richer set of options than just with `RingRange` for 
example to have ReadAll query multiple tables. This object probably should 
include: query, keyspace, table, entity and RingRange.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374658)
Time Spent: 1h 50m  (was: 1h 40m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously*

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374651
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r367999232
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/QueryFn.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ColumnMetadata;
+import com.datastax.driver.core.PreparedStatement;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.Token;
+import java.util.Iterator;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class QueryFn extends DoFn, T> {
+
+  private static final Logger LOG = LoggerFactory.getLogger(CassandraIO.class);
+
+  private final CassandraConfig read;
+
+  private transient Cluster cluster;
+
+  private transient Session session;
+
+  private String partitionKey;
+
+  public QueryFn(CassandraConfig read) {
 
 Review comment:
   Remove public
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374651)
Time Spent: 1.5h  (was: 1h 20m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374655
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368000993
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/RingRange.java
 ##
 @@ -17,10 +17,11 @@
  */
 package org.apache.beam.sdk.io.cassandra;
 
+import java.io.Serializable;
 import java.math.BigInteger;
 
 /** Models a Cassandra token range. */
-final class RingRange {
+final class RingRange implements Serializable {
 
 Review comment:
   Since RingRange may end up been compared for equality too can you please add 
equals/hashcode too. And add a test that tests that Serializability and 
equality works in Beam we have the 
`SerializableUtils.ensureSerializableRoundTrip` that should make it.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374655)
Time Spent: 1h 50m  (was: 1h 40m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374649
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368011332
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/QueryFn.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ColumnMetadata;
+import com.datastax.driver.core.PreparedStatement;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.Token;
+import java.util.Iterator;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class QueryFn extends DoFn, T> {
 
 Review comment:
   this can be the flatted version no? `DoFn` or actually the new 
Query objectr I mention above `DoFn`.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374649)
Time Spent: 1.5h  (was: 1h 20m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374646
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r367998534
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/QueryFn.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ColumnMetadata;
+import com.datastax.driver.core.PreparedStatement;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.Token;
+import java.util.Iterator;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class QueryFn extends DoFn, T> {
 
 Review comment:
   remove public and rename to ReadFn since the goal here would be just to Read
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374646)
Time Spent: 1h 10m  (was: 1h)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374660
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r368013780
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
 ##
 @@ -732,6 +505,35 @@ public T getCurrent() throws NoSuchElementException {
 DELETE
   }
 
+  /**
+   * Check if the current partitioner is the Murmur3 (default in Cassandra 
version newer than 2).
+   */
+  @VisibleForTesting
+  static boolean isMurmur3Partitioner(Cluster cluster) {
+return MURMUR3PARTITIONER.equals(cluster.getMetadata().getPartitioner());
+  }
+
+  static String generateRangeQuery(CassandraConfig spec, String partitionKey) {
+final String rangeFilter =
+Joiner.on(" AND ")
+.skipNulls()
+.join(
+String.format("(token(%s) >= ?)", partitionKey),
+String.format("(token(%s) < ?)", partitionKey));
+final String query =
+(spec.query() == null)
+? buildQuery(spec) + " WHERE " + rangeFilter
+: buildQuery(spec) + " AND " + rangeFilter;
+LOG.debug("CassandraIO generated query : {}", query);
+return query;
+  }
+
+  private static String buildQuery(CassandraConfig spec) {
 
 Review comment:
   Same for this one, move to `QueryFn`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374660)
Time Spent: 2h  (was: 1h 50m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9008) add readAll() method to CassandraIO



 [ 
https://issues.apache.org/jira/browse/BEAM-9008?focusedWorklogId=374650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374650
 ]

ASF GitHub Bot logged work on BEAM-9008:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:36
Start Date: 20/Jan/20 16:36
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10546: [BEAM-9008] 
add CassandraIO readAll method
URL: https://github.com/apache/beam/pull/10546#discussion_r365717276
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraConfig.java
 ##
 @@ -0,0 +1,60 @@
+package org.apache.beam.sdk.io.cassandra;
+
+import com.datastax.driver.core.Session;
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+
+@AutoValue
+abstract class CassandraConfig implements Serializable {
+  @Nullable
+  abstract ValueProvider> hosts();
+
+  @Nullable
+  abstract ValueProvider query();
+
+  @Nullable
+  abstract ValueProvider port();
+
+  @Nullable
+  abstract ValueProvider keyspace();
+
+  @Nullable
+  abstract ValueProvider table();
+
+  @Nullable
+  abstract ValueProvider username();
+
+  @Nullable
+  abstract ValueProvider password();
+
+  @Nullable
+  abstract ValueProvider localDc();
+
+  @Nullable
+  abstract ValueProvider consistencyLevel();
+
+  @Nullable
+  abstract SerializableFunction mapperFactoryFn();
+
+  @Nullable
+  abstract Class entity();
+
+  public static  CassandraConfig Create(ValueProvider> 
hosts,
 
 Review comment:
   lowercase c
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374650)
Time Spent: 1.5h  (was: 1h 20m)

> add readAll() method to CassandraIO
> ---
>
> Key: BEAM-9008
> URL: https://issues.apache.org/jira/browse/BEAM-9008
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-cassandra
>Affects Versions: 2.16.0
>Reporter: vincent marquez
>Assignee: vincent marquez
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When querying a large cassandra database, it's often *much* more useful to 
> programatically generate the queries needed to to be run rather than reading 
> all partitions and attempting some filtering.  
> As an example:
> {code:java}
> public class Event { 
>@PartitionKey(0) public UUID accountId;
>@PartitionKey(1)public String yearMonthDay; 
>@ClusteringKey public UUID eventId;  
>//other data...
> }{code}
> If there is ten years worth of data, you may want to only query one year's 
> worth.  Here each token range would represent one 'token' but all events for 
> the day. 
> {code:java}
> Set accounts = getRelevantAccounts();
> Set dateRange = generateDateRange("2018-01-01", "2019-01-01");
> PCollection tokens = generateTokens(accounts, dateRange); 
> {code}
>  
>  I propose an additional _readAll()_ PTransform that can take a PCollection 
> of token ranges and can return a PCollection of what the query would 
> return. 
> *Question: How much code should be in common between both methods?* 
> Currently the read connector already groups all partitions into a List of 
> Token Ranges, so it would be simple to refactor the current read() based 
> method to a 'ParDo' based one and have them both share the same function.  
> Reasons against sharing code between read and readAll
>  * Not having the read based method return a BoundedSource connector would 
> mean losing the ability to know the size of the data returned
>  * Currently the CassandraReader executes all the grouped TokenRange queries 
> *asynchronously* which is (maybe?) fine when all that's happening is 
> splitting up all the partition ranges but terrible for executing potentially 
> millions of queries. 
>  Reasons _for_ sharing code would be simplified code base and that both of 
> the above issues would most likely have a negligable performance impact. 
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core

2020-01-20 Thread Aaron Dixon (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019629#comment-17019629
 ] 

Aaron Dixon edited comment on BEAM-9144 at 1/20/20 4:34 PM:


Attempting to validate with 2.20.0-SNAPSHOT [1]

My pipeline however is unable to start in Dataflow, I get:

{noformat}
java.lang.ClassNotFoundException: 
org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message$Builder
{noformat}

When I inspect my build artifacts I find that when I depend on Beam 2.16.0 
(last Beam build that works for me), I find this class present (v1p21p0). When 
I build against Beam 2.20.0-SNAPSHOT, only v1p26p0 is resolved and packaged.

* I assume that the Dataflow runner needs to somehow know to look for v1p26p0 
somehow?  I.e., do I need to send it a flag that I'm suing 2.20.0-SNAPSHOT so 
that it can know to try to resolve the later versions of the rpc message (?) 
classes?
* --or-- Is this a bug in 2.20.0-SNAPSHOT?


===
[1] Complete list of 2.20.0-SNAPSHOT dependencies resolved by my build:


{noformat}
.../beam-sdks-java-io-kafka/2.20.0-SNAPSHOT/beam-sdks-java-io-kafka-2.20.0-20200120.072626-5.pom
.../beam-sdks-java-core/2.20.0-SNAPSHOT/beam-sdks-java-core-2.20.0-20200120.071349-5.pom
.../beam-model-pipeline/2.20.0-SNAPSHOT/beam-model-pipeline-2.20.0-20200120.070348-5.pom
.../beam-model-job-management/2.20.0-SNAPSHOT/beam-model-job-management-2.20.0-20200120.070335-5.pom
.../beam-runners-google-cloud-dataflow-java/2.20.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.20.0-20200120.070606-5.pom
.../beam-sdks-java-extensions-google-cloud-platform-core/2.20.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.20.0-20200120.072059-5.pom
.../beam-sdks-java-io-google-cloud-platform/2.20.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.20.0-20200120.072441-5.pom
.../beam-sdks-java-extensions-protobuf/2.20.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.20.0-20200120.072149-5.pom
.../beam-runners-core-construction-java/2.20.0-SNAPSHOT/beam-runners-core-construction-java-2.20.0-20200120.070440-5.pom
.../beam-runners-direct-java/2.20.0-SNAPSHOT/beam-runners-direct-java-2.20.0-20200120.070532-5.pom
.../beam-sdks-java-core/2.20.0-SNAPSHOT/beam-sdks-java-core-2.20.0-20200120.071349-5.jar
.../beam-model-job-management/2.20.0-SNAPSHOT/beam-model-job-management-2.20.0-20200120.070335-5.jar
.../beam-model-pipeline/2.20.0-SNAPSHOT/beam-model-pipeline-2.20.0-20200120.070348-5.jar
.../beam-sdks-java-io-kafka/2.20.0-SNAPSHOT/beam-sdks-java-io-kafka-2.20.0-20200120.072626-5.jar
.../beam-runners-google-cloud-dataflow-java/2.20.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.20.0-20200120.070606-5.jar
.../beam-sdks-java-extensions-google-cloud-platform-core/2.20.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.20.0-20200120.072059-5.jar
.../beam-sdks-java-io-google-cloud-platform/2.20.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.20.0-20200120.072441-5.jar
.../beam-sdks-java-extensions-protobuf/2.20.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.20.0-20200120.072149-5.jar
.../beam-runners-core-construction-java/2.20.0-SNAPSHOT/beam-runners-core-construction-java-2.20.0-20200120.070440-5.jar
.../beam-runners-direct-java/2.20.0-SNAPSHOT/beam-runners-direct-java-2.20.0-20200120.070532-5.jar
{noformat}


was (Author: atdixon):
Attempting to validate with 2.20.0-SNAPSHOT [1]

My pipeline however is unable to start in Dataflow, I get:
`java.lang.ClassNotFoundException: 
org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message$Builder`

When I inspect my build artifacts I find that when I depend on Beam 2.16.0 
(last Beam build that works for me), I find this class present (v1p21p0). When 
I build against Beam 2.20.0-SNAPSHOT, only v1p26p0 is resolved and packaged.

* I assume that the Dataflow runner needs to somehow know to look for v1p26p0 
somehow?  I.e., do I need to send it a flag that I'm suing 2.20.0-SNAPSHOT so 
that it can know to try to resolve the later versions of the rpc message (?) 
classes?
* --or-- Is this a bug in 2.20.0-SNAPSHOT?


===
[1] Complete list of 2.20.0-SNAPSHOT dependencies resolved by my build:
```
.../beam-sdks-java-io-kafka/2.20.0-SNAPSHOT/beam-sdks-java-io-kafka-2.20.0-20200120.072626-5.pom
.../beam-sdks-java-core/2.20.0-SNAPSHOT/beam-sdks-java-core-2.20.0-20200120.071349-5.pom
.../beam-model-pipeline/2.20.0-SNAPSHOT/beam-model-pipeline-2.20.0-20200120.070348-5.pom
.../beam-model-job-management/2.20.0-SNAPSHOT/beam-model-job-management-2.20.0-20200120.070335-5.pom
.../beam-runners-google-cloud-dataflow-java/2.20.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.20.0-20200120.070606-5.pom
.../beam-sdks-java-extensions-google-cloud-platform-core/2.20.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.20.0-20200120.072059-5.pom

[jira] [Commented] (BEAM-9144) Beam's own Avro TimeConversion class in beam-sdk-java-core

2020-01-20 Thread Aaron Dixon (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019629#comment-17019629
 ] 

Aaron Dixon commented on BEAM-9144:
---

Attempting to validate with 2.20.0-SNAPSHOT [1]

My pipeline however is unable to start in Dataflow, I get:
`java.lang.ClassNotFoundException: 
org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message$Builder`

When I inspect my build artifacts I find that when I depend on Beam 2.16.0 
(last Beam build that works for me), I find this class present (v1p21p0). When 
I build against Beam 2.20.0-SNAPSHOT, only v1p26p0 is resolved and packaged.

* I assume that the Dataflow runner needs to somehow know to look for v1p26p0 
somehow?  I.e., do I need to send it a flag that I'm suing 2.20.0-SNAPSHOT so 
that it can know to try to resolve the later versions of the rpc message (?) 
classes?
* --or-- Is this a bug in 2.20.0-SNAPSHOT?


===
[1] Complete list of 2.20.0-SNAPSHOT dependencies resolved by my build:
```
.../beam-sdks-java-io-kafka/2.20.0-SNAPSHOT/beam-sdks-java-io-kafka-2.20.0-20200120.072626-5.pom
.../beam-sdks-java-core/2.20.0-SNAPSHOT/beam-sdks-java-core-2.20.0-20200120.071349-5.pom
.../beam-model-pipeline/2.20.0-SNAPSHOT/beam-model-pipeline-2.20.0-20200120.070348-5.pom
.../beam-model-job-management/2.20.0-SNAPSHOT/beam-model-job-management-2.20.0-20200120.070335-5.pom
.../beam-runners-google-cloud-dataflow-java/2.20.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.20.0-20200120.070606-5.pom
.../beam-sdks-java-extensions-google-cloud-platform-core/2.20.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.20.0-20200120.072059-5.pom
.../beam-sdks-java-io-google-cloud-platform/2.20.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.20.0-20200120.072441-5.pom
.../beam-sdks-java-extensions-protobuf/2.20.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.20.0-20200120.072149-5.pom
.../beam-runners-core-construction-java/2.20.0-SNAPSHOT/beam-runners-core-construction-java-2.20.0-20200120.070440-5.pom
.../beam-runners-direct-java/2.20.0-SNAPSHOT/beam-runners-direct-java-2.20.0-20200120.070532-5.pom
.../beam-sdks-java-core/2.20.0-SNAPSHOT/beam-sdks-java-core-2.20.0-20200120.071349-5.jar
.../beam-model-job-management/2.20.0-SNAPSHOT/beam-model-job-management-2.20.0-20200120.070335-5.jar
.../beam-model-pipeline/2.20.0-SNAPSHOT/beam-model-pipeline-2.20.0-20200120.070348-5.jar
.../beam-sdks-java-io-kafka/2.20.0-SNAPSHOT/beam-sdks-java-io-kafka-2.20.0-20200120.072626-5.jar
.../beam-runners-google-cloud-dataflow-java/2.20.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.20.0-20200120.070606-5.jar
.../beam-sdks-java-extensions-google-cloud-platform-core/2.20.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.20.0-20200120.072059-5.jar
.../beam-sdks-java-io-google-cloud-platform/2.20.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.20.0-20200120.072441-5.jar
.../beam-sdks-java-extensions-protobuf/2.20.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.20.0-20200120.072149-5.jar
.../beam-runners-core-construction-java/2.20.0-SNAPSHOT/beam-runners-core-construction-java-2.20.0-20200120.070440-5.jar
.../beam-runners-direct-java/2.20.0-SNAPSHOT/beam-runners-direct-java-2.20.0-20200120.070532-5.jar
```

> Beam's own Avro TimeConversion class in beam-sdk-java-core 
> ---
>
> Key: BEAM-9144
> URL: https://issues.apache.org/jira/browse/BEAM-9144
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: avro-beam-dependency-graph.png
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> From Aaron's comment in 
> https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476
>  .
> {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues 
> resolved in 1.9.x) so downgrading Avro is not possible for us.
>  Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to 
> 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are 
> not available in 1.9.x.
> {quote}
> The Java class is 
> {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8.
>  It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9.
> h1. Beam Java SDK cannot upgrade Avro to 1.9
> Beam has Spark runners and Spark has not yet upgraded to Avro 1.9.
> Illustration of the dependency
> !avro-beam-dependency-graph.png|width=799,height=385!
> h1. Short-term Solution
> As illustrated above, as long as Beam Java SDK uses only the intersection of 
> Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide 
> flexibility in runtime

[jira] [Assigned] (BEAM-9125) Update BigQuery Storage API documentation



 [ 
https://issues.apache.org/jira/browse/BEAM-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-9125:
--

Assignee: Kenneth Jung

> Update BigQuery Storage API documentation
> -
>
> Key: BEAM-9125
> URL: https://issues.apache.org/jira/browse/BEAM-9125
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Jung
>Assignee: Kenneth Jung
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Some of the [documented limitations|#storage-api]] for using the BigQuery 
> Storage API in Beam are no longer accurate. In particular:
>  * The BigQuery Storage API no longer needs to be enabled manually
>  * Dynamic work rebalancing has been supported in Beam since 2.15.0
>  * The sample code should use .withSelectedFields(...) rather than 
> .withReadOptions(...).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9125) Update BigQuery Storage API documentation



 [ 
https://issues.apache.org/jira/browse/BEAM-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9125:
---
Status: Open  (was: Triage Needed)

> Update BigQuery Storage API documentation
> -
>
> Key: BEAM-9125
> URL: https://issues.apache.org/jira/browse/BEAM-9125
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Jung
>Assignee: Kenneth Jung
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Some of the [documented limitations|#storage-api]] for using the BigQuery 
> Storage API in Beam are no longer accurate. In particular:
>  * The BigQuery Storage API no longer needs to be enabled manually
>  * Dynamic work rebalancing has been supported in Beam since 2.15.0
>  * The sample code should use .withSelectedFields(...) rather than 
> .withReadOptions(...).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9145) Integrate Google Cloud AI functionalities



 [ 
https://issues.apache.org/jira/browse/BEAM-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9145:
---
Status: Open  (was: Triage Needed)

> Integrate Google Cloud AI functionalities
> -
>
> Key: BEAM-9145
> URL: https://issues.apache.org/jira/browse/BEAM-9145
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp, io-py-gcp
>Reporter: Kamil Wasilewski
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9147) [Java] PTransform that integrates Video Intelligence functionality



 [ 
https://issues.apache.org/jira/browse/BEAM-9147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9147:
---
Status: Open  (was: Triage Needed)

> [Java] PTransform that integrates Video Intelligence functionality
> --
>
> Key: BEAM-9147
> URL: https://issues.apache.org/jira/browse/BEAM-9147
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Kamil Wasilewski
>Assignee: Michal Walenia
>Priority: Major
>
> The goal is to create a PTransform that integrates Google Cloud Video 
> Intelligence functionality [1].
> The transform should be able to take both video GCS location or video data 
> bytes as an input.
> A module with the transform should be put into _`sdks/java/extensions`_ 
> folder.
> [1] [https://cloud.google.com/video-intelligence/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9133) CassandraIOTest.classMethod test is still flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9133:
---
Status: Open  (was: Triage Needed)

> CassandraIOTest.classMethod test is still flaky
> ---
>
> Key: BEAM-9133
> URL: https://issues.apache.org/jira/browse/BEAM-9133
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.17.0
>Reporter: Alexey Romanenko
>Assignee: Etienne Chauchot
>Priority: Critical
>
> CassandraIOTest is still flaky. For example:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
> {code}
> Error Message
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> Stacktrace
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
>   at 
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
>   at 
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
>   at 
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
>   at 
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
>   at 
>

[jira] [Updated] (BEAM-9146) [Python] PTransform that integrates Video Intelligence functionality



 [ 
https://issues.apache.org/jira/browse/BEAM-9146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9146:
---
Status: Open  (was: Triage Needed)

> [Python] PTransform that integrates Video Intelligence functionality
> 
>
> Key: BEAM-9146
> URL: https://issues.apache.org/jira/browse/BEAM-9146
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-py-gcp
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>
> The goal is to create a PTransform that integrates Google Cloud Video 
> Intelligence functionality [1].
> The transform should be able to take both video GCS location or video data 
> bytes as an input.
> The transform should be put into _sdks/python/apache_beam/io/gcp/ai_ folder.
> [1] https://cloud.google.com/video-intelligence/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9153) The heading level of "Deploy source release to dist.apache.org" in release guide is incorrect



 [ 
https://issues.apache.org/jira/browse/BEAM-9153?focusedWorklogId=374635=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374635
 ]

ASF GitHub Bot logged work on BEAM-9153:


Author: ASF GitHub Bot
Created on: 20/Jan/20 16:21
Start Date: 20/Jan/20 16:21
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10638: 
[website][BEAM-9153] Fix release guide heading level
URL: https://github.com/apache/beam/pull/10638
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374635)
Time Spent: 20m  (was: 10m)

> The heading level of "Deploy source release to dist.apache.org" in release 
> guide is incorrect
> -
>
> Key: BEAM-9153
> URL: https://issues.apache.org/jira/browse/BEAM-9153
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Deploy source release  to dist.apache.org" in [1] should have the same 
> heading level as "Deploy Python artifacts to PyPI.
>  
>  
> [1][https://beam.apache.org/contribute/release-guide/#deploy-source-release-to-distapacheorg]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9150) beam_PostRelease_Python_Candidate fails



 [ 
https://issues.apache.org/jira/browse/BEAM-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9150:
---
Status: Open  (was: Triage Needed)

> beam_PostRelease_Python_Candidate fails
> ---
>
> Key: BEAM-9150
> URL: https://issues.apache.org/jira/browse/BEAM-9150
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Udi Meiri
>Assignee: Yifan Zou
>Priority: Major
>
> https://builds.apache.org/job/beam_PostRelease_Python_Candidate/171/timestamps/?time=HH:mm:ss=GMT-8=en_US
> Some interesting lines:
> {code}
> 17:48:36  
> release/src/main/python-release/run_release_candidate_python_quickstart.sh: 
> line 144: 20935 Killed  python -m 
> apache_beam.examples.streaming_wordcount --input_topic 
> projects/$PROJECT_ID/topics/$PUBSUB_TOPIC1 --output_topic 
> projects/$PROJECT_ID/topics/$PUBSUB_TOPIC2 --streaming
> {code}
> {code}
> 18:05:25  
> 
> 18:05:25  Running HourlyTeamScore example with DirectRunner
> 18:05:25  
> 
> 18:05:25  
> /tmp/tmp.gnpMNIFhcL/temp_virtualenv_python2.7/local/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>  UserWarning: You are using Apache Beam with Python 2. New releases of Apache 
> Beam will soon support Python 3 only.
> 18:05:25'You are using Apache Beam with Python 2. '
> 18:05:33  INFO:root:Missing pipeline option (runner). Executing pipeline 
> using the default runner: DirectRunner.
> 18:05:33  INFO:root:Setting socket default timeout to 60 seconds.
> 18:05:33  INFO:root:socket default timeout is 60.0 seconds.
> 18:05:33  INFO:oauth2client.transport:Attempting refresh to obtain initial 
> access_token
> 18:05:34  
> /tmp/tmp.gnpMNIFhcL/temp_virtualenv_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1220:
>  BeamDeprecationWarning: options is deprecated since First stable release. 
> References to .options will not be supported
> 18:05:34experiments = p.options.view_as(DebugOptions).experiments or []
> 18:05:37  INFO:root:  annotate_downstream_side_inputs at 0x7f131dc46f50> 
> 18:05:37  INFO:root:  fix_side_input_pcoll_coders at 0x7f131dc3e0c8> 
> 18:05:37  INFO:root:  0x7f131dc3e140> 
> 18:05:37  INFO:root:  0x7f131dc3e1b8> 
> 18:05:37  INFO:root:  0x7f131dc3e230> 
> 18:05:37  INFO:root:  0x7f131dc3e320> 
> 18:05:37  INFO:root:  0x7f131dc3e398> 
> 18:05:37  INFO:root:  0x7f131dc3e410> 
> 18:05:37  INFO:root:  0x7f131dc3e488> 
> 18:05:37  INFO:root:  at 0x7f131dc3e5f0> 
> 18:05:37  INFO:root:  0x7f131dc3e668> 
> 18:05:37  INFO:root:  at 0x7f131dc3e6e0> 
> 18:05:37  INFO:root:Creating state cache with size 100
> 18:05:37  INFO:root:Created Worker handler 
>  at 0x7f12fc89c290> for environment urn: "beam:env:embedded_python:v1"
> 18:05:37  
> 18:05:37  INFO:root:Running 
> ((ref_AppliedPTransform_ReadInputText/Read/_SDFBoundedSourceWrapper/Impulse_5)+(ReadInputText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+((ReadInputText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction)+(ref_PCollection_PCollection_1_split/Write))
> 18:05:38  INFO:root:Running 
> (ref_PCollection_PCollection_1_split/Read)+ReadInputText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)/Process)+((ref_AppliedPTransform_HourlyTeamScore/ParseGameEventFn_8)+((ref_AppliedPTransform_HourlyTeamScore/FilterStartTime_9)+((ref_AppliedPTransform_HourlyTeamScore/FilterEndTime_10)+((ref_AppliedPTransform_HourlyTeamScore/AddEventTimestamps_11)+((ref_AppliedPTransform_HourlyTeamScore/FixedWindowsTeam_12)+(ref_AppliedPTransform_HourlyTeamScore/ExtractAndSumScore/Map(  at 
> hourly_team_score.py:142>)_14)))+(HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/Precombine))+(HourlyTeamScore/ExtractAndSumScore/CombinePerKey(sum)/Group/Write))
> 18:05:39  INFO:root:Running 
>

[jira] [Updated] (BEAM-9137) beam_PostCommit_Py_ValCont should run with dataflow_worker_jar



 [ 
https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9137:
---
Status: Open  (was: Triage Needed)

> beam_PostCommit_Py_ValCont should run with dataflow_worker_jar
> --
>
> Key: BEAM-9137
> URL: https://issues.apache.org/jira/browse/BEAM-9137
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Boyuan Zhang
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> For the first failure, please refer to 
> https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9151) Dataflow legacy worker tests are mis-configured



 [ 
https://issues.apache.org/jira/browse/BEAM-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9151:
---
Status: Open  (was: Triage Needed)

> Dataflow legacy worker tests are mis-configured
> ---
>
> Key: BEAM-9151
> URL: https://issues.apache.org/jira/browse/BEAM-9151
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Please refer to the last comment of https://github.com/apache/beam/pull/8183



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-9153) The heading level of "Deploy source release to dist.apache.org" in release guide is incorrect



 [ 
https://issues.apache.org/jira/browse/BEAM-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-9153.

Fix Version/s: (was: 2.20.0)
   Not applicable
   Resolution: Fixed

> The heading level of "Deploy source release to dist.apache.org" in release 
> guide is incorrect
> -
>
> Key: BEAM-9153
> URL: https://issues.apache.org/jira/browse/BEAM-9153
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Deploy source release  to dist.apache.org" in [1] should have the same 
> heading level as "Deploy Python artifacts to PyPI.
>  
>  
> [1][https://beam.apache.org/contribute/release-guide/#deploy-source-release-to-distapacheorg]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9148) test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql



 [ 
https://issues.apache.org/jira/browse/BEAM-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9148:
---
Status: Open  (was: Triage Needed)

> test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql
> --
>
> Key: BEAM-9148
> URL: https://issues.apache.org/jira/browse/BEAM-9148
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp, sdk-py-core, test-failures
>Reporter: Udi Meiri
>Assignee: Boyuan Zhang
>Priority: Major
>
> There might be other flaky test cases from the same class, but I'm focusing 
> on test_big_query_standard_sql here.
> {code}
> 19:39:12  
> ==
> 19:39:12  FAIL: test_big_query_standard_sql 
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 19:39:12  
> --
> 19:39:12  Traceback (most recent call last):
> 19:39:12File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
>  line 172, in test_big_query_standard_sql
> 19:39:12  big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 19:39:12File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
>  line 84, in run_bq_pipeline
> 19:39:12  result = p.run()
> 19:39:12File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 112, in run
> 19:39:12  else test_runner_api))
> 19:39:12File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
>  line 461, in run
> 19:39:12  self._options).run(False)
> 19:39:12File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
>  line 474, in run
> 19:39:12  return self.runner.run_pipeline(self, self._options)
> 19:39:12File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
>  line 53, in run_pipeline
> 19:39:12  hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 19:39:12  AssertionError: 
> 19:39:12  Expected: (Test pipeline expected terminated in state: DONE and 
> Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
> 19:39:12   but: Expected checksum is 
> 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is 
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> 19:39:12  
> 19:39:12   >> begin captured logging << 
> 
> 19:39:12  root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12  root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12  apache_beam.runners.direct.direct_runner: INFO: Running pipeline 
> with DirectRunner.
> 19:39:12  apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM 
> (SELECT "apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not 
> reference any tables.
> 19:39:12  apache_beam.io.gcp.bigquery_tools: WARNING: Dataset 
> apache-beam-testing:temp_dataset_90f5797bdb5f4137af750399f91a8e66 does not 
> exist so we will create it as temporary with location=None
> 19:39:12  apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table 
>  19:39:12   datasetId: 'python_query_to_table_15792323245106'
> 19:39:12   projectId: 'apache-beam-testing'
> 19:39:12   tableId: 'output_table'> with schema {'fields': [{'name': 'fruit', 
> 'type': 'STRING', 'mode': 'NULLABLE'}]}.
> 19:39:12  apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id 
> output_table
> 19:39:12  apache_beam.io.gcp.bigquery_tools: INFO: Created table 
> apache-beam-testing.python_query_to_table_15792323245106.output_table with 
> schema  19:39:12   fields: [ 19:39:12   fields: []
> 19:39:12   mode: 'NULLABLE'
> 19:39:12   name: 'fruit'
> 19:39:12   type: 'STRING'>]>. Result:  19:39:12   creationTime: 1579232328576
> 19:39:12   etag: 'WYysl6UIvc8IWMmTiiKhbg=='
> 19:39:12   id: 
> 'apache-beam-testing:python_query_to_table_15792323245106.output_table'
> 19:39:12   kind: 'bigquery#table'
> 19:39:12   lastModifiedTime: 1579232328629
> 19:39:12   location: 'US'
> 19:39:12   numBytes: 0
> 19:39:12   numLongTermBytes: 0
> 19:39:12   numRows: 0
> 19:39:12   schema:  19:39:12   fields: [ 19:39:12   fields: []
> 19:39:12   mode: 'NULLABLE'
> 19:39:12   name: 'fruit'
> 19:39:12   type: 'STRING'>]>
> 19:39:12   selfLink: 
> 'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15792323245106/tables/output_table'
> 19:39:12   tableReference:  19:39:12   datasetId:

[jira] [Updated] (BEAM-9154) Move Chicago Taxi Example to Python 3



 [ 
https://issues.apache.org/jira/browse/BEAM-9154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9154:
---
Status: Open  (was: Triage Needed)

> Move Chicago Taxi Example to Python 3
> -
>
> Key: BEAM-9154
> URL: https://issues.apache.org/jira/browse/BEAM-9154
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>
> The Chicago Taxi Example[1] should be moved to the latest version of Python 
> supported by Beam (currently it's Python 3.7).
> At the moment, the following error occurs when running the benchmark on 
> Python 3.7 (requires futher investigation):
> {code:java}
> Traceback (most recent call last):
>   File "preprocess.py", line 259, in 
> main()
>   File "preprocess.py", line 254, in main
> project=known_args.metric_reporting_project
>   File "preprocess.py", line 155, in transform_data
> ('Analyze' >> tft_beam.AnalyzeDataset(preprocessing_fn)))
>   File 
> "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 987, in __ror__
> return self.transform.__ror__(pvalueish, self.label)
>   File 
> "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 547, in __ror__
> result = p.apply(self, pvalueish, label)
>   File 
> "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/pipeline.py", line 
> 532, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/pipeline.py", line 
> 573, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
>   File 
> "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/runners/runner.py", 
> line 193, in apply
> return m(transform, input, options)
>   File 
> "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/runners/runner.py", 
> line 223, in apply_PTransform
> return transform.expand(input)
>   File 
> "/Users/kamilwasilewski/proj/beam/build/gradleenv/2022703441/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py",
>  line 825, in expand
> input_metadata))
>   File 
> "/Users/kamilwasilewski/proj/beam/build/gradleenv/2022703441/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py",
>  line 716, in expand
> output_signature = self._preprocessing_fn(copied_inputs)
>   File "preprocess.py", line 102, in preprocessing_fn
> _fill_in_missing(inputs[key]),
> KeyError: 'company'
> {code}
> [1] sdks/python/apache_beam/testing/benchmarks/chicago_taxi



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9153) The heading level of "Deploy source release to dist.apache.org" in release guide is incorrect



 [ 
https://issues.apache.org/jira/browse/BEAM-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9153:
---
Status: Open  (was: Triage Needed)

> The heading level of "Deploy source release to dist.apache.org" in release 
> guide is incorrect
> -
>
> Key: BEAM-9153
> URL: https://issues.apache.org/jira/browse/BEAM-9153
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Deploy source release  to dist.apache.org" in [1] should have the same 
> heading level as "Deploy Python artifacts to PyPI.
>  
>  
> [1][https://beam.apache.org/contribute/release-guide/#deploy-source-release-to-distapacheorg]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9046) Kafka connector for Python throws ClassCastException when reading KafkaRecord

2020-01-20 Thread Maximilian Michels (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019581#comment-17019581
 ] 

Maximilian Michels commented on BEAM-9046:
--

Are you also using the 2.13.0 version of the Python SDK? If not, you'll likely 
run into issues with non-matching versions. Please try with a virtual 
environment and something like {{pip install apache_beam==2.13.0}}.

That said, the patch should also work with the newest version, although I 
haven't verified that, so better to test this with the 2.13.0 branch.

> Kafka connector for Python throws ClassCastException when reading KafkaRecord
> -
>
> Key: BEAM-9046
> URL: https://issues.apache.org/jira/browse/BEAM-9046
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-kafka
>Affects Versions: 2.16.0
>Reporter: Berkay Öztürk
>Priority: Major
>  Labels: KafkaIO, Python
> Fix For: Not applicable
>
>
>  I'm trying to read the data streaming from Apache Kafka using the Python SDK 
> for Apache Beam with the Flink runner. After running Kafka 2.4.0 and Flink 
> 1.8.3, I follow these steps:
>  * Compile and run Beam 2.16 with Flink 1.8 runner.
> {code:java}
> git clone --single-branch --branch release-2.16.0 
> https://github.com/apache/beam.git beam-2.16.0
> cd beam-2.16.0
> nohup ./gradlew :runners:flink:1.8:job-server:runShadow 
> -PflinkMasterUrl=localhost:8081 &
> {code}
>  * Run the Python pipeline.
> {code:python}
> from apache_beam import Pipeline
> from apache_beam.io.external.kafka import ReadFromKafka
> from apache_beam.options.pipeline_options import PipelineOptions
> if __name__ == '__main__':
> with Pipeline(options=PipelineOptions([
> '--runner=FlinkRunner',
> '--flink_version=1.8',
> '--flink_master_url=localhost:8081',
> '--environment_type=LOOPBACK',
> '--streaming'
> ])) as pipeline:
> (
> pipeline
> | 'read' >> ReadFromKafka({'bootstrap.servers': 
> 'localhost:9092'}, ['test'])  # [BEAM-3788] ???
> )
> result = pipeline.run()
> result.wait_until_finish()
> {code}
>  * Publish some data to Kafka.
> {code:java}
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> >{"hello":"world!"}
> {code}
> The Python script throws this error:
> {code:java}
> [flink-runner-job-invoker] ERROR 
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Error 
> during job invocation BeamApp-USER-somejob. 
> org.apache.flink.client.program.ProgramInvocationException: Job failed. 
> (JobID: xxx)
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:483)
> at 
> org.apache.beam.runners.flink.FlinkExecutionEnvironments$BeamFlinkRemoteStreamEnvironment.executeRemotely(FlinkExecutionEnvironments.java:360)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:310)
> at 
> org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator$StreamingTranslationContext.execute(FlinkStreamingPortablePipelineTranslator.java:173)
> at 
> org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:104)
> at 
> org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:80)
> at 
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:78)
> at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
> at 
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
> ... 13 more
> Caused by: java.lang.ClassCastException: 
> org.apache.beam.sdk.io.kafka.KafkaRecord cannot be cast to [B
> at 
>

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374622
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:33
Start Date: 20/Jan/20 15:33
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368608083
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -356,6 +366,15 @@
 @Nullable
 abstract Map getOffsetConsumerConfig();
 
+@Nullable
 
 Review comment:
   On the class header (don't know why I pointed it here)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374622)
Time Spent: 3.5h  (was: 3h 20m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374620
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368600159
 
 

 ##
 File path: 
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
 ##
 @@ -391,6 +451,84 @@ public static void addCountingAsserts(
 PAssert.thatSingleton(input.apply("Max", Max.globally())).isEqualTo(max);
   }
 
+  private static class MockSchemaRegistryClientFactoryFn
+  implements SerializableFunction {
+
+private static final String SCHEMA_STRING =
+"{\"namespace\": \"example.avro\",\n"
++ " \"type\": \"record\",\n"
++ " \"name\": \"AvroGeneratedUser\",\n"
++ " \"fields\": [\n"
++ " {\"name\": \"name\", \"type\": \"string\"},\n"
++ " {\"name\": \"favorite_number\", \"type\": [\"int\", 
\"null\"]},\n"
++ " {\"name\": \"favorite_color\", \"type\": [\"string\", 
\"null\"]}\n"
++ " ]\n"
++ "}";
+
+private static final Schema SCHEMA = new 
Schema.Parser().parse(SCHEMA_STRING);
+
+private final String keySchemaSubject;
+private final String valueSchemaSubject;
+
+private MockSchemaRegistryClientFactoryFn(String keySubject, String 
valueSubject) {
+  this.keySchemaSubject = keySubject;
+  this.valueSchemaSubject = valueSubject;
+}
+
+@Override
+public SchemaRegistryClient apply(String schemaRegistryURL) {
+  SchemaRegistryClient registryClient = new MockSchemaRegistryClient();
+  registerSchemaBySubject(registryClient, keySchemaSubject);
+  registerSchemaBySubject(registryClient, valueSchemaSubject);
+  return registryClient;
+}
+
+private void registerSchemaBySubject(
+SchemaRegistryClient registryClient, String schemaSubject) {
+  try {
+registryClient.register(schemaSubject, SCHEMA);
+  } catch (IOException | RestClientException e) {
+throw new IllegalArgumentException(
+"Unable to register schema for subject: " + schemaSubject, e);
+  }
+}
+  }
+
+  @Test
+  public void testReadAvroKeyValuesWithSchemaRegistry() {
+int numElements = 100;
+String topic = "my_topic";
+String keySubject = topic + "-key";
 
 Review comment:
   Rename to `keySchemaSubject` (same for `valueSchemaSubject`)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374620)
Time Spent: 3h 10m  (was: 3h)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374610
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r367987462
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -356,6 +366,15 @@
 @Nullable
 abstract Map getOffsetConsumerConfig();
 
+@Nullable
 
 Review comment:
   Can you please add a usage example as part of the Javadoc of the class.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374610)
Time Spent: 2h 10m  (was: 2h)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374621=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374621
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368604901
 
 

 ##
 File path: 
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
 ##
 @@ -160,14 +169,36 @@
   private static final String TIMESTAMP_START_MILLIS_CONFIG = 
"test.timestamp.start.millis";
   private static final String TIMESTAMP_TYPE_CONFIG = "test.timestamp.type";
 
+  private static Serializer getSerializer(boolean isKey) {
+SchemaRegistryClient registryClient = new MockSchemaRegistryClient();
+Map map = new HashMap<>();
+map.put(KafkaAvroDeserializerConfig.AUTO_REGISTER_SCHEMAS, true);
+map.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, 
"mock://my-scope-name");
+Serializer serializer = (Serializer) new 
KafkaAvroSerializer(registryClient);
+serializer.configure(map, isKey);
+return serializer;
+  }
+
+  private static Deserializer getDeserializer(
+  boolean key, SchemaRegistryClient registryClient) {
+Map map = new HashMap<>();
+map.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true");
+map.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, 
"mock://my-scope-name");
+Deserializer deserializer =
+(Deserializer) new KafkaAvroDeserializer(registryClient);
+deserializer.configure(map, key);
+return deserializer;
+  }
+
   // Update mock consumer with records distributed among the given topics, 
each with given number
   // of partitions. Records are assigned in round-robin order among the 
partitions.
   private static MockConsumer mkMockConsumer(
   List topics,
   int partitionsPerTopic,
   int numElements,
   OffsetResetStrategy offsetResetStrategy,
-  Map config) {
+  Map config,
+  boolean isAvro) {
 
 Review comment:
   maybe rename `useAvro` is a more clear name.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374621)
Time Spent: 3h 20m  (was: 3h 10m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374618
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368600655
 
 

 ##
 File path: 
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
 ##
 @@ -391,6 +451,84 @@ public static void addCountingAsserts(
 PAssert.thatSingleton(input.apply("Max", Max.globally())).isEqualTo(max);
   }
 
+  private static class MockSchemaRegistryClientFactoryFn
+  implements SerializableFunction {
+
+private static final String SCHEMA_STRING =
+"{\"namespace\": \"example.avro\",\n"
++ " \"type\": \"record\",\n"
++ " \"name\": \"AvroGeneratedUser\",\n"
++ " \"fields\": [\n"
++ " {\"name\": \"name\", \"type\": \"string\"},\n"
++ " {\"name\": \"favorite_number\", \"type\": [\"int\", 
\"null\"]},\n"
++ " {\"name\": \"favorite_color\", \"type\": [\"string\", 
\"null\"]}\n"
++ " ]\n"
++ "}";
+
+private static final Schema SCHEMA = new 
Schema.Parser().parse(SCHEMA_STRING);
+
+private final String keySchemaSubject;
+private final String valueSchemaSubject;
+
+private MockSchemaRegistryClientFactoryFn(String keySubject, String 
valueSubject) {
 
 Review comment:
   `keySchemaSubject` and `valueSchemaSubject`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374618)
Time Spent: 2h 50m  (was: 2h 40m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374616=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374616
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368593945
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
 ##
 @@ -155,6 +172,25 @@ public boolean start() throws IOException {
 return advance();
   }
 
+  private SchemaRegistryClient getSchemaRegistryClient(Read spec) {
+SchemaRegistryClient registryClient;
+String schemaRegistryURL =
+(String)
+
spec.getConsumerConfig().get(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG);
 
 Review comment:
   Can you make `AbstractKafkaAvroSerDeConfig` into 
`KafkaAvroDeserializerConfig` as below.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374616)
Time Spent: 2h 40m  (was: 2.5h)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374613
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r367990288
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -586,6 +612,55 @@ public void setValueDeserializer(String 
valueDeserializer) {
   return 
toBuilder().setValueDeserializer(valueDeserializer).setValueCoder(valueCoder).build();
 }
 
+/**
+ * Sets a Confluent Schema Registry URL in case of using Avro schema 
serialised key/values. It
+ * should be used together with {@link #withKeySchemaSubject(String)} 
and/or {@link
+ * #withValueSchemaSubject(String)}.
+ */
+public Read withSchemaRegistry(String schemaRegistryUrl) {
 
 Review comment:
   Is this enough to support authentication? how do we pass a password or other 
credentials?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374613)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374614=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374614
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368586845
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -862,19 +958,56 @@ public void setValueDeserializer(String 
valueDeserializer) {
   // Infer key/value coders if not specified explicitly
   CoderRegistry registry = input.getPipeline().getCoderRegistry();
 
-  Coder keyCoder =
-  getKeyCoder() != null ? getKeyCoder() : inferCoder(registry, 
getKeyDeserializer());
-  checkArgument(
-  keyCoder != null,
-  "Key coder could not be inferred from key deserializer. Please 
provide"
-  + "key coder explicitly using withKeyDeserializerAndCoder()");
+  final String schemaRegistryURL =
+  (String) 
getConsumerConfig().get(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG);
 
-  Coder valueCoder =
-  getValueCoder() != null ? getValueCoder() : inferCoder(registry, 
getValueDeserializer());
-  checkArgument(
-  valueCoder != null,
-  "Value coder could not be inferred from value deserializer. Please 
provide"
-  + "value coder explicitly using 
withValueDeserializerAndCoder()");
+  // Set up key coder
+  Schema avroKeySchema = null;
+  if (schemaRegistryURL != null && getKeySchemaSubject() != null) {
+avroKeySchema = fetchAvroSchema(schemaRegistryURL, 
getKeySchemaSubject());
+checkArgument(avroKeySchema != null, "Avro key schema can't be null");
+  }
+
+  Coder keyCoder;
+  if (avroKeySchema != null) {
+keyCoder = (Coder) AvroCoder.of(avroKeySchema);
+checkArgument(
 
 Review comment:
   You can extract this check since is the same in both conditions, probably 
you can do it after the extracted method mentioned above.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374614)
Time Spent: 2.5h  (was: 2h 20m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374615
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368596391
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -817,6 +892,27 @@ public void setValueDeserializer(String 
valueDeserializer) {
   return new TypedWithoutMetadata<>(this);
 }
 
+private Schema fetchAvroSchema(String schemaRegistryUrl, String subject) {
+  SchemaRegistryClient registryClient;
 
 Review comment:
   Can you refactor the code of `KafkaUnboundedReader#getSchemaRegistryClient` 
into a method that you use here to avoid the repeated code to get a client.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374615)
Time Spent: 2h 40m  (was: 2.5h)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374617
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368597064
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -862,19 +958,56 @@ public void setValueDeserializer(String 
valueDeserializer) {
   // Infer key/value coders if not specified explicitly
   CoderRegistry registry = input.getPipeline().getCoderRegistry();
 
-  Coder keyCoder =
-  getKeyCoder() != null ? getKeyCoder() : inferCoder(registry, 
getKeyDeserializer());
-  checkArgument(
-  keyCoder != null,
-  "Key coder could not be inferred from key deserializer. Please 
provide"
-  + "key coder explicitly using withKeyDeserializerAndCoder()");
+  final String schemaRegistryURL =
+  (String) 
getConsumerConfig().get(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG);
 
-  Coder valueCoder =
-  getValueCoder() != null ? getValueCoder() : inferCoder(registry, 
getValueDeserializer());
-  checkArgument(
-  valueCoder != null,
-  "Value coder could not be inferred from value deserializer. Please 
provide"
-  + "value coder explicitly using 
withValueDeserializerAndCoder()");
+  // Set up key coder
+  Schema avroKeySchema = null;
+  if (schemaRegistryURL != null && getKeySchemaSubject() != null) {
+avroKeySchema = fetchAvroSchema(schemaRegistryURL, 
getKeySchemaSubject());
+checkArgument(avroKeySchema != null, "Avro key schema can't be null");
+  }
+
+  Coder keyCoder;
+  if (avroKeySchema != null) {
+keyCoder = (Coder) AvroCoder.of(avroKeySchema);
+checkArgument(
 
 Review comment:
   but probably that check should be `checkState` per 
https://github.com/google/guava/wiki/PreconditionsExplained
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374617)
Time Spent: 2h 50m  (was: 2h 40m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374619
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368594146
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
 ##
 @@ -155,6 +172,25 @@ public boolean start() throws IOException {
 return advance();
   }
 
+  private SchemaRegistryClient getSchemaRegistryClient(Read spec) {
+SchemaRegistryClient registryClient;
+String schemaRegistryURL =
+(String)
+
spec.getConsumerConfig().get(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG);
+
+if (spec.getSchemaRegistryClientFactoryFn() != null) {
+  registryClient = 
spec.getSchemaRegistryClientFactoryFn().apply(schemaRegistryURL);
+} else {
+  registryClient =
+  new CachedSchemaRegistryClient(
+  (String)
 
 Review comment:
   This is `schemaRegistryURL` no?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374619)
Time Spent: 3h  (was: 2h 50m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374612
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r368585962
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -862,19 +958,56 @@ public void setValueDeserializer(String 
valueDeserializer) {
   // Infer key/value coders if not specified explicitly
   CoderRegistry registry = input.getPipeline().getCoderRegistry();
 
-  Coder keyCoder =
-  getKeyCoder() != null ? getKeyCoder() : inferCoder(registry, 
getKeyDeserializer());
-  checkArgument(
-  keyCoder != null,
-  "Key coder could not be inferred from key deserializer. Please 
provide"
-  + "key coder explicitly using withKeyDeserializerAndCoder()");
+  final String schemaRegistryURL =
+  (String) 
getConsumerConfig().get(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG);
 
-  Coder valueCoder =
-  getValueCoder() != null ? getValueCoder() : inferCoder(registry, 
getValueDeserializer());
-  checkArgument(
-  valueCoder != null,
-  "Value coder could not be inferred from value deserializer. Please 
provide"
-  + "value coder explicitly using 
withValueDeserializerAndCoder()");
+  // Set up key coder
 
 Review comment:
   Can you extract this logic to obtain the `KeyCoder` into an internal method 
that returns the keyCoder and assigns it here (same for ValueCoder) just to 
make the expand code more readable.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374612)
Time Spent: 2h 20m  (was: 2h 10m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO



 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=374611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374611
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 20/Jan/20 15:32
Start Date: 20/Jan/20 15:32
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10563: [BEAM-7310] 
Add support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#discussion_r367991473
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -817,6 +892,27 @@ public void setValueDeserializer(String 
valueDeserializer) {
   return new TypedWithoutMetadata<>(this);
 }
 
+private Schema fetchAvroSchema(String schemaRegistryUrl, String subject) {
+  SchemaRegistryClient registryClient;
+  if (getSchemaRegistryClientFactoryFn() != null) {
+registryClient = 
getSchemaRegistryClientFactoryFn().apply(schemaRegistryUrl);
+  } else {
+registryClient = new CachedSchemaRegistryClient(schemaRegistryUrl, 
Integer.MAX_VALUE);
+  }
+
+  SchemaMetadata latestSchemaMetadata;
+  try {
+latestSchemaMetadata = registryClient.getLatestSchemaMetadata(subject);
+  } catch (IOException | RestClientException e) {
+throw new IllegalArgumentException(
+"Unable to get latest schema metadata for subject: " + subject, e);
+  }
+
+  final Schema avroSchema = new 
Schema.Parser().parse(latestSchemaMetadata.getSchema());
+  checkArgument(avroSchema != null, "Avro schema can't be null");
 
 Review comment:
   can this happen? I mean if the schema is not parseable it will throw an 
exception so probably this is not needed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374611)
Time Spent: 2h 20m  (was: 2h 10m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5086) Beam Dependency Update Request: org.apache.kudu



 [ 
https://issues.apache.org/jira/browse/BEAM-5086?focusedWorklogId=374570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374570
 ]

ASF GitHub Bot logged work on BEAM-5086:


Author: ASF GitHub Bot
Created on: 20/Jan/20 14:24
Start Date: 20/Jan/20 14:24
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10561: [BEAM-5086] Update 
kudu-client to version 1.11.1
URL: https://github.com/apache/beam/pull/10561#issuecomment-576295502
 
 
   I agree with you in the sense that this is a question of judgment, but I 
don’t think we should assume any extreme position in particular for 
dependencies that ‘are working’. Of course it is clear that the goal is that 
Beam users have a experience with no issues and we should fix linkage issues 
that impede the use of such libraries on Beam for example if we cannot read 
Data from Kudu because some of those linkage errors.
   
   Taking the responsibility of fixing every linkage issue that is not related 
to Beam is a considerable amount of work for in many cases not much in return. 
Linkage errors on bad shaded libraries provided by other projects happen 
commonly on unused execution paths, otherwise they would be  detected as errors 
in the dependency and fixed by the provider project.
   
   The important deal is to have a clear frontier of which are client provider 
problems that we may be inclined to fix so they don’t break the usage of the 
library for our users, but even in that case it makes more sense to provide the 
solution in the dependent project first. To save Beam of the extra maintenance.
   
   We should not be adding the extra work of dealing with this to Beam. I am 
afraid that this would be an endless task because I have seen the devil of 
doing this for Hadoop and friends or basically that nobody shades some deps 
well (yes I am looking at you netty) and we can be ‘fixing’ them for some IO 
just to discover that the new added (and unused path) dependencies conflict 
with the version that a runner use. Also we may end up with leaking 
dependencies that we pinned in our side to fix linkage errors, but that the 
upstream project may fix in the future and it will be also hard to discover / 
track the extra dependencies removal in our side.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374570)
Time Spent: 1h 40m  (was: 1.5h)

> Beam Dependency Update Request: org.apache.kudu
> ---
>
> Key: BEAM-5086
> URL: https://issues.apache.org/jira/browse/BEAM-5086
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-java-kudu
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> 2018-08-06 12:13:43.606551
> Please review and upgrade the org.apache.kudu to the latest version 
> None 
>  
> cc: 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5086) Beam Dependency Update Request: org.apache.kudu



 [ 
https://issues.apache.org/jira/browse/BEAM-5086?focusedWorklogId=374569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374569
 ]

ASF GitHub Bot logged work on BEAM-5086:


Author: ASF GitHub Bot
Created on: 20/Jan/20 14:21
Start Date: 20/Jan/20 14:21
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10561: [BEAM-5086] Update 
kudu-client to version 1.11.1
URL: https://github.com/apache/beam/pull/10561#issuecomment-576295502
 
 
   I agree with you in the sense that this is a question of judgment, but I 
don’t think we should assume any extreme position in particular for 
dependencies that ‘are working’. Of course it is clear that the goal is that 
Beam users have a experience with no issues and we should fix linkage issues 
that impede the use of such libraries on Beam for example if we cannot read 
Data from Kudu because some of those linkage errors.
   
   Taking the responsibility of fixing every linkage issue that is not related 
to Beam is a considerable amount of work for in many cases not much in return. 
Linkage errors on bad shaded libraries provided by other projects happen 
commonly on unused execution paths, otherwise they would be  detected as errors 
in the dependency and fixed by the provider project.
   
   The important deal is to have a clear frontier of which are client provider 
problems that we may be inclined to fix so they don’t break the usage of the 
library for our users, but even in that case it makes more sense to provide the 
solution in the dependent project first. To save Beam of the extra maintenance.
   
   We should not be adding the extra work of dealing with this to Beam. I am 
afraid that this would be an endless task because I have seen the devil of 
doing this for Hadoop and friends or basically that nobody shades some deps 
well (yes I am looking at you netty) and we can be ‘fixing’ them for some IO 
just to discover that the new added (and unused paths) conflict with the 
version that a runner use. Also we may end up with leaking dependencies that we 
pinned in our side to fix linkage errors, but that the upstream project may fix 
in the future and it will be also hard to discover / track the extra 
dependencies removal in our side.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374569)
Time Spent: 1.5h  (was: 1h 20m)

> Beam Dependency Update Request: org.apache.kudu
> ---
>
> Key: BEAM-5086
> URL: https://issues.apache.org/jira/browse/BEAM-5086
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-java-kudu
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> 2018-08-06 12:13:43.606551
> Please review and upgrade the org.apache.kudu to the latest version 
> None 
>  
> cc: 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-7663) Create SideInput Python Load Test Jenkins Job [Flink]

2020-01-20 Thread Kamil Wasilewski (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Wasilewski reassigned BEAM-7663:
--

Assignee: Kamil Wasilewski

> Create SideInput Python Load Test Jenkins Job [Flink]
> -
>
> Key: BEAM-7663
> URL: https://issues.apache.org/jira/browse/BEAM-7663
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8742) Add stateful processing to ParDo load test



 [ 
https://issues.apache.org/jira/browse/BEAM-8742?focusedWorklogId=374537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374537
 ]

ASF GitHub Bot logged work on BEAM-8742:


Author: ASF GitHub Bot
Created on: 20/Jan/20 12:34
Start Date: 20/Jan/20 12:34
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #10157: [BEAM-8742] Add 
stateful processing to ParDo load test
URL: https://github.com/apache/beam/pull/10157#issuecomment-576254787
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 374537)
Time Spent: 1h 40m  (was: 1.5h)

> Add stateful processing to ParDo load test
> --
>
> Key: BEAM-8742
> URL: https://issues.apache.org/jira/browse/BEAM-8742
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> So far, the ParDo load test is not stateful. We should add a basic counter to 
> test the stateful processing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8002) Beam Dependency Update Request: Gradle:

[
https://issues.apache.org/jira/browse/BEAM-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8002:

Description:

- 2019-08-19 12:04:19.192688 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:04:17.802062 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:04:53.057461 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-09 12:03:53.298241 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-16 12:03:47.037326 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-23 12:04:12.360299 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-30 12:05:19.680040 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-07 12:04:48.065365 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-14 12:04:37.533903 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-21 12:05:15.993810 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-28 12:07:10.976374 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-04 12:05:57.793513 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.4

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-11 12:05:24.076889 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 6.0

cc:
Please refer to [Beam

[jira] [Updated] (BEAM-6901) Beam Dependency Update Request: sqlline:sqlline



 [ 
https://issues.apache.org/jira/browse/BEAM-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-6901:

Description: 


 - 2019-03-25 04:23:12.026437 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-01 12:15:11.859956 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-08 12:15:43.829522 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-15 12:35:59.407782 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-22 12:13:31.922817 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-12 22:55:03.099214 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-12 23:32:43.232674 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-15 19:48:15.961831 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-18 12:13:32.145836 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-18 20:50:42.825702 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 20:08:43.049616 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 21:16:52.815933

[jira] [Updated] (BEAM-8857) Beam Dependency Update Request: software.amazon.awssdk:sqs

[
https://issues.apache.org/jira/browse/BEAM-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8857:

Description:

- 2019-12-02 12:24:16.109021 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:53.668958 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:40.910919 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:19:17.637354 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:47.875693 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:22:37.646038 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:20:07.374883 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-12-02 12:24:16.109021 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:53.668958 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:40.910919 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:19:17.637354 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:47.875693 -

Please consider upgrading the dependency software.amazon.awssdk:sqs.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

[jira] [Updated] (BEAM-8779) Beam Dependency Update Request: software.amazon.awssdk:sns

[
https://issues.apache.org/jira/browse/BEAM-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8779:

Description:

- 2019-11-19 21:16:49.149055 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:24:08.775701 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:46.101446 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:33.501651 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:19:09.662186 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:40.041350 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:22:30.059400 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:59.904063 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-19 21:16:49.149055 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:24:08.775701 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:46.101446 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:33.501651 -

Please consider upgrading the dependency software.amazon.awssdk:sns.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

[jira] [Updated] (BEAM-8777) Beam Dependency Update Request: software.amazon.awssdk:dynamodb

[
https://issues.apache.org/jira/browse/BEAM-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8777:

Description:

- 2019-11-19 21:16:41.659704 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:53.778671 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:30.990902 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:18.421491 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:53.835313 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:24.711735 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:22:14.406352 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:44.741846 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-19 21:16:41.659704 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:53.778671 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:30.990902 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:18.421491 -

Please consider upgrading the dependency
software.amazon.awssdk:dynamodb.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide

[jira] [Updated] (BEAM-8778) Beam Dependency Update Request: software.amazon.awssdk:sdk-core

[
https://issues.apache.org/jira/browse/BEAM-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8778:

Description:

- 2019-11-19 21:16:45.434855 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:24:01.285164 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:38.577293 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:26.029788 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:19:01.746810 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:32.397218 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:22:22.226230 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:52.287348 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-19 21:16:45.434855 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:24:01.285164 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:38.577293 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:26.029788 -

Please consider upgrading the dependency
software.amazon.awssdk:sdk-core.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide

[jira] [Updated] (BEAM-8726) Beam Dependency Update Request: org.xerial.snappy:snappy-java

[
https://issues.apache.org/jira/browse/BEAM-8726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8726:

Description:

- 2019-11-15 19:47:57.926453 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:20.157675 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:23.009498 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:22:55.693773 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:41.447596 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:14.805098 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:21:46.304893 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:21:36.274037 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:07.253081 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-15 19:47:57.926453 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:20.157675 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:23.009498 -

Please consider upgrading the dependency org.xerial.snappy:snappy-java.

The current version is 1.1.4. The latest version is 1.1.7.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify

[jira] [Updated] (BEAM-8776) Beam Dependency Update Request: software.amazon.awssdk:cloudwatch

[
https://issues.apache.org/jira/browse/BEAM-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8776:

Description:

- 2019-11-19 21:16:38.170041 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:46.235220 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:23.480913 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:10.854747 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:45.950047 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:16.965096 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:22:06.515615 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:37.222826 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-19 21:16:38.170041 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:46.235220 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:23.480913 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:10.854747 -

Please consider upgrading the dependency
software.amazon.awssdk:cloudwatch.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide

[jira] [Updated] (BEAM-8728) Beam Dependency Update Request: software.amazon.awssdk:apache-client

[
https://issues.apache.org/jira/browse/BEAM-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8728:

Description:

- 2019-11-15 19:48:03.214467 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.17

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:27.485013 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:30.895766 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:08.380893 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:56.045342 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:30.476238 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:01.507260 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:21:51.126902 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:21.921016 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-15 19:48:03.214467 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.17

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:27.485013 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:30.895766 -

Please consider upgrading the dependency
software.amazon.awssdk:apache-client.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to

[jira] [Updated] (BEAM-8775) Beam Dependency Update Request: software.amazon.awssdk:auth

[
https://issues.apache.org/jira/browse/BEAM-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8775:

Description:

- 2019-11-19 21:16:34.486345 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:38.773852 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:15.913920 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:03.492425 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:38.095217 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.41

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:22:09.217642 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.42

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:21:58.800101 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.47

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:29.631274 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.52

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-19 21:16:34.486345 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.19

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:38.773852 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.26

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:23:15.913920 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.30

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:24:03.492425 -

Please consider upgrading the dependency software.amazon.awssdk:auth.

The current version is 2.5.71. The latest version is 2.10.40

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The

[jira] [Updated] (BEAM-8928) Beam Dependency Update Request: ru.yandex.clickhouse:clickhouse-jdbc

[
https://issues.apache.org/jira/browse/BEAM-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8928:

Description:

- 2019-12-09 12:23:04.385538 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:48.655753 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:22.680137 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:21:53.947962 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:21:43.636323 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:19:14.444686 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-12-09 12:23:04.385538 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:48.655753 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:22.680137 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:21:53.947962 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:21:43.636323 -

Please consider upgrading the dependency
ru.yandex.clickhouse:clickhouse-jdbc.

The current version is 0.1.47. The latest version is 0.2.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

> Beam Dependency Update Request: ru.yandex.clickhouse:clickhouse-jdbc
>
>
> Key: BEAM-8928
> URL: https://issues.apache.org/jira/browse/BEAM-8928
> Project: Beam
> Issue

[jira] [Updated] (BEAM-5577) Beam Dependency Update Request: org.mongodb:mongo-java-driver

[
https://issues.apache.org/jira/browse/BEAM-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5577:

Description:

- 2018-10-01 19:31:53.116006 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.8.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-08 12:19:50.933043 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.8.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-15 12:13:43.378953 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.8.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:14:02.941839 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.8.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:18:53.503708 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.8.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:15:38.415073 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.8.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:15:38.514349 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.9.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:16:19.793075 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.9.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:15:20.150770 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.2.2. The latest version is 3.9.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:04.576425 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.9.1. The latest version is 3.12.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:17:39.058418 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.9.1. The latest version is 3.12.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:21:11.689590 -

Please consider upgrading the dependency org.mongodb:mongo-java-driver.

The current version is 3.9.1. The latest version is 3.12.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

[jira] [Updated] (BEAM-8724) Beam Dependency Update Request: org.slf4j:slf4j-simple

[
https://issues.apache.org/jira/browse/BEAM-8724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8724:

Description:

- 2019-11-15 19:47:47.257481 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:07.039581 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:10.542496 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:22:42.211988 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:29.735248 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:02.884139 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:21:34.655808 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:21:24.435761 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:18:55.428490 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-15 19:47:47.257481 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:07.039581 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:10.542496 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The

[jira] [Updated] (BEAM-9112) Beam Dependency Update Request: org.jboss.modules:jboss-modules



 [ 
https://issues.apache.org/jira/browse/BEAM-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-9112:

Description: 


 - 2020-01-13 12:20:44.226066 -

Please consider upgrading the dependency 
org.jboss.modules:jboss-modules. 

The current version is 1.1.0.Beta1. The latest version is 1.9.2.Final 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:18:22.442656 -

Please consider upgrading the dependency 
org.jboss.modules:jboss-modules. 

The current version is 1.1.0.Beta1. The latest version is 1.9.2.Final 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2020-01-13 12:20:44.226066 -

Please consider upgrading the dependency 
org.jboss.modules:jboss-modules. 

The current version is 1.1.0.Beta1. The latest version is 1.9.2.Final 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: org.jboss.modules:jboss-modules
> ---
>
> Key: BEAM-9112
> URL: https://issues.apache.org/jira/browse/BEAM-9112
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2020-01-13 12:20:44.226066 
> -
> Please consider upgrading the dependency 
> org.jboss.modules:jboss-modules. 
> The current version is 1.1.0.Beta1. The latest version is 1.9.2.Final 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:18:22.442656 
> -
> Please consider upgrading the dependency 
> org.jboss.modules:jboss-modules. 
> The current version is 1.1.0.Beta1. The latest version is 1.9.2.Final 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8644) Beam Dependency Update Request: org.freemarker:freemarker

[
https://issues.apache.org/jira/browse/BEAM-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8644:

Description:

- 2019-11-12 22:54:18.472454 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:15:39.185121 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:22:41.530326 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:22:15.167965 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:22:53.148923 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:17:29.309875 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:21:01.972648 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:20:30.044886 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:18:07.682746 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-12 22:54:18.472454 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:15:39.185121 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:22:41.530326 -

Please consider upgrading the dependency org.freemarker:freemarker.

The current version is 2.3.28. The latest version is 2.3.29

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

[jira] [Updated] (BEAM-8723) Beam Dependency Update Request: org.postgresql:postgresql



 [ 
https://issues.apache.org/jira/browse/BEAM-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8723:

Description: 


 - 2019-11-15 19:47:37.517620 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.8 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 21:15:53.175329 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.8 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:22:55.910553 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.8 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:22:27.862221 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.9 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:23:15.528808 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.9 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:17:48.399081 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.9 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-06 12:21:20.484055 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.9 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-13 12:21:09.820006 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.9 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:18:41.433568 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.9 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-11-15 19:47:37.517620 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.8 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 21:15:53.175329 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2. The latest version is 42.2.8 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:22:55.910553 -

Please consider upgrading the dependency org.postgresql:postgresql. 

The current version is 42.2.2.

[jira] [Updated] (BEAM-5595) Beam Dependency Update Request: org.elasticsearch.client:transport



 [ 
https://issues.apache.org/jira/browse/BEAM-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-5595:

Description: 


 - 2018-10-01 19:32:41.617633 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.1 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-08 12:21:34.320882 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-15 12:14:37.848340 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-22 12:15:03.921280 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-29 12:20:12.396930 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-05 12:17:32.861881 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-12 12:17:20.640439 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 6.4.3 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-19 12:18:08.155543 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-26 12:17:15.982748 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-12-03 12:17:28.265192 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-12-10 12:19:57.676671 -

Please consider upgrading the dependency 
org.elasticsearch.client:transport. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~echauchot], [~timrobertson100], 
 Please refer to [Beam Dependency Guide

[jira] [Updated] (BEAM-9110) Beam Dependency Update Request: org.jboss.marshalling:jboss-marshalling



 [ 
https://issues.apache.org/jira/browse/BEAM-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-9110:

Description: 


 - 2020-01-13 12:20:39.162035 -

Please consider upgrading the dependency 
org.jboss.marshalling:jboss-marshalling. 

The current version is 1.4.11.Final. The latest version is 2.0.9.Final 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:18:15.211191 -

Please consider upgrading the dependency 
org.jboss.marshalling:jboss-marshalling. 

The current version is 1.4.11.Final. The latest version is 2.0.9.Final 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2020-01-13 12:20:39.162035 -

Please consider upgrading the dependency 
org.jboss.marshalling:jboss-marshalling. 

The current version is 1.4.11.Final. The latest version is 2.0.9.Final 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: org.jboss.marshalling:jboss-marshalling
> ---
>
> Key: BEAM-9110
> URL: https://issues.apache.org/jira/browse/BEAM-9110
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2020-01-13 12:20:39.162035 
> -
> Please consider upgrading the dependency 
> org.jboss.marshalling:jboss-marshalling. 
> The current version is 1.4.11.Final. The latest version is 
> 2.0.9.Final 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:18:15.211191 
> -
> Please consider upgrading the dependency 
> org.jboss.marshalling:jboss-marshalling. 
> The current version is 1.4.11.Final. The latest version is 
> 2.0.9.Final 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9104) Beam Dependency Update Request: org.conscrypt:conscrypt-openjdk-uber



 [ 
https://issues.apache.org/jira/browse/BEAM-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-9104:

Description: 


 - 2020-01-13 12:19:34.357551 -

Please consider upgrading the dependency 
org.conscrypt:conscrypt-openjdk-uber. 

The current version is 1.3.0. The latest version is 2.2.1 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:17:04.413200 -

Please consider upgrading the dependency 
org.conscrypt:conscrypt-openjdk-uber. 

The current version is 1.3.0. The latest version is 2.2.1 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2020-01-13 12:19:34.357551 -

Please consider upgrading the dependency 
org.conscrypt:conscrypt-openjdk-uber. 

The current version is 1.3.0. The latest version is 2.2.1 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: org.conscrypt:conscrypt-openjdk-uber
> 
>
> Key: BEAM-9104
> URL: https://issues.apache.org/jira/browse/BEAM-9104
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2020-01-13 12:19:34.357551 
> -
> Please consider upgrading the dependency 
> org.conscrypt:conscrypt-openjdk-uber. 
> The current version is 1.3.0. The latest version is 2.2.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:17:04.413200 
> -
> Please consider upgrading the dependency 
> org.conscrypt:conscrypt-openjdk-uber. 
> The current version is 1.3.0. The latest version is 2.2.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-5752) Beam Dependency Update Request: org.eclipse.jetty:jetty-server

[
https://issues.apache.org/jira/browse/BEAM-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5752:

Description:

- 2018-10-15 12:13:17.575546 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:26.908573 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:10.104039 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:14:56.891103 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:14:57.086704 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:38.454968 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:38.329092 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:15:03.778648 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:33.026807 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:54.249911 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:13.071360 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:25.376755 -

Please consider upgrading the dependency

[jira] [Updated] (BEAM-5753) Beam Dependency Update Request: org.eclipse.jetty:jetty-servlet

[
https://issues.apache.org/jira/browse/BEAM-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5753:

Description:

- 2018-10-15 12:13:19.256622 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:32.240729 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:15.388249 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:15:02.810121 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:15:02.741564 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:44.462009 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:43.766710 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:15:09.151091 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:38.451204 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:59.925429 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:18.611955 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:30.751791 -

Please consider upgrading the dependency

[jira] [Updated] (BEAM-5551) Beam Dependency Update Request: org.elasticsearch:elasticsearch-hadoop



 [ 
https://issues.apache.org/jira/browse/BEAM-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-5551:

Description: 


 - 2018-10-01 19:30:14.876535 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.1 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-08 12:16:54.276690 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-15 12:11:33.317764 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-22 12:11:22.408698 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-10-29 12:13:36.588552 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-05 12:12:20.114978 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.2 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-12 12:12:11.983383 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 6.4.3 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-19 12:13:00.354059 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-11-26 12:12:07.620947 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-12-03 12:12:26.756619 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-12-10 12:14:47.418801 -

Please consider upgrading the dependency 
org.elasticsearch:elasticsearch-hadoop. 

The current version is 5.0.0. The latest version is 7.0.0-alpha1 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2018-12-17

[jira] [Updated] (BEAM-9108) Beam Dependency Update Request: org.eclipse.jetty.npn:npn-api



 [ 
https://issues.apache.org/jira/browse/BEAM-9108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-9108:

Description: 


 - 2020-01-13 12:20:01.165080 -

Please consider upgrading the dependency org.eclipse.jetty.npn:npn-api. 

The current version is 1.1.1.v20141010. The latest version is 
8.1.2.v20120308 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:17:35.116951 -

Please consider upgrading the dependency org.eclipse.jetty.npn:npn-api. 

The current version is 1.1.1.v20141010. The latest version is 
8.1.2.v20120308 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2020-01-13 12:20:01.165080 -

Please consider upgrading the dependency org.eclipse.jetty.npn:npn-api. 

The current version is 1.1.1.v20141010. The latest version is 
8.1.2.v20120308 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: org.eclipse.jetty.npn:npn-api
> -
>
> Key: BEAM-9108
> URL: https://issues.apache.org/jira/browse/BEAM-9108
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2020-01-13 12:20:01.165080 
> -
> Please consider upgrading the dependency 
> org.eclipse.jetty.npn:npn-api. 
> The current version is 1.1.1.v20141010. The latest version is 
> 8.1.2.v20120308 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:17:35.116951 
> -
> Please consider upgrading the dependency 
> org.eclipse.jetty.npn:npn-api. 
> The current version is 1.1.1.v20141010. The latest version is 
> 8.1.2.v20120308 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8643) Beam Dependency Update Request: org.elasticmq:elasticmq-rest-sqs_2.12

[
https://issues.apache.org/jira/browse/BEAM-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8643:

Description:

- 2019-11-12 22:53:27.839978 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:14:43.714596 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:21:44.584613 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:21:17.964835 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:21:55.839630 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:17:03.842366 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:20:36.785881 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:20:04.792866 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:17:42.187005 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-12 22:53:27.839978 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:14:43.714596 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:21:44.584613 -

Please consider upgrading the dependency
org.elasticmq:elasticmq-rest-sqs_2.12.

The current version is 0.14.1. The latest version is 0.15.3

cc:
Please refer to

[jira] [Updated] (BEAM-8640) Beam Dependency Update Request: org.apache.zookeeper:zookeeper



 [ 
https://issues.apache.org/jira/browse/BEAM-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8640:

Description: 


 - 2019-11-12 22:52:54.286486 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 21:13:55.530746 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:20:56.060900 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:20:29.190236 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:21:07.886362 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:16:12.752794 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-06 12:19:47.983448 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-13 12:18:53.610144 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:16:17.757264 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-11-12 22:52:54.286486 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 21:13:55.530746 -

Please consider upgrading the dependency 
org.apache.zookeeper:zookeeper. 

The current version is 3.4.11. The latest version is 3.5.6 

cc: [~timrobertson100], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:20:56.060900 -

Please consider upgrading the dependency

[jira] [Updated] (BEAM-8641) Beam Dependency Update Request: org.assertj:assertj-core

[
https://issues.apache.org/jira/browse/BEAM-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8641:

Description:

- 2019-11-12 22:52:57.856672 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:14:03.054594 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:21:03.797893 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:20:36.808168 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:21:15.594712 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:16:20.756947 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:19:55.600432 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:19:01.286632 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:16:25.344491 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-12 22:52:57.856672 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:14:03.054594 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:21:03.797893 -

Please consider upgrading the dependency org.assertj:assertj-core.

The current version is 3.11.1. The latest version is 3.14.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

[jira] [Updated] (BEAM-8925) Beam Dependency Update Request: org.apache.tika:tika-core



 [ 
https://issues.apache.org/jira/browse/BEAM-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8925:

Description: 


 - 2019-12-09 12:20:22.212496 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:20:53.356760 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:15:58.081400 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-06 12:19:33.456649 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-13 12:18:38.940974 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:16:03.428169 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-12-09 12:20:22.212496 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:20:53.356760 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:15:58.081400 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-06 12:19:33.456649 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-13 12:18:38.940974 -

Please consider upgrading the dependency org.apache.tika:tika-core. 

The current version is 1.20. The latest version is 1.23 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: org.apache.tika:tika-core
> -
>
> Key: BEAM-8925
> URL: https://issues.apache.org/jira/browse/BEAM-8925
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2019-12-09 12:20:22.212496 
>

[jira] [Updated] (BEAM-8721) Beam Dependency Update Request: org.codehaus.woodstox:stax2-api

[
https://issues.apache.org/jira/browse/BEAM-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8721:

Description:

- 2019-11-15 19:46:10.219873 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:14:12.192606 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:21:12.929815 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:20:45.950044 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:21:24.432934 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:16:30.174927 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-06 12:20:04.580632 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-13 12:19:18.815444 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2020-01-20 12:16:48.565620 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-15 19:46:10.219873 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:14:12.192606 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:21:12.929815 -

Please consider upgrading the dependency
org.codehaus.woodstox:stax2-api.

The current version is 3.1.4. The latest version is 4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The

[jira] [Updated] (BEAM-9103) Beam Dependency Update Request: org.bouncycastle:bcprov-jdk15on



 [ 
https://issues.apache.org/jira/browse/BEAM-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-9103:

Description: 


 - 2020-01-13 12:19:13.393574 -

Please consider upgrading the dependency 
org.bouncycastle:bcprov-jdk15on. 

The current version is 1.54. The latest version is 1.64 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2020-01-20 12:16:39.630067 -

Please consider upgrading the dependency 
org.bouncycastle:bcprov-jdk15on. 

The current version is 1.54. The latest version is 1.64 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2020-01-13 12:19:13.393574 -

Please consider upgrading the dependency 
org.bouncycastle:bcprov-jdk15on. 

The current version is 1.54. The latest version is 1.64 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: org.bouncycastle:bcprov-jdk15on
> ---
>
> Key: BEAM-9103
> URL: https://issues.apache.org/jira/browse/BEAM-9103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2020-01-13 12:19:13.393574 
> -
> Please consider upgrading the dependency 
> org.bouncycastle:bcprov-jdk15on. 
> The current version is 1.54. The latest version is 1.64 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:16:39.630067 
> -
> Please consider upgrading the dependency 
> org.bouncycastle:bcprov-jdk15on. 
> The current version is 1.54. The latest version is 1.64 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8926) Beam Dependency Update Request: org.apache.tika:tika-parsers