Re: Spring with Apache Beam

2019-10-09 Thread Luke Cwik
1. won't work since it is happening at pipeline construction time and not pipeline execution time. 2. only works if your application context is scoped to the DoFn instance and doesn't have things you want to possibly share across DoFn instances. You could also try and make it a PipelineOption

Re: Spring with Apache Beam

2019-10-09 Thread Luke Cwik
-d...@beam.apache.org, +user@beam.apache.org How are you trying to inject your application context? Have you looked at the JvmInitializer.beforeProcessing[1] to create your application context? 1:

Re: Feedback on how we use Apache Beam in my company

2019-10-09 Thread Etienne Chauchot
Very nice ! Thanks ccing dev list Etienne On 09/10/2019 16:55, Pierre Vanacker wrote: Hi Apache Beam community, We’ve been working with Apache Beam in production for a few years now in my company (Dailymotion). If you’re interested to know how we use Apache Beam in combination with

Feedback on how we use Apache Beam in my company

2019-10-09 Thread Pierre Vanacker
Hi Apache Beam community, We’ve been working with Apache Beam in production for a few years now in my company (Dailymotion). If you’re interested to know how we use Apache Beam in combination with Google Dataflow, we shared this experience in the following article :

Re: Beam discarding massive amount of events due to Window object or inner processing

2019-10-09 Thread Reza Rokni
Hi, When inserting into PubSub can you set message metadata with the timestamp from the event? If yes then you can make use of: https://beam.apache.org/releases/javadoc/2.16.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String- Cheers Reza On Wed, 9

Re: Beam discarding massive amount of events due to Window object or inner processing

2019-10-09 Thread Eddy G
Thanks a lot for the quick response! I can recall having already played with this when I first deployed this consumer and couldn't get around the following issue that I'm getting now again... java.lang.IllegalArgumentException: Cannot output with timestamp 2019-10-09T03:12:04.250Z. Output