Re: Infinite retry in streaming - is there a workaround?

​Kindly ping as I'm really curious about this. :p Derek​ On Thu, Oct 19, 2017 at 2:15 PM, Derek Hao Hu wrote: > Hi, > > ​We are trying to use Dataflow in Prod and right now one of our main > concerns is this "infinite retry" behavior which might stall the whole >

[Events] Call for speakers for a NY Meetup on 12/5 or 12/6, can be remote

Hi folks - I'm looking for 2 speakers for the Big Data Warehousing Meetup [1] in New York on December 5th or 6th, if you don't live in NY but would love to share your project you can give your talk via Hangout. Any topic related to Apache Beam or how

Re: How to window by quantity of data?

Final, working version is in the original gist: https://gist.github.com/jacobmarble/6ca40e0a14828e6a0dfe89b9cb2e4b4c The result is heavily inspired by GroupIntoBatches, and doesn't use windowing. Jacob On Wed, Oct 18, 2017 at 2:49 PM, Jacob Marble wrote: > Thomas, I

Re: Is anyone using Beam for geo use cases?

I wrote an OSM PBF parser for Beam. It took ~8h to parse the planet on one core, but that could be improved with a reshuffle after the actual read transform. I also wrote a simple key class. Useful for bucketing geo data into smaller chunks. - constructor takes desired grid size in kilometers -

Re: [VOTE] [DISCUSSION] Remove support for Java 7

On Wed, Oct 18, 2017 at 10:59 AM, Eugene Kirpichov wrote: > So far we seem to have unanimous consensus in favor of dropping Java7, and > we seem to also be converging on declaring that this doesn't require > increasing major version - but the discussion has been going on

Re: KafkaIO and Avro

This is due to Java doing type erasure in any expression that involves a raw type. This will compile if you extract the result of .apply(KafkaIO.read()...) into a local variable. On Fri, Oct 20, 2017, 1:51 AM Andrew Jones wrote: > Thanks Eugene. That does compile,

Re: Is anyone using Beam for geo use cases?

I am interested! I need to do this soon but havent started yet. Would love to see an example of how you do it. :) Br, Vilhelm von Ehrenheim On 20 Oct 2017 12:40, "Csaba Kassai" wrote: > Hi Jacob, > > we are doing the opposite direction: we enrich data with geo

Re: Is anyone using Beam for geo use cases?

Hi Jacob, we are doing the opposite direction: we enrich data with geo coordinates from textual address using Google Maps API with Cloud Dataflow. Are you interested in this use-case? Csabi On Thu, 19 Oct 2017 at 21:00 Jacob Marble wrote: > Is anyone using Beam to solve

Re: KafkaIO and Avro

Thanks Eugene. That does compile, although the rest of the pipeline doesn't seem happy. The next line is: .apply(Values.create()) But that now doesn't compile with the following error: /usr/src/kafka/src/main/java/com/andrewjones/KafkaAvroConsumerExample.j- ava:[54,17] cannot find symbol