Re: [DISCUSS] Beam IO native IO

Thanks Raghu,Is there a Maven dependency for Kafka 0.9.x pls?Am trying to find the nightly built jar that has Beam KafkaIO.Thanks for your help.Cheers From: Raghu Angadi To: dev@beam.incubator.apache.org Sent: Thursday, April 28, 2016 11:21 AM Subject: Re:

Re: [DISCUSS] Beam IO native IO

Sorry JB.Where can I download a " SNAPSHOT/nightly build" jar?My apologies...I searched couldnt find it. From: Jean-Baptiste Onofré To: dev@beam.incubator.apache.org Sent: Thursday, April 28, 2016 11:06 AM Subject: Re: [DISCUSS] Beam IO native IO The KafkaIO is

Re: [DISCUSS] Beam IO native IO

It would also be good to see examples of native I/O being noticeably more performant. The CassandraIO and PubsubIO are examples of corresponding Beam sources missing rather than Beam sources being slow. I think it is better to look into why Beam IO can not be performant. I think in most cases it

Re: [DISCUSS] Adding Some Sort of SideInputRunner

No worries :-) and thanks for the detailed answers! I still have one question, though: you wrote that "The side input is considered ready when there has been any data output/added to the PCollection that it is being read as a side input. So the upstream trigger controls this." How does this work

Re: [DISCUSS] DSL and type converter

Hi Davor, agree that's a DSL concern, and it will be part of the DSL. I tried to express my thinking about the DSL work in progress ;) Thanks ! Regards JB On 04/28/2016 06:34 PM, Davor Bonaci wrote: Typing is often hard ;) This sounds like a DSL-specific design decision. Perhaps we could

Re: [DISCUSS] Beam IO native IO

Hi Robert, I fully agree that we should *ALWAYS* use the Beam IO. The reason why I'm asking is when we have difference between the Beam IO and runner native connector, let's take Cassandra. For now, we don't have a Cassandra IO, but it's gonna change. So, let's assume we have a Cassandra

IO timelines (Was: How to read/write avro data using FlinkKafka Consumer/Producer)

[ Moving over to the dev@ list ] I think we should be aiming a little higher than "trying out Beam" ;) Beam SDK currently has built-in IOs for Kafka, as well as for all important Google Cloud Platform services. Additionally, there are pull requests for Firebase and Cassandra. This is not bad,

Re: [DISCUSS] Beam IO native IO

+1 I agree with what Robert said and Davor laid out in more detail. Portability is one of the primary concerns of Beam. On Thu, 28 Apr 2016 at 18:27 Davor Bonaci wrote: > Generally speaking, the SDKs define all user APIs, including all IOs. We > should strive that

Re: [DISCUSS] Adding Some Sort of SideInputRunner

On Thu, Apr 28, 2016 at 1:26 AM, Aljoscha Krettek wrote: > Bump. > > I'm afraid this might have gotten lost during the conferences/summits. > > On Thu, 21 Apr 2016 at 13:30 Aljoscha Krettek wrote: > > > Ok, I'll try and start such a design. Before I can

Re: [DISCUSS] DSL and type converter

Typing is often hard ;) This sounds like a DSL-specific design decision. Perhaps we could start by specifying what the objectives and capabilities of this particular DSL would be. I think we would then be able to comment on the advantages and disadvantages of various choices. Otherwise, it is

Re: [DISCUSS] Beam IO native IO

On Thu, Apr 28, 2016 at 5:41 AM, Jean-Baptiste Onofré wrote: > Hi all, > > regarding the recent threads on the mailing list, I would like to start a > format discussion around the IO. > As we can expect the first contributions on this area (I already have some > work in

[DISCUSS] Beam IO native IO

Hi all, regarding the recent threads on the mailing list, I would like to start a format discussion around the IO. As we can expect the first contributions on this area (I already have some work in progress around this ;)), I think it's a fair discussion to have. Now, we have two kinds of

Re: [DISCUSS] Adding Some Sort of SideInputRunner

Bump. I'm afraid this might have gotten lost during the conferences/summits. On Thu, 21 Apr 2016 at 13:30 Aljoscha Krettek wrote: > Ok, I'll try and start such a design. Before I can start, I have a few > questions about how the side inputs actually work. Some of it is in