Re: Slack Channel Request

2017-04-13 Thread Satish Duggana 
Georg,
Invite sent to you.

On Fri, Apr 14, 2017 at 11:06 AM, Georg Heiler 
wrote:

> Please add me toi
>
> Mingmin Xu  schrieb am Fr. 14. Apr. 2017 um 07:06:
>
>> both sent.
>>
>> On Thu, Apr 13, 2017 at 10:00 PM, Anant Bhandarkar > impactanalytics.co> wrote:
>>
>>> Would love to be part of beam group on slack.
>>> Also please add anil.b...@impactanaytics.co
>>> Thanks,
>>> Anant
>>>
>>> On 14-Apr-2017 9:15 AM, "Mingmin Xu"  wrote:
>>>
 @James, @Jingsong, @Tom, invite sent.

 On Thu, Apr 13, 2017 at 8:34 PM, Tom Pollard <
 tpoll...@flashpoint-intel.com> wrote:

> If it's not inconvenient, I'd also like an invitation to the Slack
> channel.
>
> Tom
>
>
> On Apr 13, 2017, at 11:31 PM, JingsongLee 
> wrote:
>
> Please add me too.
>
> Best,
>
> JingsongLee
>
>
> --
> From:James 
> Time:2017 Apr 14 (Fri) 11:00
> To:user 
> Subject:Re: Slack Channel Request
>
> Could I also have an invite please?
>
> On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote:
> > Invite sent.
> >
> > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  > wrote:
> >
> > > Hi Jean,
> > >
> > > Thank you for your reply. I am eagerly waiting for the
> other options.
> > >
> > > Regards,
> > > Prabeesh K.
> > >
> > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  > wrote:
> > >
> > >> Unfortunately we reached the max number of people on Slack (90).
> > >>
> > >> Let me see what we can do.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Can someone please add me to the Apache Beam slack channel?
> > >>>
> > >>> Regards,
> > >>>
> > >>> Prabeesh K.
> > >>>
> > >>>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbono...@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> > >
> > >
> >
>
>
>


 --
 
 Mingmin

>>>
>>
>>
>> --
>> 
>> Mingmin
>>
>

Re: Slack Channel Request

2017-04-13 Thread Georg Heiler 
Please add me toi
Mingmin Xu  schrieb am Fr. 14. Apr. 2017 um 07:06:

> both sent.
>
> On Thu, Apr 13, 2017 at 10:00 PM, Anant Bhandarkar <
> anant.bhandar...@impactanalytics.co> wrote:
>
>> Would love to be part of beam group on slack.
>> Also please add anil.b...@impactanaytics.co
>> Thanks,
>> Anant
>>
>> On 14-Apr-2017 9:15 AM, "Mingmin Xu"  wrote:
>>
>>> @James, @Jingsong, @Tom, invite sent.
>>>
>>> On Thu, Apr 13, 2017 at 8:34 PM, Tom Pollard <
>>> tpoll...@flashpoint-intel.com> wrote:
>>>
 If it's not inconvenient, I'd also like an invitation to the Slack
 channel.

 Tom


 On Apr 13, 2017, at 11:31 PM, JingsongLee 
 wrote:

 Please add me too.

 Best,

 JingsongLee


 --
 From:James 
 Time:2017 Apr 14 (Fri) 11:00
 To:user 
 Subject:Re: Slack Channel Request

 Could I also have an invite please?

 On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote:
 > Invite sent.
 >
 > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  wrote:
 >
 > > Hi Jean,
 > >

 > > Thank you for your reply. I am eagerly waiting for the other options.
 > >
 > > Regards,
 > > Prabeesh K.
 > >
 > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  wrote:
 > >
 > >> Unfortunately we reached the max number of people on Slack (90).
 > >>
 > >> Let me see what we can do.
 > >>
 > >> Regards
 > >> JB
 > >>
 > >>
 > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
 > >>
 > >>> Hi,
 > >>>
 > >>> Can someone please add me to the Apache Beam slack channel?
 > >>>
 > >>> Regards,
 > >>>
 > >>> Prabeesh K.
 > >>>
 > >>>
 > >> --
 > >> Jean-Baptiste Onofré
 > >> jbono...@apache.org
 > >> http://blog.nanthrax.net
 > >> Talend - http://www.talend.com
 > >>
 > >
 > >
 >



>>>
>>>
>>> --
>>> 
>>> Mingmin
>>>
>>
>
>
> --
> 
> Mingmin
>

Re: Slack Channel Request

2017-04-13 Thread Mingmin Xu 
both sent.

On Thu, Apr 13, 2017 at 10:00 PM, Anant Bhandarkar <
anant.bhandar...@impactanalytics.co> wrote:

> Would love to be part of beam group on slack.
> Also please add anil.b...@impactanaytics.co
> Thanks,
> Anant
>
> On 14-Apr-2017 9:15 AM, "Mingmin Xu"  wrote:
>
>> @James, @Jingsong, @Tom, invite sent.
>>
>> On Thu, Apr 13, 2017 at 8:34 PM, Tom Pollard <
>> tpoll...@flashpoint-intel.com> wrote:
>>
>>> If it's not inconvenient, I'd also like an invitation to the Slack
>>> channel.
>>>
>>> Tom
>>>
>>>
>>> On Apr 13, 2017, at 11:31 PM, JingsongLee 
>>> wrote:
>>>
>>> Please add me too.
>>>
>>> Best,
>>>
>>> JingsongLee
>>>
>>>
>>> --
>>> From:James 
>>> Time:2017 Apr 14 (Fri) 11:00
>>> To:user 
>>> Subject:Re: Slack Channel Request
>>>
>>> Could I also have an invite please?
>>>
>>> On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote:
>>> > Invite sent.
>>> >
>>> > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K. >> > wrote:
>>> >
>>> > > Hi Jean,
>>> > >
>>> > > Thank you for your reply. I am eagerly waiting for the o
>>> ther options.
>>> > >
>>> > > Regards,
>>> > > Prabeesh K.
>>> > >
>>> > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré >> > wrote:
>>> > >
>>> > >> Unfortunately we reached the max number of people on Slack (90).
>>> > >>
>>> > >> Let me see what we can do.
>>> > >>
>>> > >> Regards
>>> > >> JB
>>> > >>
>>> > >>
>>> > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
>>> > >>
>>> > >>> Hi,
>>> > >>>
>>> > >>> Can someone please add me to the Apache Beam slack channel?
>>> > >>>
>>> > >>> Regards,
>>> > >>>
>>> > >>> Prabeesh K.
>>> > >>>
>>> > >>>
>>> > >> --
>>> > >> Jean-Baptiste Onofré
>>> > >> jbono...@apache.org
>>> > >> http://blog.nanthrax.net
>>> > >> Talend - http://www.talend.com
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>>
>>>
>>
>>
>> --
>> 
>> Mingmin
>>
>


-- 

Mingmin

Re: Slack Channel Request

2017-04-13 Thread Mingmin Xu 
@James, @Jingsong, @Tom, invite sent.

On Thu, Apr 13, 2017 at 8:34 PM, Tom Pollard 
wrote:

> If it's not inconvenient, I'd also like an invitation to the Slack channel.
>
> Tom
>
>
> On Apr 13, 2017, at 11:31 PM, JingsongLee  wrote:
>
> Please add me too.
>
> Best,
>
> JingsongLee
>
>
> --
> From:James 
> Time:2017 Apr 14 (Fri) 11:00
> To:user 
> Subject:Re: Slack Channel Request
>
> Could I also have an invite please?
>
> On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote:
> > Invite sent.
> >
> > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  > wrote:
> >
> > > Hi Jean,
> > >
> > > Thank you for your reply. I am eagerly waiting for the other options.
> > >
> > > Regards,
> > > Prabeesh K.
> > >
> > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  > wrote:
> > >
> > >> Unfortunately we reached the max number of people on Slack (90).
> > >>
> > >> Let me see what we can do.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Can someone please add me to the Apache Beam slack channel?
> > >>>
> > >>> Regards,
> > >>>
> > >>> Prabeesh K.
> > >>>
> > >>>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbono...@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> > >
> > >
> >
>
>
>


-- 

Mingmin

Re: Slack Channel Request

2017-04-13 Thread Tom Pollard 
If it's not inconvenient, I'd also like an invitation to the Slack channel.

Tom

> On Apr 13, 2017, at 11:31 PM, JingsongLee  wrote:
> 
> Please add me too. 
> 
> Best, 
> 
> JingsongLee
> 
> 
> --
> From:James 
> Time:2017 Apr 14 (Fri) 11:00
> To:user 
> Subject:Re: Slack Channel Request
> 
> Could I also have an invite please?
> 
> On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote: 
> > Invite sent.
> > 
> > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  wrote:
> > 
> > > Hi Jean,
> > >
> > > Thank you for your reply. I am eagerly waiting for the other options.
> > >
> > > Regards,
> > > Prabeesh K.
> > >
> > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  
> > > wrote:
> > >
> > >> Unfortunately we reached the max number of people on Slack (90).
> > >>
> > >> Let me see what we can do.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Can someone please add me to the Apache Beam slack channel?
> > >>>
> > >>> Regards,
> > >>>
> > >>> Prabeesh K.
> > >>>
> > >>>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbono...@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> > >
> > >
> > 



smime.p7s
Description: S/MIME cryptographic signature

Re: Slack Channel Request

2017-04-13 Thread JingsongLee 
Please add me too. 
Best, 
JingsongLee

--From:James 
Time:2017 Apr 14 (Fri) 11:00To:user 
Subject:Re: Slack Channel Request
Could I also have an invite please?

On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote: 
> Invite sent.
> 
> On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  wrote:
> 
> > Hi Jean,
> >
> > Thank you for your reply. I am eagerly waiting for the other options.
> >
> > Regards,
> > Prabeesh K.
> >
> > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  wrote:
> >
> >> Unfortunately we reached the max number of people on Slack (90).
> >>
> >> Let me see what we can do.
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
> >>
> >>> Hi,
> >>>
> >>> Can someone please add me to the Apache Beam slack channel?
> >>>
> >>> Regards,
> >>>
> >>> Prabeesh K.
> >>>
> >>>
> >> --
> >> Jean-Baptiste Onofré
> >> jbono...@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >
> >
> 

Re: Slack Channel Request

2017-04-13 Thread James 
Could I also have an invite please?

On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote: 
> Invite sent.
> 
> On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  wrote:
> 
> > Hi Jean,
> >
> > Thank you for your reply. I am eagerly waiting for the other options.
> >
> > Regards,
> > Prabeesh K.
> >
> > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  wrote:
> >
> >> Unfortunately we reached the max number of people on Slack (90).
> >>
> >> Let me see what we can do.
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
> >>
> >>> Hi,
> >>>
> >>> Can someone please add me to the Apache Beam slack channel?
> >>>
> >>> Regards,
> >>>
> >>> Prabeesh K.
> >>>
> >>>
> >> --
> >> Jean-Baptiste Onofré
> >> jbono...@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >
> >
>

Re: beam + scala + streamline

2017-04-13 Thread Davor Bonaci 
Hi Georg --
Great to see you are evaluating Beam for your scenario.


> > someone told me that e.g. the flink runner for beam seems to be slower
>> than a
>> > native flink job. Is this true? Did you observe such characteristics
>> for several
>> > runners?
>>
>
This should not be true in a general sense -- the performance should be
~equivalent. The Flink runner in Beam constructs a "native" Flink pipeline;
the overhead of invoking user-defined functions is often to set a few
fields and invoke a function, which is negligible. The actual performance
of a pipeline tend to depend on other factors -- stragglers, how fast the
system can adopt to changing load, etc.

(If there's a gap somewhere, it is likely a bug -- and we'd like to know
about it and fix it.)

> in case I want to use some low level functionality (specific to a runner)
>> like
>> > ML, graph processing or sql-tables api, is it possible to just drop
>> from the
>> > beam API one level deeper to the actual runner and sort of mesh beam
>> with runner
>> > native code to integrate these features?
>
>
The Beam API, in a general sense, doesn't provide such hooks, as that would
break portability.

I wouldn't advise this, but technically, it wouldn't be hard -- you'd
create a PTransform in Beam, and modify the runner to replace it with their
own specific implementation. Instead, I'd suggest using Beam's abstractions
and, in the case of a missing pattern or a feature, to work with us to
augment the Beam model accordingly.

Hope this helps -- and that you find Beam fitting for your case. Please let
us know if we can assist any further -- thanks!

Davor

Re: Unhelpful ExceptionInChainedStubException errors with Flink runner

2017-04-13 Thread Aljoscha Krettek 
Sorry for the slow response. I've had this marked in my inbox but due to
the preparations for Flink Forward I didn't have time to look at it yet.
I'll definitely still do that.




On Thu, Apr 13, 2017, at 05:15, Davor Bonaci wrote:

> Aljoscha, any ideas perhaps?

> 

> On Wed, Apr 5, 2017 at 12:52 PM, peay  wrote:

>> Hello,

>> 

>> I've been having some trouble with debugging exceptions in user code
>> when using the Flink runner. Here's an example from a
>> window/DoFn/GroupByKey pipeline.
>> 

>> ERROR o.a.f.runtime.operators.BatchTask - Error in task code:  CHAIN
>> MapPartition (MapPartition at ParDo(MyDoFn)) -> FlatMap
>> (Transform/Windowing/Window.Assign.out) -> Map (Key Extractor) ->
>> GroupCombine (GroupCombine at GroupCombine: GroupByKey) -> Map (Key
>> Extractor) (1/8)
>> org.apache.beam.sdk.util.UserCodeException: org.apache.flink.runtime-
>> .operators.chaining.ExceptionInChainedStubException
>> at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeE-
>> xception.java:36) ~[beam-sdks-java-core-0.6.0.jar:0.6.0]
>> at org.org.my.pipelines.MyDoFn$auxiliary$s09rfuPj.invokeProc-
>> essElement(Unknown Source) ~[na:na]
>> at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProce-
>> ssElement(SimpleDoFnRunner.java:198) ~[beam-runners-core-java-
>> 0.6.0.jar:0.6.0]
>> at org.apache.beam.runners.core.SimpleDoFnRunner.processElem-
>> ent(SimpleDoFnRunner.java:156) ~[beam-runners-core-java-
>> 0.6.0.jar:0.6.0]
>> at org.apache.beam.runners.flink.translation.functions.Flink-
>> DoFnFunction.mapPartition(FlinkDoFnFunction.java:109) 
>> ~[beam-runners-flink_2.10-
>> 0.6.0.jar:0.6.0]
>> at org.apache.flink.runtime.operators.MapPartitionDriver.run-
>> (MapPartitionDriver.java:103) ~[flink-runtime_2.10-
>> 1.2.0.jar:1.2.0]
>> at org.apache.flink.runtime.operators.BatchTask.run(BatchTas-
>> k.java:490) [flink-runtime_2.10-1.2.0.jar:1.2.0]
>> at org.apache.flink.runtime.operators.BatchTask.invoke(Batch-
>> Task.java:355) [flink-runtime_2.10-1.2.0.jar:1.2.0]
>> at
>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
>> [flink-runtime_2.10-1.2.0.jar:1.2.0]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

>> org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubEx-
>> ception: null
>> at org.apache.flink.runtime.operators.chaining.ChainedFlatMa-
>> pDriver.collect(ChainedFlatMapDriver.java:82) ~[flink-runtime_2.10-
>> 1.2.0.jar:1.2.0]
>> at org.apache.flink.runtime.operators.util.metrics.CountingC-
>> ollector.collect(CountingCollector.java:35) ~[flink-runtime_2.10-
>> 1.2.0.jar:1.2.0]
>> at org.apache.beam.runners.flink.translation.functions.Flink-
>> DoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java-
>> :138) ~[beam-runners-flink_2.10-0.6.0.jar:0.6.0]
>> at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnContext-
>> .outputWindowedValue(SimpleDoFnRunner.java:351) 
>> ~[beam-runners-core-java-
>> 0.6.0.jar:0.6.0]
>> at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcess-
>> Context.output(SimpleDoFnRunner.java:545) ~[beam-runners-core-java-
>> 0.6.0.jar:0.6.0]
>> at org.my.pipelines.MyDoFn.processElement(MyDoFn.java:49) 
>> ~[pipelines-
>> 0.1.jar:na]
>> 

>> The top stacktrace references some kind of anonymous
>> `invokeProcessElement(Unknown Source)` which is not really
>> informative. The bottom stacktrace references my call to
>> `context.output()`, which is even more confusing. I've gone through
>> fixing a couple issue by manually try/catching and logging directly
>> from within `processElement`, but this is far from ideal. Any advice
>> on how to interpret those and possibly set things up in order to get
>> more helpful error messages would be really helpful.
>> 

>> Running Beam 0.6, Flink 1.2.

>> 

>> Thanks!

>> 

>>