On Thu, Mar 8, 2018 at 9:38 PM Eugene Kirpichov
wrote:
> I think it may have been an API design mistake to put the S3 region into
> PipelineOptions.
>
+1, IMHO it's generally a mistake to put any transform configuration into
PipelineOptions for exactly this reason.
>
I think it may have been an API design mistake to put the S3 region into
PipelineOptions. PipelineOptions are global per pipeline, whereas it's
totally reasonable to access S3 files in different regions even from the
code of a single DoFn running on a single element. The same applies to
The "hint" would probably to use hints :) - indees this joke refers to the
hint thread.
Long story short with hints you should be able to say "use that specialize
config here".
Now, personally, I'd like to see a way to specialize config per transform.
With an hint an easy way is to use a prefix:
Thank you, Kenn!
Shen
On Thu, Mar 8, 2018 at 9:58 PM, Kenneth Knowles wrote:
>
>
> On Thu, Mar 8, 2018 at 6:50 PM Shen Li wrote:
>
>> Hi Kenn,
>>
>> I just want to confirm that I understand it correctly.
>>
>> > - You know that W is expired only when you
On Thu, Mar 8, 2018 at 6:50 PM Shen Li wrote:
> Hi Kenn,
>
> I just want to confirm that I understand it correctly.
>
> > - You know that W is expired only when you can be sure that no main
> input element could reference it.
>
> This is determined by the *main input*
Hi Kenn,
I just want to confirm that I understand it correctly.
> - You know that W is expired only when you can be sure that no main
input element could reference it.
This is determined by the *main input* watermark, allowedLateness, and
maximumLookback, right?
I see. Thank you Kenn and Lukasz.
Best,
Shen
On Thu, Mar 8, 2018 at 7:46 PM, Kenneth Knowles wrote:
> I think the description of when a side input is ready vs expired is the
> trouble here.
>
> - You know that W is expired only when you can be sure that no main input
>
Great talk, Eugene.
Ted, will share more info on Kafka IO for Python soon :)
- Cham
On Thu, Mar 8, 2018 at 4:55 PM Ted Yu wrote:
> I see.
>
> I have added myself as watcher on BEAM-3788.
>
> Thanks
>
> On Thu, Mar 8, 2018 at 4:51 PM, Eugene Kirpichov
I see.
I have added myself as watcher on BEAM-3788.
Thanks
On Thu, Mar 8, 2018 at 4:51 PM, Eugene Kirpichov
wrote:
> Hi Ted - KafkaIO is not yet implemented using Splittable DoFn's (it was
> implemented before SDFs existed and hasn't been rewritten yet), but it will
>
Hi Ted - KafkaIO is not yet implemented using Splittable DoFn's (it was
implemented before SDFs existed and hasn't been rewritten yet), but it will
be, once more runners catch up with the support: currently we have Dataflow
and Flink. +Chamikara Jayalath is currently working
I think the description of when a side input is ready vs expired is the
trouble here.
- You know that W is expired only when you can be sure that no main input
element could reference it.
- You know that W is ready *even if it got no data* if the input that
would end up in W would be dropped
Eugene:
Very informative talk.
I looked at:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java
Is there some example showing how OffsetRangeTracker works with Kafka
partition(s) ?
Thanks
On Thu, Mar 8, 2018 at 3:58 PM, Eugene Kirpichov
Hi Lukasz,
Let's explain this problem using a specific example.
Say I have a main input element X, which accesses side input window W. When
X arrives at a ParDo operator, W is not ready and not expired either. So,
in this case, the ParDo should push back X and wait for W to become ready.
Say,
Hi Thomas!
In case of tailing a Kafka partition, the restriction would be
[start_offset, infinity), and it would keep being split by checkpointing
into [start_offset, end_offset) and [end_offset, infinity)
On Thu, Mar 8, 2018 at 3:52 PM Thomas Weise wrote:
> Eugene,
>
> I
Eugene,
I actually had one question regarding the application of SDF for the Kafka
consumer. Reading through a topic partition can be parallel by splitting a
partition into multiple restrictions (for use cases where order does not
matter). But how would the tail read be managed? I assume there
Yes, the release guide has a segment "Update release specific
configurations" that has a tidbit about this.
On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold wrote:
> The dataflow java worker version wasn't updated on the branch as in past
> releases ... should it be?
>
The dataflow java worker version wasn't updated on the branch as in past
releases ... should it be?
https://issues.apache.org/jira/browse/BEAM-3815
On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau
wrote:
> Can still be provided as a generic one (like the an offset or
I believe your missing over this point: "and also to not expire the side
input till the main input watermark advances beyond the garbage collection
hold of the side input."
On Thu, Mar 8, 2018 at 3:33 PM, Shen Li wrote:
> Hi Lukasz,
>
> Thanks again.
>
> > the runner is
Hi Lukasz,
Thanks again.
> the runner is required to hold back the main input till the side input
is ready
Yes, I understand these requirements. But what if the side input expires
before it becomes ready?
Shen
Terrific! Thanks Eugene. Just the slides themselves are so good, can't wait
for the video.
Do you know when the video might be available?
On Thu, Mar 8, 2018 at 12:16 PM Eugene Kirpichov
wrote:
> Oops that's just the template I used. Thanks for noticing, will regenerate
>
I want to nitpick slightly the wording of "Java-only runner". I would
like/expect that a runner using some specialized Java execution paths would
still be accepting a portable pipeline and using the URNs and URLs to pick
out special codepaths, so it is still different than just leaving the old
Neither, the runner is required to hold back the main input till the side
input is ready and also to not expire the side input till the main input
watermark advances beyond the garbage collection hold of the side input.
On Thu, Mar 8, 2018 at 1:52 PM, Shen Li wrote:
> Hi
Can still be provided as a generic one (like the an offset or key based
one) but good enough for now, right, was just surprising to not see it when
checking the breakage.
Le 8 mars 2018 22:05, "Eugene Kirpichov" a écrit :
All SDF-related method annotations in DoFn are
The runner/engine is responsible for pushing back the main input until the
side input becomes ready. So having the side input significantly delayed
can cause a serious backlog on the main input.
On Thu, Mar 8, 2018 at 1:34 PM, Shen Li wrote:
> Hi Lukasz,
>
> Thanks for the
The side input expires relative to the input watermark of the ParDo so what
your suggesting could only happen if the runner had a bug and expired the
side input before it should have happened or the user pipeline has a bug
and is attempting to access a window for something that would always be
Hi,
When a main input element tries to access an expired side input window
(violating maximumLookback), should ParDo discard the element or treat it
as an error?
Besides, what should ParDo do in the following situation:
1. The side input window W is not expired but unready when the main input
All SDF-related method annotations in DoFn are marked @Experimental. I
guess that should apply to RestrictionTracker too, but I wouldn't be too
worried about that, since it only makes sense to use in the context of
those methods.
On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau
All runners should support portable execution for Java, which should be
just as easy as supporting execution of non-Java pipelines over this API.
As for non-portable "specialized" execution of Java, I think it's a
tradeoff between the overhead of the portability framework vs. the
maintenance cost
Love it. Great flashy title, too :-)
On Thu, Mar 8, 2018 at 12:16 PM Eugene Kirpichov
wrote:
> Oops that's just the template I used. Thanks for noticing, will regenerate
> the PDF and reupload when I get to it.
>
> On Thu, Mar 8, 2018, 11:59 AM Dan Halperin
+1 to Luke's answer of "yes" for everything to be "portable by default".
However, I (always) favor decentralizing this decision as long as the "Beam
model" is respected.
Baseline:
- the input pipeline should always be in portable format
- the results of execution should match portable
Hmm, does sdf api misses some @Experimental then?
To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1
or +0 since none of my tests pass reliably in current state without a retry
strategy making the call useless.
Le 8 mars 2018 21:02, "Reuven Lax" a écrit
@Luskasz: not sure Im the best to host it since I know more gradle
internals that user interface/ecosystem but happy to help. Will also
require a "sudo" merger for this day to merge fixes asap - guess we can
bypass reviews or have a fast cycle plan for this day to avoid it to be a
week?
Le 8 mars
Thanks I had a look, It’s nice suggestions.
Preparing the next step to chose one of them.
Yanael
On Wed 7 Mar 2018 at 9:46 PM, Kenneth Knowles wrote:
> Hi Yanael,
>
> Glad to hear from you! Here is a saved filter for Jira tickets describing
> GSoC project ideas in Beam:
>
>
On Thu, Mar 8, 2018 at 12:02 PM Reuven Lax wrote:
> Does Nexmark use SerializableCoder?
>
That's what the errors (and fix) for RC1 seemed to indicate.
> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw
> wrote:
>
>> I put the validation checklist
Oops that's just the template I used. Thanks for noticing, will regenerate
the PDF and reupload when I get to it.
On Thu, Mar 8, 2018, 11:59 AM Dan Halperin wrote:
> Looks like it was a good talk! Why is it Google Confidential &
> Proprietary, though?
>
> Dan
>
> On Thu,
@Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
what I mean is that I have added enough hints that it works OOTB already.
The rest of my instructions are just how you should override IntelliJ's
defaults to have a proper dev env - mostly just about storing files outside
the
I really like slide 19:
Author: "I made a bigdata programming model"
Reader: "Cool, how does data get in and out?"
Author: "Brb"
On Thu, Mar 8, 2018 at 11:49 AM, Eugene Kirpichov
wrote:
> Hey all,
>
> The slides for my yesterday's talk at Strata San Jose
Looks like it was a good talk! Why is it Google Confidential & Proprietary,
though?
Dan
On Thu, Mar 8, 2018 at 11:49 AM, Eugene Kirpichov
wrote:
> Hey all,
>
> The slides for my yesterday's talk at Strata San Jose https://conferences.
>
Hey all,
The slides for my yesterday's talk at Strata San Jose
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63696
have
been posted on the talk page. They may be of interest both to users and IO
authors.
Thanks.
Romain, would you like to host/plan/run the Gradle fixit day?
On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
wrote:
> +1 for the general idea of fixit day/week for Gradle.
>
> Agree with what Łukasz said. Some of these performance tests are new and
> are flaky due to
+1 for the general idea of fixit day/week for Gradle.
Agree with what Łukasz said. Some of these performance tests are new and
are flaky due to other issues that were discovered during the process of
adding the test.
I think the high level blocker is updating performance testing framework to
use
I ran some very pessimistic pipelines that were shuffle heavy (Random KV ->
GBK -> IdentityDoFn) and found that the performance overhead was 15% when
executed with Dataflow. This is a while back and there was a lot of
inefficiencies due to coder encode/decode cycles and based upon profiling
Performance, due to the extra gRPC hop.
On Thu, Mar 8, 2018 at 11:08 AM, Lukasz Cwik wrote:
> The goal is to use containers (and similar technologies) in the future. It
> really hinders pipeline portability between runners if you also have to
> deal with the dependency
The goal is to use containers (and similar technologies) in the future. It
really hinders pipeline portability between runners if you also have to
deal with the dependency conflicts between Flink/Dataflow/Spark/...
execution runtimes.
What kinds of penalty are you referring to (perf, user
I'm curious if pipelines that are exclusively Java will be executed (when
running on Flink or other JVM based runnner) in separate harness containers
also? This would impose a significant penalty compared to the current
execution model. Will this be something the user can control?
Thanks,
Thomas
Yes, SDF is an experimental API at this point, so backwards incompatible
changes are allowed and expected.
On Thu, Mar 8, 2018, 10:42 AM Robert Bradshaw wrote:
> I put the validation checklist spreadsheet is up at
>
I put the validation checklist spreadsheet is up at
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475
Regarding the direct runner regression on query 10, this is understandable
given how mutation detection has been changed for
I support leaving things as they stand now -- thanks for finding a good way
out of an uncomfortable situation.
That said, two things need to happen:
(1) SGA needs to be filed asap, per Board feedback in the last report, and
(2) releases cannot contain any code from the Go SDK before formally
Hi David,
you can use a simple ParDo(DoFn) at the end of your pipeline.
For instance, you can mimic what we do in the ElasticsearchIO Write:
https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L789
I also
Hi ,
I have a use case where I would like to post the results to an API , I was
thinking of using Sink , then its documented to not use that , what is the
recommended way of doing this .
My flow is
Read from a flow--> do transformation / validation --> post the results to rest
API and also
2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
>
I confirm that the new release fixes both problems reported previously:
- python package name
- nexmark query 10 mutability issue with the direct runner.
One extra regression is that the the fix produced a way longer
execution time on the query.
Not sure if a blocker but worth tracking.
Query
+1
I was wondering if we can also add a playlist that links to
presentations Beamers have done in different conferences, e.g. some of
the public available talks from the past by Frances/Tyler/others are
worth to be included so they can be easily found. (of course not sure
if we need approval from
53 matches
Mail list logo