Re: [discuss] Using a logger hierarchy in Python

[Pablo Estrada]

Thanks all,
2/3 of PRs are merged (using _LOGGER). It should be pretty easy to
switch the variable name to _log via sed.
Best
-P.

On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver  wrote:

> +1 for per-module loggers (what Robert said).
>
> On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri  wrote:
>
>> +1, but can we use something less verbose and shift key heavy than
>> _LOGGER like log or _log?
>>
>> Also please dedupe with these existing bugs:
>> https://issues.apache.org/jira/browse/BEAM-3523
>> https://issues.apache.org/jira/browse/BEAM-1825
>>
>> On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise  wrote:
>>
>>> Awesome, thanks Chad!
>>>
>>> On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova 
>>> wrote:
>>>
 Hi Thomas,


> Will this include the ability for users to configure logging via
> pipeline options?
>

 We're working on a proposal to allow pluggable logging handlers that
 can be configured via pipeline options.  For example, it would allow you to
 add a new logging handler for StackDriver or Elasticsearch.  Will hopefully
 have a document to share soon.

 -chad

Re: [ANNOUNCE] New committer: Brian Hulette

[Chamikara Jayalath]

Congrats, Brian!

On Fri, Nov 15, 2019 at 11:39 AM Yichi Zhang  wrote:

> Congratulations, Brian!
>
> On Fri, Nov 15, 2019 at 11:12 AM Alan Myrvold  wrote:
>
>> Congratulations, Brian!
>>
>> On Fri, Nov 15, 2019 at 11:11 AM Yifan Zou  wrote:
>>
>>> Well deserved. Congratulations!
>>>
>>> On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov 
>>> wrote:
>>>
 Congratulations, Brian!

 On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:

> Congrats Brian!
>
>
> On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang 
> wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu 
>> wrote:
>>
>>> Congrats, Brian!
>>>
>>> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
>>> danolive...@google.com> wrote:
>>>
 Congratulations Brian! It's well deserved.

 On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
 aromanenko@gmail.com> wrote:

> Congratulations, Brian!
>
> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>
> Congrats!
>
>
> -Rui
>
> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise 
> wrote:
>
>> Congratulations!
>>
>>
>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done Brian!!!
>>>
>>> Kenn thank you for sharing
>>>
>>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
>>> wrote:
>>>
 Congrats Brian!

 On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
 wrote:

> Congratulations Brian!
> Happy to see this happening and eager to see more of your work!
>
> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka <
> goe...@google.com> wrote:
> >
> > Congrats Brian!
> >
> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
> wrote:
> >>
> >> Congrats Brian!
> >>
> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
> >>
> >> Great news!
> >>
> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov <
> g...@spotify.com> wrote:
> >>>
> >>> Congratulations!
> >>>
> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> 
>  Congratulations, Brian!
> 
>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
> sunjincheng...@gmail.com> wrote:
> >
> > Congratulation Brian!
> >
> > Best,
> > Jincheng
> >
> > Kyle Weaver  于2019年11月15日周五
> 上午7:19写道：
> >>
> >> Thanks for your contributions and congrats Brian!
> >>
> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
> k...@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please join me and the rest of the Beam PMC in
> welcoming a new committer: Brian Hulette
> >>>
> >>> Brian introduced himself to dev@ earlier this year
> and has been contributing since then. His contributions to Beam 
> include
> explorations of integration with Arrow, standardizing coders, 
> portability
> for schemas, and presentations at Beam events.
> >>>
> >>> In consideration of Brian's contributions, the Beam
> PMC trusts him with the responsibilities of a Beam committer [1].
> >>>
> >>> Thank you, Brian, for your contributions and looking
> forward to many more!
> >>>
> >>> Kenn, on behalf of the Apache Beam PMC
> >>>
> >>> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
> >>
> >>
> >>
> >> --
> >>
> >> This email may be confidential and privileged. If you
> received this communication by mistake, please don't forward it 
> to anyone
> else, please erase all copies and attachments, and please let me 
> know that
> it has gone to the wrong person.
> >>
> >> The above terms reflect a potential business arrangement,
> are provided solely as a basis for further discussion, and are 
> not intended
> to be and do not constitute a legally binding obligation. No 
> legally

Re: [Discuss] Beam mascot

[Alex Van Boxel]

I like the "Angler fish" as well... it certainly faster then a
cuttlefish

 _/
_/ Alex Van Boxel


On Fri, Nov 15, 2019 at 9:57 PM Kirill Kozlov 
wrote:

> Angler fish? Found a few animated examples that may look interesting [1,
> 2, 3, 4].
> [1] https://www.pinterest.com/pin/121175046208927340/
> [2] https://www.pinterest.com/pin/353180795779533157/
> [3] https://www.pinterest.com/pin/121175046208927334/
> [4]
> https://graphicriver.net/item/cartoon-angler-fish/16828573?ref=gvector_id=1413785108_back=true
>
> On Fri, Nov 15, 2019 at 11:39 AM Aizhamal Nurmamat kyzy <
> aizha...@apache.org> wrote:
>
>> That last sketch of a cuttlefish with a hat is really really good. I vote
>> for that.
>>
>> On Fri, Nov 15, 2019 at 10:33 AM Kenneth Knowles  wrote:
>>
>>> PSA: cuttlefish tentacles look more like their face, not their legs.
>>> Please find attached illustrations to that effect. And also evidence that
>>> they are occasionally fancy. Definitely *not* from a respectable designer
>>> or anyone who could execution a professional logo.
>>>
>>> I do think the PMC needs to shepherd this. I would suggest starting with
>>> an approval vote [1] for animal (no plant ideas?) and then an ASF-style
>>> vote to record the validated result. From there, we can go through more
>>> standard design process.
>>>
>>> Kenn
>>>
>>> [1] https://www.electionscience.org/library/approval-voting/
>>>
>>> On Fri, Nov 15, 2019 at 6:54 AM Maximilian Michels 
>>> wrote:
>>>
 It's great we're having this discussion and we came up with a lot of
 great ideas through it. However, it is unclear how we proceed from
 here.
 Certainly, we can't let designers work with an open-ended discussion on
 the type of mascot.

 Personally, I'm fine with _any_ kind of mascot, as long as it is made
 by
 a decent designer. The respectable designer I'm talking to was very
 generous to donate these sketches for free. I don't think that this is
 to be expected at all. Coming back to the original idea of hiring
 multiple designers, I don't see how we will pay those designers to all
 come up with a logo, unless we all donate money.

 The sketches I've sent might not look like much because there is still
 a
 decent process involved in coming up with the final logo which looks
 good in different sizes, possibly with many iterations. So far I've
 tried to incorporate as many of the suggestions here, but for the sake
 of protecting the designer, I think I'll have to stop doing that. It
 simply won't work.

 How to proceed from here? I think _any_ professionally executed logo
 will do, this is really more about the community agreeing for the
 greater good. Ultimately, I think the PMC might have to decide on how
 to
 proceed here.

 Cheers,
 Max

 On 15.11.19 13:59, Hannah Jiang wrote:
 > I also vote for firefly.
 >
 > On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev <
 valen...@google.com
 > > wrote:
 >
 > I like the firefly sketch a lot, it's my favorite so far.
 >
 > On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw
 > mailto:rober...@google.com>> wrote:
 >
 > #37 from the sketches was the cuttlefish, which would put it
 at
 > (with
 > 4 votes) the most popular so far. I do like the firefly too.
 >
 > On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas >>> > > wrote:
 >  >
 >  > Hi everyone, so exciting to see this convo taking off!
 >  >
 >  > I loved Alex's firefly! -- it can have so many cool
 > variations, and as a stuffed animal is very original.
 >  >
 >  > Other ideas I had are a caterpillar because it looks like a
 > data pipeline, lol or the beaver!
 >  >
 >  > Feedback on the current sketches.
 >  > - They resemble a lot either the Octocat or Totoro [1]
 >  > - I'd like to see something that is completely new and
 > original, pancakes from gRPC is an example[2]
 >  > - Something more caricaturesque is better, since we can
 dress
 > it up and modify it
 >  >
 >  > To move forward, it seems that the animals that were
 winners
 > in this thread are:
 >  >
 >  > Beaver (3)
 >  > Firefly (3)
 >  > Lemur or votes on sketches (3)
 >  > Cuttlefish (2)
 >  > Hedgehog (1)
 >  > Salmon (1)
 >  >
 >  > So let's focus the design proposals on the three winners:
 > beaver, firefly and lemur.
 >  > I'd like to see more options on beavers and fireflies, the
 > current sketch options I think are based on the cuttlefish 

Re: [UPDATE] Preparing for Beam 2.17.0 release

[Mikhail Gryzykhin]

Hi everyone,

There's still an outstanding cherry-pick PR that I can't merge due to tests
failing on it and release branch validation PR
. Once I get tests green, I'll
send another update and review outstanding open issues.

--Mikhail

On Fri, Nov 15, 2019 at 10:40 AM Thomas Weise  wrote:

> Any update regarding the release?
>
> The list still shows 10 open issues:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%202.17.0%20and%20resolution%20is%20EMPTY
>
> Is the RC blocked on those?
>
>
>
>
>
>
> On Mon, Oct 28, 2019 at 12:46 PM Ahmet Altay  wrote:
>
>>
>>
>> On Mon, Oct 28, 2019 at 12:44 PM Gleb Kanterov  wrote:
>>
>>> It looks like BigQueryIO DIRECT_READ is broken since 2.16.0, I've added
>>> a ticket describing the problem and possible fix, see BEAM-8504
>>>  [1].
>>>
>>
>> Should this be added to 2.16 blog post as a known issue?
>>
>>
>>>
>>> [1]: https://issues.apache.org/jira/browse/BEAM-8504
>>>
>>> On Wed, Oct 23, 2019 at 9:19 PM Kenneth Knowles  wrote:
>>>
 I opened https://github.com/apache/beam/pull/9862 to raise the
 documentation of Fix Version to the top level. It also includes the write
 up of Jira priorities, to make clear that "Blocker" priority does not refer
 to release blocking.

 On Wed, Oct 23, 2019 at 11:16 AM Kenneth Knowles 
 wrote:

> I've gone over the tickets and removed Fix Version from many of them
> that do not seem to be critical defects. If I removed Fix Version from a
> ticket you care about, please feel free to add it back. I am not trying to
> decide what is in/out of the release, just trying to triage the Jira data
> to match expected practices.
>
> It should probably be documented somewhere outside of the release
> guide. As far as I can tell, the fact that we triage them down to zero is
> the only place we mention that it is used to indicate release blockers and
> not used for feature targets.
>
> Kenn
>
> On Wed, Oct 23, 2019 at 10:40 AM Kenneth Knowles 
> wrote:
>
>>  Wow, 28 release blocking tickets! That is the most I've ever seen,
>> by far. Many appear to be feature requests, not release-blocking 
>> defects. I
>> believe this is not according to our normal best practice. The release
>> cadence should not wait for features in progress, with exceptions 
>> discussed
>> on dev@. As a matter of best practice, I think we should triage
>> feature requests to not have Fix Version set until it has been discussed 
>> on
>> dev@.
>>
>> Kenn
>>
>> On Wed, Oct 23, 2019 at 9:55 AM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi all,
>>>
>>> Beam 2.17 release branch cut is scheduled today (2019/10/23)
>>> according to the release calendar [1].  I'll start working on the
>>> branch cutoff and later work on cherry picking blocker fixes.
>>>
>>> If you have release blocking issues for 2.17 please mark their "Fix
>>> Version" as 2.17.0 [2]. This tag is already created in JIRA in case you
>>> would like to move any non-blocking issues to that version.
>>>
>>> There is a decent amount of open bugs to be resolved in 2.17.0 [2]
>>> and only 4 [3] are marked as blockers. Please, review those if these 
>>> bugs
>>> are actually to be resolved in 2.17.0 and prioritize fixes if possible.
>>>
>>> Any thoughts, comments, objections?
>>>
>>> Regards.
>>> Mikhail.
>>>
>>>
>>> [1]
>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>> [2]
>>> https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.17.0
>>> [3]
>>> https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%202.17.0
>>>
>>

Re: session window puzzle

[Kenneth Knowles]

On Wed, Nov 13, 2019 at 7:39 PM Aaron Dixon  wrote:

> This is a great help. Thank you. I like the custom window solution
> pattern as a way to hold the watermark and merge down to keep the watermark
> where it is needed. Perhaps there is some interesting generalized session
> window here.. I'll have to digest the stateful DoFn approach. Avoiding
> unnecessary shuffles is a good note.
>
> As a side note, there is MIN, MAX and END_OF_WINDOW TimestampCombiner. Has
> it been discussed to ever allow more customization here? Seems like
> customizing the combiner with element-awareness would have solved this
> problem, as well.
>

Good question :-). In fact, the original design was OutputTimeFn that was a
custom user-defined function.

 - Discussed a bit here:
https://docs.google.com/document/d/12r7frmxNickxB5tbpuEh_n35_IJeVZn1peOrBrhhP6Y/edit#heading=h.45gyyckqhg4c
 - Removed here: https://github.com/apache/beam/pull/2683

Users found the name and choices confusing. Also, the runner needs to grok
the behavior of it in order to implement many optimizations. Before
portability there were a bunch of instanceof checks and anything not on the
happy path was slower. With portability, it needs to be represented in
protobuf. Reinstating this would mean adding a new CUSTOM enum to
TimestampCombiner and it would have performance costs.

Kenn


>
>
> On Wed, Nov 13, 2019 at 7:56 PM Kenneth Knowles  wrote:
>
>> You've done a very good analysis* and I think your solution is pretty
>> clever. The simple fact is this: the watermark has to be held to the
>> minimum of any output you intend to produce. So for your use case, the hold
>> has to be the timestamp of the Green element. Your solution does hold the
>> watermark to the right time. I have a couple thoughts that may be helpful.
>>
>> 0. If you partition by user does the stream contain a bunch of Orange,
>> Green, Blue elements? Is it possible that a session contains multiple
>> [Orange, Green, Blue] sequences? Is it possible that an [Orange, Green,
>> Blue] sequence is split across multiple sessions?
>>
>> 1. In your proposed solution, it probably could be expressed as a new
>> merging WindowFn. You would assign each Green element to two tagged windows
>> that were GreenFromOrange and GreenToBlue type, and have a separate window
>> tag for OrangeWindow and BlueWindow. Then GreenFromOrange merges with
>> OrangeWindow only, etc.
>>
>> 2. This might also turn out simply as a stateful DoFn, where you manually
>> manage what state the funnel is in. When you set a timer to wait for the
>> Orange element, you may need an upcoming feature where you set a timer for
>> a future event time but the watermark is held to the Green element's
>> timestamp. CC Reuven on that use case.
>>
>> What I would like to avoid is you having to do two shuffles (on whatever
>> runner). This should be doable with one.
>>
>> *SessionWindow plus EARLIEST holding up the watermark/pipeline was an
>> early complaint. That is part of why we switched the default to
>> end-of-window (also it is easier to understand and more efficient to
>> compute)
>>
>> Kenn
>>
>> On Wed, Nov 13, 2019 at 3:25 PM Aaron Dixon  wrote:
>>
>>> This is a real use case we have, but simplified:
>>>
>>> My user session look like this: user visits a page, and clicks three
>>> buttons: Orange then Green then Blue.
>>>
>>> I need to compute the average time between Orange & Blue clicks but I
>>> need to window on the timestamp of the green button click.
>>>
>>> In requirements terms: Compute average time between Orange and Blue for
>>> all Green clicks that occur on Monday. (So User could click Orange on
>>> Sunday, Green on Monday and Blue on Tuesday.)
>>>
>>> One strategy is to try to use a single SessionWindow to capture the
>>> entire user session; then calculate the *span* (time between Orange and
>>> Blue clicks) and *then* compute average of all spans.
>>>
>>> To do this the *span*/counts would have to all "land" in a window
>>> representing Monday.
>>>
>>> If I use a SessionWindow w/ TimestampCombiner/EARLIEST then I can make
>>> sure they land in this window using .outputWithTimestamp without worrying
>>> that I'll be regressing the event timestamp.
>>>
>>> Except when I use this Combiner/EARLIEST strategy my watermark is held
>>> up substantially (and incidentally seems to drag the pipeline).
>>>
>>> But if I use Beam's default TimestampCombiner/END_OF_WINDOW then I won't
>>> be able to output the *span* result at a timestamp representing the
>>> Green click.
>>>
>>> So a single SessionWindow seems out. (Unless I'm missing something.)
>>>
>>> The only other strategy I can conceive of at the moment is to capture
>>> *two* sessions, representing each "leg" of the overall session. One
>>> windows on the [Orange,Green] (using END_OF_WINDOW); the other [Green,Blue]
>>> (using EARLIEST). Then I can "join" these two to get both legs together and
>>> compute the overall span. This seems like a quite complicated way to solve
>>> this 

Re: [discuss] Using a logger hierarchy in Python

[Udi Meiri]

+1, but can we use something less verbose and shift key heavy than _LOGGER
like log or _log?

Also please dedupe with these existing bugs:
https://issues.apache.org/jira/browse/BEAM-3523
https://issues.apache.org/jira/browse/BEAM-1825

On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise  wrote:

> Awesome, thanks Chad!
>
> On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova  wrote:
>
>> Hi Thomas,
>>
>>
>>> Will this include the ability for users to configure logging via
>>> pipeline options?
>>>
>>
>> We're working on a proposal to allow pluggable logging handlers that can
>> be configured via pipeline options.  For example, it would allow you to add
>> a new logging handler for StackDriver or Elasticsearch.  Will hopefully
>> have a document to share soon.
>>
>> -chad
>>
>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Discuss] Beam mascot

[Kirill Kozlov]

Angler fish? Found a few animated examples that may look interesting [1, 2,
3, 4].
[1] https://www.pinterest.com/pin/121175046208927340/
[2] https://www.pinterest.com/pin/353180795779533157/
[3] https://www.pinterest.com/pin/121175046208927334/
[4]
https://graphicriver.net/item/cartoon-angler-fish/16828573?ref=gvector_id=1413785108_back=true

On Fri, Nov 15, 2019 at 11:39 AM Aizhamal Nurmamat kyzy 
wrote:

> That last sketch of a cuttlefish with a hat is really really good. I vote
> for that.
>
> On Fri, Nov 15, 2019 at 10:33 AM Kenneth Knowles  wrote:
>
>> PSA: cuttlefish tentacles look more like their face, not their legs.
>> Please find attached illustrations to that effect. And also evidence that
>> they are occasionally fancy. Definitely *not* from a respectable designer
>> or anyone who could execution a professional logo.
>>
>> I do think the PMC needs to shepherd this. I would suggest starting with
>> an approval vote [1] for animal (no plant ideas?) and then an ASF-style
>> vote to record the validated result. From there, we can go through more
>> standard design process.
>>
>> Kenn
>>
>> [1] https://www.electionscience.org/library/approval-voting/
>>
>> On Fri, Nov 15, 2019 at 6:54 AM Maximilian Michels 
>> wrote:
>>
>>> It's great we're having this discussion and we came up with a lot of
>>> great ideas through it. However, it is unclear how we proceed from here.
>>> Certainly, we can't let designers work with an open-ended discussion on
>>> the type of mascot.
>>>
>>> Personally, I'm fine with _any_ kind of mascot, as long as it is made by
>>> a decent designer. The respectable designer I'm talking to was very
>>> generous to donate these sketches for free. I don't think that this is
>>> to be expected at all. Coming back to the original idea of hiring
>>> multiple designers, I don't see how we will pay those designers to all
>>> come up with a logo, unless we all donate money.
>>>
>>> The sketches I've sent might not look like much because there is still a
>>> decent process involved in coming up with the final logo which looks
>>> good in different sizes, possibly with many iterations. So far I've
>>> tried to incorporate as many of the suggestions here, but for the sake
>>> of protecting the designer, I think I'll have to stop doing that. It
>>> simply won't work.
>>>
>>> How to proceed from here? I think _any_ professionally executed logo
>>> will do, this is really more about the community agreeing for the
>>> greater good. Ultimately, I think the PMC might have to decide on how to
>>> proceed here.
>>>
>>> Cheers,
>>> Max
>>>
>>> On 15.11.19 13:59, Hannah Jiang wrote:
>>> > I also vote for firefly.
>>> >
>>> > On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev <
>>> valen...@google.com
>>> > > wrote:
>>> >
>>> > I like the firefly sketch a lot, it's my favorite so far.
>>> >
>>> > On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw
>>> > mailto:rober...@google.com>> wrote:
>>> >
>>> > #37 from the sketches was the cuttlefish, which would put it at
>>> > (with
>>> > 4 votes) the most popular so far. I do like the firefly too.
>>> >
>>> > On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas >> > > wrote:
>>> >  >
>>> >  > Hi everyone, so exciting to see this convo taking off!
>>> >  >
>>> >  > I loved Alex's firefly! -- it can have so many cool
>>> > variations, and as a stuffed animal is very original.
>>> >  >
>>> >  > Other ideas I had are a caterpillar because it looks like a
>>> > data pipeline, lol or the beaver!
>>> >  >
>>> >  > Feedback on the current sketches.
>>> >  > - They resemble a lot either the Octocat or Totoro [1]
>>> >  > - I'd like to see something that is completely new and
>>> > original, pancakes from gRPC is an example[2]
>>> >  > - Something more caricaturesque is better, since we can
>>> dress
>>> > it up and modify it
>>> >  >
>>> >  > To move forward, it seems that the animals that were winners
>>> > in this thread are:
>>> >  >
>>> >  > Beaver (3)
>>> >  > Firefly (3)
>>> >  > Lemur or votes on sketches (3)
>>> >  > Cuttlefish (2)
>>> >  > Hedgehog (1)
>>> >  > Salmon (1)
>>> >  >
>>> >  > So let's focus the design proposals on the three winners:
>>> > beaver, firefly and lemur.
>>> >  > I'd like to see more options on beavers and fireflies, the
>>> > current sketch options I think are based on the cuttlefish and
>>> > the lemur (?)
>>> >  >
>>> >  > I think it's a good idea to get sketches from multiple
>>> > designers, since like someone else pointed out, we'll get
>>> > variations based on their personal styles, and someone else
>>> > mentioned here that we have teams/companies
>>> 

Beam Dependency Check Report (2019-11-15)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
Sphinx
1.8.5
2.2.1
2019-05-20
2019-10-28BEAM-7370
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.esotericsoftware:kryo
4.0.2
5.0.0-RC4
2018-03-20
2019-04-14BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.27.0
2019-02-11
2019-10-21BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
com.google.api.grpc:proto-google-common-protos
1.12.0
1.17.0
2018-06-29
2019-10-04BEAM-6899
com.google.auth:google-auth-library-credentials
0.13.0
0.18.0
2019-01-17
2019-10-09BEAM-6478
com.google.code.gson:gson
2.7
2.8.6
2016-06-14
2019-10-04BEAM-5558
io.grpc:grpc-auth
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5896
io.grpc:grpc-context
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5897
io.grpc:grpc-core
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5898
io.grpc:grpc-netty
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5899
io.grpc:grpc-protobuf
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5900
io.grpc:grpc-stub
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5901
io.grpc:grpc-testing
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5902
io.grpc:protoc-gen-grpc-java
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5903
io.opencensus:opencensus-api
0.21.0
0.24.0
2019-05-01
2019-08-27BEAM-5580
io.opencensus:opencensus-contrib-grpc-metrics
0.21.0
0.24.0
2019-05-01
2019-08-27BEAM-5581
javax.servlet:javax.servlet-api
3.1.0
4.0.1
2013-04-25
2018-04-20BEAM-5750
net.java.dev.javacc:javacc
4.0
7.0.5
2006-03-17
2019-10-20BEAM-5570
net.java.dev.jna:jna
4.1.0
5.5.0
2014-03-06
2019-10-30BEAM-5573
org.apache.hbase:hbase-common
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5560
org.apache.hbase:hbase-hadoop-compat
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5561
org.apache.hbase:hbase-hadoop2-compat
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5562
org.apache.hbase:hbase-server
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5563
org.apache.hbase:hbase-shaded-client
1.2.6
2.2.1
2017-05-29
2019-09-04BEAM-5564
org.apache.hive:hive-cli
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5566
org.apache.hive:hive-common
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5567
org.apache.hive:hive-exec
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5568
org.apache.hive.hcatalog:hive-hcatalog-core
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5569
org.apache.qpid:proton-j
0.13.1
0.33.2
2016-07-02
2019-08-09BEAM-5582
org.apache.solr:solr-core
5.5.4
8.3.0
2017-02-14
2019-10-31BEAM-5589
org.apache.solr:solr-solrj
5.5.4
8.3.0
2017-02-14
2019-10-31BEAM-5590
org.apache.solr:solr-test-framework
5.5.4
8.3.0
2017-02-14
2019-10-31BEAM-5591
org.conscrypt:conscrypt-openjdk
1.1.3
2.2.1
2018-06-04
2019-08-08BEAM-5748
org.eclipse.jetty:jetty-server
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5752
org.eclipse.jetty:jetty-servlet
9.2.10.v20150310
10.0.0-alpha0

Re: [ANNOUNCE] New committer: Brian Hulette

[Yichi Zhang]

Congratulations, Brian!

On Fri, Nov 15, 2019 at 11:12 AM Alan Myrvold  wrote:

> Congratulations, Brian!
>
> On Fri, Nov 15, 2019 at 11:11 AM Yifan Zou  wrote:
>
>> Well deserved. Congratulations!
>>
>> On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov 
>> wrote:
>>
>>> Congratulations, Brian!
>>>
>>> On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:
>>>
 Congrats Brian!


 On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang 
 wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:
>
>> Congrats, Brian!
>>
>> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
>> danolive...@google.com> wrote:
>>
>>> Congratulations Brian! It's well deserved.
>>>
>>> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 Congratulations, Brian!

 On 15 Nov 2019, at 18:27, Rui Wang  wrote:

 Congrats!


 -Rui

 On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise 
 wrote:

> Congratulations!
>
>
> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
> conne...@google.com> wrote:
>
>> Well done Brian!!!
>>
>> Kenn thank you for sharing
>>
>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
>> wrote:
>>
>>> Congrats Brian!
>>>
>>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
>>> wrote:
>>>
 Congratulations Brian!
 Happy to see this happening and eager to see more of your work!

 On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka <
 goe...@google.com> wrote:
 >
 > Congrats Brian!
 >
 > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
 wrote:
 >>
 >> Congrats Brian!
 >>
 >> On 11/15/19 9:58 AM, Reza Rokni wrote:
 >>
 >> Great news!
 >>
 >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov <
 g...@spotify.com> wrote:
 >>>
 >>> Congratulations!
 >>>
 >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 
  Congratulations, Brian!
 
  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
 sunjincheng...@gmail.com> wrote:
 >
 > Congratulation Brian!
 >
 > Best,
 > Jincheng
 >
 > Kyle Weaver  于2019年11月15日周五
 上午7:19写道：
 >>
 >> Thanks for your contributions and congrats Brian!
 >>
 >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
 k...@apache.org> wrote:
 >>>
 >>> Hi all,
 >>>
 >>> Please join me and the rest of the Beam PMC in
 welcoming a new committer: Brian Hulette
 >>>
 >>> Brian introduced himself to dev@ earlier this year and
 has been contributing since then. His contributions to Beam include
 explorations of integration with Arrow, standardizing coders, 
 portability
 for schemas, and presentations at Beam events.
 >>>
 >>> In consideration of Brian's contributions, the Beam PMC
 trusts him with the responsibilities of a Beam committer [1].
 >>>
 >>> Thank you, Brian, for your contributions and looking
 forward to many more!
 >>>
 >>> Kenn, on behalf of the Apache Beam PMC
 >>>
 >>> [1]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
 >>
 >>
 >>
 >> --
 >>
 >> This email may be confidential and privileged. If you
 received this communication by mistake, please don't forward it to 
 anyone
 else, please erase all copies and attachments, and please let me 
 know that
 it has gone to the wrong person.
 >>
 >> The above terms reflect a potential business arrangement,
 are provided solely as a basis for further discussion, and are not 
 intended
 to be and do not constitute a legally binding obligation. No 
 legally
 binding obligations will be created, implied, or inferred until an
 agreement in final form is executed in writing by all parties 
 involved.

>>>

>
> --
> 

Re: [Discuss] Beam mascot

[Aizhamal Nurmamat kyzy]

That last sketch of a cuttlefish with a hat is really really good. I vote
for that.

On Fri, Nov 15, 2019 at 10:33 AM Kenneth Knowles  wrote:

> PSA: cuttlefish tentacles look more like their face, not their legs.
> Please find attached illustrations to that effect. And also evidence that
> they are occasionally fancy. Definitely *not* from a respectable designer
> or anyone who could execution a professional logo.
>
> I do think the PMC needs to shepherd this. I would suggest starting with
> an approval vote [1] for animal (no plant ideas?) and then an ASF-style
> vote to record the validated result. From there, we can go through more
> standard design process.
>
> Kenn
>
> [1] https://www.electionscience.org/library/approval-voting/
>
> On Fri, Nov 15, 2019 at 6:54 AM Maximilian Michels  wrote:
>
>> It's great we're having this discussion and we came up with a lot of
>> great ideas through it. However, it is unclear how we proceed from here.
>> Certainly, we can't let designers work with an open-ended discussion on
>> the type of mascot.
>>
>> Personally, I'm fine with _any_ kind of mascot, as long as it is made by
>> a decent designer. The respectable designer I'm talking to was very
>> generous to donate these sketches for free. I don't think that this is
>> to be expected at all. Coming back to the original idea of hiring
>> multiple designers, I don't see how we will pay those designers to all
>> come up with a logo, unless we all donate money.
>>
>> The sketches I've sent might not look like much because there is still a
>> decent process involved in coming up with the final logo which looks
>> good in different sizes, possibly with many iterations. So far I've
>> tried to incorporate as many of the suggestions here, but for the sake
>> of protecting the designer, I think I'll have to stop doing that. It
>> simply won't work.
>>
>> How to proceed from here? I think _any_ professionally executed logo
>> will do, this is really more about the community agreeing for the
>> greater good. Ultimately, I think the PMC might have to decide on how to
>> proceed here.
>>
>> Cheers,
>> Max
>>
>> On 15.11.19 13:59, Hannah Jiang wrote:
>> > I also vote for firefly.
>> >
>> > On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev <
>> valen...@google.com
>> > > wrote:
>> >
>> > I like the firefly sketch a lot, it's my favorite so far.
>> >
>> > On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw
>> > mailto:rober...@google.com>> wrote:
>> >
>> > #37 from the sketches was the cuttlefish, which would put it at
>> > (with
>> > 4 votes) the most popular so far. I do like the firefly too.
>> >
>> > On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas > > > wrote:
>> >  >
>> >  > Hi everyone, so exciting to see this convo taking off!
>> >  >
>> >  > I loved Alex's firefly! -- it can have so many cool
>> > variations, and as a stuffed animal is very original.
>> >  >
>> >  > Other ideas I had are a caterpillar because it looks like a
>> > data pipeline, lol or the beaver!
>> >  >
>> >  > Feedback on the current sketches.
>> >  > - They resemble a lot either the Octocat or Totoro [1]
>> >  > - I'd like to see something that is completely new and
>> > original, pancakes from gRPC is an example[2]
>> >  > - Something more caricaturesque is better, since we can dress
>> > it up and modify it
>> >  >
>> >  > To move forward, it seems that the animals that were winners
>> > in this thread are:
>> >  >
>> >  > Beaver (3)
>> >  > Firefly (3)
>> >  > Lemur or votes on sketches (3)
>> >  > Cuttlefish (2)
>> >  > Hedgehog (1)
>> >  > Salmon (1)
>> >  >
>> >  > So let's focus the design proposals on the three winners:
>> > beaver, firefly and lemur.
>> >  > I'd like to see more options on beavers and fireflies, the
>> > current sketch options I think are based on the cuttlefish and
>> > the lemur (?)
>> >  >
>> >  > I think it's a good idea to get sketches from multiple
>> > designers, since like someone else pointed out, we'll get
>> > variations based on their personal styles, and someone else
>> > mentioned here that we have teams/companies
>> 
>> >  with designers in
>> > their teams, so let's take advantage of that as well :)
>> >  >
>> >  > I'd suggest to fork the conversation into a call for sketched
>> > and we vote on that.
>> >  >
>> >  > [1]
>> >
>> https://www.google.com/search?q=totoro=ACYBGNTFW6vq76cHp05g4vBaR-SVJNI1iw:1573674471669=lnms=isch=X=0ahUKEwiTwICf-uflAhUKHqwKHVtzAykQ_AUIEigB=1440=735
>> >  

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

[Pulasthi Supun Wickramasinghe]

Thanks for the information. I will take a look.

Best Regards,
Pulasthi

On Fri, Nov 15, 2019 at 2:07 PM Luke Cwik  wrote:

> They are serialized but not with Java serialization. There is a
> CloudObject serialization[1] layer that only Dataflow uses while all other
> runners who need to serialize are using the Coder -> Proto serialization
> layer[2]. The CloudObject representation is slated for deletion once we can
> migrate Dataflow to the pure proto representation. Feel free to send a PR
> to improve the wording in the Javadoc.
>
> 1:
> https://github.com/apache/beam/blob/5de27d5c0cb86962e28b5168b7f1dec62352230b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjects.java#L87
> 2:
> https://github.com/apache/beam/blob/5de27d5c0cb86962e28b5168b7f1dec62352230b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L68
>
> On Fri, Nov 15, 2019 at 11:00 AM Pulasthi Supun Wickramasinghe <
> pulasthi...@gmail.com> wrote:
>
>> Hi Luke,
>>
>> Aren't the coders supposed to be serializable? The doc on the Coder
>> interface has the following java doc comment, which seems to mean that they
>> should be, and most of the basic coders seem to serializable.
>>
>> " {@link Coder} instances are serialized during job creation and
>> deserialized before use. This will generally be performed by serializing
>> the object via Java Serialization."
>>
>> However "TimestampedValueCoder" does not have a default constructor so it
>> cannot be deserialized.  adding a private non-arg constructor would solve
>> the problem, but this may not be the only class that has this issue. I can
>> work on a pull request to add the private non-args constructors to coders
>> that have them missing if this was not done intentionally. WDYT?
>>
>> Best Regards,
>> Pulasthi
>>
>> On Fri, Nov 15, 2019 at 12:05 AM Pulasthi Supun Wickramasinghe <
>> pulasthi...@gmail.com> wrote:
>>
>>> Hi Luke,
>>>
>>> That is the approach i am taking currently to handle the functions. I
>>> Might have to do the same for Coders as well since some coders have the
>>> same issue of not having default constructors.
>>>
>>> I also initially considered converting the pipeline into a JSON format
>>> and sending that over to the workers, Will take a look at the option you
>>> have mentioned since we do plan to implement a Portable pipeline runner for
>>> Twister2 as well. Thanks for the information
>>>
>>> Best Regards,
>>> Pulasthi
>>>
>>>
>>> On Thu, Nov 14, 2019 at 2:30 PM Luke Cwik  wrote:
>>>
 You should create placeholders inside of your Twister2/OpenMPI
 implementation that represent these functions and then instantiate actual
 instances of them on the workers if you want to write your own pipeline
 representation and format for OpenMPI/Twister2.

 Or consider converting the pipeline to its proto representation and
 building a portable pipeline runner. This way you could run Go and Python
 pipelines as well. The best example of this is the current Flink
 integration[1]

 1:
 https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java

 On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe <
 pulasthi...@gmail.com> wrote:

> Hi Dev's
>
> Currently, the Pipeline class in Beam is not Serializable. This is not
> a problem for the current runners since the pipeline is translated and
> submitted through a centralized Driver like model. However, if the runner
> has a decentralized model similar to OpenMPI (MPI), which is also the case
> with Twister2, which I am developing a runner currently, it would have 
> been
> better if the pipeline itself was Serializable.
>
> Currently, I am trying to transform the Pipeline into a Twister2 graph
> and then send over to the workers, however since there are some functions
> such as "SystemReduceFn" that are not serializable this also is somewhat
> troublesome.
>
> Was the decision to make Pipelines not Serializable made due to some
> specific reason or because all the current use cases did not present any
> valid requirement to make them Serializable?
>
> Best Regards,
> Pulasthi
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035 <(224)%20386-9035>
>

>>>
>>> --
>>> Pulasthi S. Wickramasinghe
>>> PhD Candidate  | Research Assistant
>>> School of Informatics and Computing | Digital Science Center
>>> Indiana University, Bloomington
>>> cell: 224-386-9035 <(224)%20386-9035>
>>>
>>
>>
>> --
>> Pulasthi S. Wickramasinghe
>> PhD Candidate  | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana 

Re: slides?

[Thomas Weise]

It would be great to have an index for those materials.

Maybe as cwiki page, which is easy to edit and watch. Similar to:

https://cwiki.apache.org/confluence/display/BEAM/Design+Documents

Thomas


On Fri, Nov 15, 2019 at 10:52 AM Kenneth Knowles  wrote:

> We have a section for this:
> https://beam.apache.org/community/presentation-materials/.
>
> Right now "Presentation Materials" has the appearance of carefully curated
> stuff from a core team. That was probably true three years ago, but now it
> is simply out of date. A lot of the material is so old that it is actually
> incorrect. It would be good to invite more people to maintain this.
>
> Perhaps:
>
>  - "Presentation Materials" -> "Presentations"
>  - Replace the Google Drive link with readable HTML page with a (possibly
> large) archive of events and slides, with licenses so new folks can
> copy/paste/modify to kick start their talk
>
> This has the added benefit that there is a clear date and author on each
> piece, so you know how old the material is and can put it in context. And
> link to video of people presenting with the deck, too. That can be better
> than long written speaker notes.
>
> Kenn
>
> On Thu, Nov 14, 2019 at 10:00 PM Austin Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> Hi Dev and User,
>>
>> Wondering if people would find a benefit from collecting slides from
>> Meetups/Talks?
>>
>> Seems that this could be appropriate on the website, for instance.  Not
>> sure whether this has been asked previously, so bringing it to the group.
>>
>> Cheers,
>> Austin
>>
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Alan Myrvold]

Congratulations, Brian!

On Fri, Nov 15, 2019 at 11:11 AM Yifan Zou  wrote:

> Well deserved. Congratulations!
>
> On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov 
> wrote:
>
>> Congratulations, Brian!
>>
>> On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:
>>
>>> Congrats Brian!
>>>
>>>
>>> On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang  wrote:
>>>
 Congrats Brian!

 On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:

> Congrats, Brian!
>
> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
> danolive...@google.com> wrote:
>
>> Congratulations Brian! It's well deserved.
>>
>> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Congratulations, Brian!
>>>
>>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>>
>>> Congrats!
>>>
>>>
>>> -Rui
>>>
>>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>>
 Congratulations!


 On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done Brian!!!
>
> Kenn thank you for sharing
>
> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
> wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
>> wrote:
>>
>>> Congratulations Brian!
>>> Happy to see this happening and eager to see more of your work!
>>>
>>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>>> wrote:
>>> >
>>> > Congrats Brian!
>>> >
>>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
>>> wrote:
>>> >>
>>> >> Congrats Brian!
>>> >>
>>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>>> >>
>>> >> Great news!
>>> >>
>>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>>> wrote:
>>> >>>
>>> >>> Congratulations!
>>> >>>
>>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>> 
>>>  Congratulations, Brian!
>>> 
>>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>>> sunjincheng...@gmail.com> wrote:
>>> >
>>> > Congratulation Brian!
>>> >
>>> > Best,
>>> > Jincheng
>>> >
>>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>>> >>
>>> >> Thanks for your contributions and congrats Brian!
>>> >>
>>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>>> k...@apache.org> wrote:
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> Please join me and the rest of the Beam PMC in welcoming
>>> a new committer: Brian Hulette
>>> >>>
>>> >>> Brian introduced himself to dev@ earlier this year and
>>> has been contributing since then. His contributions to Beam include
>>> explorations of integration with Arrow, standardizing coders, 
>>> portability
>>> for schemas, and presentations at Beam events.
>>> >>>
>>> >>> In consideration of Brian's contributions, the Beam PMC
>>> trusts him with the responsibilities of a Beam committer [1].
>>> >>>
>>> >>> Thank you, Brian, for your contributions and looking
>>> forward to many more!
>>> >>>
>>> >>> Kenn, on behalf of the Apache Beam PMC
>>> >>>
>>> >>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> This email may be confidential and privileged. If you
>>> received this communication by mistake, please don't forward it to 
>>> anyone
>>> else, please erase all copies and attachments, and please let me 
>>> know that
>>> it has gone to the wrong person.
>>> >>
>>> >> The above terms reflect a potential business arrangement, are
>>> provided solely as a basis for further discussion, and are not 
>>> intended to
>>> be and do not constitute a legally binding obligation. No legally 
>>> binding
>>> obligations will be created, implied, or inferred until an 
>>> agreement in
>>> final form is executed in writing by all parties involved.
>>>
>>
>>>

 --
 
 Ruoyun  Huang

Re: [ANNOUNCE] New committer: Brian Hulette

[Yifan Zou]

Well deserved. Congratulations!

On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov 
wrote:

> Congratulations, Brian!
>
> On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:
>
>> Congrats Brian!
>>
>>
>> On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang  wrote:
>>
>>> Congrats Brian!
>>>
>>> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:
>>>
 Congrats, Brian!

 On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
 danolive...@google.com> wrote:

> Congratulations Brian! It's well deserved.
>
> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Congratulations, Brian!
>>
>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>
>>> Congratulations!
>>>
>>>
>>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done Brian!!!

 Kenn thank you for sharing

 On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
 wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
> wrote:
>
>> Congratulations Brian!
>> Happy to see this happening and eager to see more of your work!
>>
>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>> wrote:
>> >
>> > Congrats Brian!
>> >
>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
>> wrote:
>> >>
>> >> Congrats Brian!
>> >>
>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>> >>
>> >> Great news!
>> >>
>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>> wrote:
>> >>>
>> >>> Congratulations!
>> >>>
>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> 
>>  Congratulations, Brian!
>> 
>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>> sunjincheng...@gmail.com> wrote:
>> >
>> > Congratulation Brian!
>> >
>> > Best,
>> > Jincheng
>> >
>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>> >>
>> >> Thanks for your contributions and congrats Brian!
>> >>
>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming
>> a new committer: Brian Hulette
>> >>>
>> >>> Brian introduced himself to dev@ earlier this year and
>> has been contributing since then. His contributions to Beam include
>> explorations of integration with Arrow, standardizing coders, 
>> portability
>> for schemas, and presentations at Beam events.
>> >>>
>> >>> In consideration of Brian's contributions, the Beam PMC
>> trusts him with the responsibilities of a Beam committer [1].
>> >>>
>> >>> Thank you, Brian, for your contributions and looking
>> forward to many more!
>> >>>
>> >>> Kenn, on behalf of the Apache Beam PMC
>> >>>
>> >>> [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> This email may be confidential and privileged. If you received
>> this communication by mistake, please don't forward it to anyone 
>> else,
>> please erase all copies and attachments, and please let me know that 
>> it has
>> gone to the wrong person.
>> >>
>> >> The above terms reflect a potential business arrangement, are
>> provided solely as a basis for further discussion, and are not 
>> intended to
>> be and do not constitute a legally binding obligation. No legally 
>> binding
>> obligations will be created, implied, or inferred until an agreement 
>> in
>> final form is executed in writing by all parties involved.
>>
>
>>
>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

[Reuven Lax]

Serializable classes are not required to have default, no-arg constructors.

Reuven

On Fri, Nov 15, 2019 at 11:00 AM Pulasthi Supun Wickramasinghe <
pulasthi...@gmail.com> wrote:

> Hi Luke,
>
> Aren't the coders supposed to be serializable? The doc on the Coder
> interface has the following java doc comment, which seems to mean that they
> should be, and most of the basic coders seem to serializable.
>
> " {@link Coder} instances are serialized during job creation and
> deserialized before use. This will generally be performed by serializing
> the object via Java Serialization."
>
> However "TimestampedValueCoder" does not have a default constructor so it
> cannot be deserialized.  adding a private non-arg constructor would solve
> the problem, but this may not be the only class that has this issue. I can
> work on a pull request to add the private non-args constructors to coders
> that have them missing if this was not done intentionally. WDYT?
>
> Best Regards,
> Pulasthi
>
> On Fri, Nov 15, 2019 at 12:05 AM Pulasthi Supun Wickramasinghe <
> pulasthi...@gmail.com> wrote:
>
>> Hi Luke,
>>
>> That is the approach i am taking currently to handle the functions. I
>> Might have to do the same for Coders as well since some coders have the
>> same issue of not having default constructors.
>>
>> I also initially considered converting the pipeline into a JSON format
>> and sending that over to the workers, Will take a look at the option you
>> have mentioned since we do plan to implement a Portable pipeline runner for
>> Twister2 as well. Thanks for the information
>>
>> Best Regards,
>> Pulasthi
>>
>>
>> On Thu, Nov 14, 2019 at 2:30 PM Luke Cwik  wrote:
>>
>>> You should create placeholders inside of your Twister2/OpenMPI
>>> implementation that represent these functions and then instantiate actual
>>> instances of them on the workers if you want to write your own pipeline
>>> representation and format for OpenMPI/Twister2.
>>>
>>> Or consider converting the pipeline to its proto representation and
>>> building a portable pipeline runner. This way you could run Go and Python
>>> pipelines as well. The best example of this is the current Flink
>>> integration[1]
>>>
>>> 1:
>>> https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
>>>
>>> On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe <
>>> pulasthi...@gmail.com> wrote:
>>>
 Hi Dev's

 Currently, the Pipeline class in Beam is not Serializable. This is not
 a problem for the current runners since the pipeline is translated and
 submitted through a centralized Driver like model. However, if the runner
 has a decentralized model similar to OpenMPI (MPI), which is also the case
 with Twister2, which I am developing a runner currently, it would have been
 better if the pipeline itself was Serializable.

 Currently, I am trying to transform the Pipeline into a Twister2 graph
 and then send over to the workers, however since there are some functions
 such as "SystemReduceFn" that are not serializable this also is somewhat
 troublesome.

 Was the decision to make Pipelines not Serializable made due to some
 specific reason or because all the current use cases did not present any
 valid requirement to make them Serializable?

 Best Regards,
 Pulasthi
 --
 Pulasthi S. Wickramasinghe
 PhD Candidate  | Research Assistant
 School of Informatics and Computing | Digital Science Center
 Indiana University, Bloomington
 cell: 224-386-9035 <(224)%20386-9035>

>>>
>>
>> --
>> Pulasthi S. Wickramasinghe
>> PhD Candidate  | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana University, Bloomington
>> cell: 224-386-9035 <(224)%20386-9035>
>>
>
>
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035 <(224)%20386-9035>
>

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

[Luke Cwik]

They are serialized but not with Java serialization. There is a CloudObject
serialization[1] layer that only Dataflow uses while all other runners who
need to serialize are using the Coder -> Proto serialization layer[2]. The
CloudObject representation is slated for deletion once we can migrate
Dataflow to the pure proto representation. Feel free to send a PR to
improve the wording in the Javadoc.

1:
https://github.com/apache/beam/blob/5de27d5c0cb86962e28b5168b7f1dec62352230b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjects.java#L87
2:
https://github.com/apache/beam/blob/5de27d5c0cb86962e28b5168b7f1dec62352230b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L68

On Fri, Nov 15, 2019 at 11:00 AM Pulasthi Supun Wickramasinghe <
pulasthi...@gmail.com> wrote:

> Hi Luke,
>
> Aren't the coders supposed to be serializable? The doc on the Coder
> interface has the following java doc comment, which seems to mean that they
> should be, and most of the basic coders seem to serializable.
>
> " {@link Coder} instances are serialized during job creation and
> deserialized before use. This will generally be performed by serializing
> the object via Java Serialization."
>
> However "TimestampedValueCoder" does not have a default constructor so it
> cannot be deserialized.  adding a private non-arg constructor would solve
> the problem, but this may not be the only class that has this issue. I can
> work on a pull request to add the private non-args constructors to coders
> that have them missing if this was not done intentionally. WDYT?
>
> Best Regards,
> Pulasthi
>
> On Fri, Nov 15, 2019 at 12:05 AM Pulasthi Supun Wickramasinghe <
> pulasthi...@gmail.com> wrote:
>
>> Hi Luke,
>>
>> That is the approach i am taking currently to handle the functions. I
>> Might have to do the same for Coders as well since some coders have the
>> same issue of not having default constructors.
>>
>> I also initially considered converting the pipeline into a JSON format
>> and sending that over to the workers, Will take a look at the option you
>> have mentioned since we do plan to implement a Portable pipeline runner for
>> Twister2 as well. Thanks for the information
>>
>> Best Regards,
>> Pulasthi
>>
>>
>> On Thu, Nov 14, 2019 at 2:30 PM Luke Cwik  wrote:
>>
>>> You should create placeholders inside of your Twister2/OpenMPI
>>> implementation that represent these functions and then instantiate actual
>>> instances of them on the workers if you want to write your own pipeline
>>> representation and format for OpenMPI/Twister2.
>>>
>>> Or consider converting the pipeline to its proto representation and
>>> building a portable pipeline runner. This way you could run Go and Python
>>> pipelines as well. The best example of this is the current Flink
>>> integration[1]
>>>
>>> 1:
>>> https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
>>>
>>> On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe <
>>> pulasthi...@gmail.com> wrote:
>>>
 Hi Dev's

 Currently, the Pipeline class in Beam is not Serializable. This is not
 a problem for the current runners since the pipeline is translated and
 submitted through a centralized Driver like model. However, if the runner
 has a decentralized model similar to OpenMPI (MPI), which is also the case
 with Twister2, which I am developing a runner currently, it would have been
 better if the pipeline itself was Serializable.

 Currently, I am trying to transform the Pipeline into a Twister2 graph
 and then send over to the workers, however since there are some functions
 such as "SystemReduceFn" that are not serializable this also is somewhat
 troublesome.

 Was the decision to make Pipelines not Serializable made due to some
 specific reason or because all the current use cases did not present any
 valid requirement to make them Serializable?

 Best Regards,
 Pulasthi
 --
 Pulasthi S. Wickramasinghe
 PhD Candidate  | Research Assistant
 School of Informatics and Computing | Digital Science Center
 Indiana University, Bloomington
 cell: 224-386-9035 <(224)%20386-9035>

>>>
>>
>> --
>> Pulasthi S. Wickramasinghe
>> PhD Candidate  | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana University, Bloomington
>> cell: 224-386-9035 <(224)%20386-9035>
>>
>
>
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035 <(224)%20386-9035>
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Kirill Kozlov]

Congratulations, Brian!

On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:

> Congrats Brian!
>
>
> On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang  wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:
>>
>>> Congrats, Brian!
>>>
>>> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira 
>>> wrote:
>>>
 Congratulations Brian! It's well deserved.

 On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
 aromanenko@gmail.com> wrote:

> Congratulations, Brian!
>
> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>
> Congrats!
>
>
> -Rui
>
> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>
>> Congratulations!
>>
>>
>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done Brian!!!
>>>
>>> Kenn thank you for sharing
>>>
>>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
>>> wrote:
>>>
 Congrats Brian!

 On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
 wrote:

> Congratulations Brian!
> Happy to see this happening and eager to see more of your work!
>
> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
> wrote:
> >
> > Congrats Brian!
> >
> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
> wrote:
> >>
> >> Congrats Brian!
> >>
> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
> >>
> >> Great news!
> >>
> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
> wrote:
> >>>
> >>> Congratulations!
> >>>
> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> 
>  Congratulations, Brian!
> 
>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
> sunjincheng...@gmail.com> wrote:
> >
> > Congratulation Brian!
> >
> > Best,
> > Jincheng
> >
> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
> >>
> >> Thanks for your contributions and congrats Brian!
> >>
> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
> k...@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please join me and the rest of the Beam PMC in welcoming a
> new committer: Brian Hulette
> >>>
> >>> Brian introduced himself to dev@ earlier this year and
> has been contributing since then. His contributions to Beam include
> explorations of integration with Arrow, standardizing coders, 
> portability
> for schemas, and presentations at Beam events.
> >>>
> >>> In consideration of Brian's contributions, the Beam PMC
> trusts him with the responsibilities of a Beam committer [1].
> >>>
> >>> Thank you, Brian, for your contributions and looking
> forward to many more!
> >>>
> >>> Kenn, on behalf of the Apache Beam PMC
> >>>
> >>> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
> >>
> >>
> >>
> >> --
> >>
> >> This email may be confidential and privileged. If you received
> this communication by mistake, please don't forward it to anyone else,
> please erase all copies and attachments, and please let me know that 
> it has
> gone to the wrong person.
> >>
> >> The above terms reflect a potential business arrangement, are
> provided solely as a basis for further discussion, and are not 
> intended to
> be and do not constitute a legally binding obligation. No legally 
> binding
> obligations will be created, implied, or inferred until an agreement 
> in
> final form is executed in writing by all parties involved.
>

>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

[Pulasthi Supun Wickramasinghe]

Hi Luke,

Aren't the coders supposed to be serializable? The doc on the Coder
interface has the following java doc comment, which seems to mean that they
should be, and most of the basic coders seem to serializable.

" {@link Coder} instances are serialized during job creation and
deserialized before use. This will generally be performed by serializing
the object via Java Serialization."

However "TimestampedValueCoder" does not have a default constructor so it
cannot be deserialized.  adding a private non-arg constructor would solve
the problem, but this may not be the only class that has this issue. I can
work on a pull request to add the private non-args constructors to coders
that have them missing if this was not done intentionally. WDYT?

Best Regards,
Pulasthi

On Fri, Nov 15, 2019 at 12:05 AM Pulasthi Supun Wickramasinghe <
pulasthi...@gmail.com> wrote:

> Hi Luke,
>
> That is the approach i am taking currently to handle the functions. I
> Might have to do the same for Coders as well since some coders have the
> same issue of not having default constructors.
>
> I also initially considered converting the pipeline into a JSON format and
> sending that over to the workers, Will take a look at the option you have
> mentioned since we do plan to implement a Portable pipeline runner for
> Twister2 as well. Thanks for the information
>
> Best Regards,
> Pulasthi
>
>
> On Thu, Nov 14, 2019 at 2:30 PM Luke Cwik  wrote:
>
>> You should create placeholders inside of your Twister2/OpenMPI
>> implementation that represent these functions and then instantiate actual
>> instances of them on the workers if you want to write your own pipeline
>> representation and format for OpenMPI/Twister2.
>>
>> Or consider converting the pipeline to its proto representation and
>> building a portable pipeline runner. This way you could run Go and Python
>> pipelines as well. The best example of this is the current Flink
>> integration[1]
>>
>> 1:
>> https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
>>
>> On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe <
>> pulasthi...@gmail.com> wrote:
>>
>>> Hi Dev's
>>>
>>> Currently, the Pipeline class in Beam is not Serializable. This is not a
>>> problem for the current runners since the pipeline is translated and
>>> submitted through a centralized Driver like model. However, if the runner
>>> has a decentralized model similar to OpenMPI (MPI), which is also the case
>>> with Twister2, which I am developing a runner currently, it would have been
>>> better if the pipeline itself was Serializable.
>>>
>>> Currently, I am trying to transform the Pipeline into a Twister2 graph
>>> and then send over to the workers, however since there are some functions
>>> such as "SystemReduceFn" that are not serializable this also is somewhat
>>> troublesome.
>>>
>>> Was the decision to make Pipelines not Serializable made due to some
>>> specific reason or because all the current use cases did not present any
>>> valid requirement to make them Serializable?
>>>
>>> Best Regards,
>>> Pulasthi
>>> --
>>> Pulasthi S. Wickramasinghe
>>> PhD Candidate  | Research Assistant
>>> School of Informatics and Computing | Digital Science Center
>>> Indiana University, Bloomington
>>> cell: 224-386-9035 <(224)%20386-9035>
>>>
>>
>
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035
>


-- 
Pulasthi S. Wickramasinghe
PhD Candidate  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035

Re: How to unsubscribe the Apache projects and jira issues notification

[Luke Cwik]

https://apache.org/foundation/mailinglists.html#request-addresses-for-unsubscribing

If you want to subscribe to l...@apache.org then you need to send a message
to

list-subscr...@apache.org

To get off a list, send a message to

list-unsubscr...@apache.org

On Fri, Nov 15, 2019 at 2:40 AM P. Ramanjaneya Reddy 
wrote:

> Hi
>
> Following blogs want to unsubscribe kindly guide.
>
> I tried from google..still mails receiving
>
> Also should unubscribe..
>
> j...@apache.org
>
>
> u...@flink.apache.org
>
> d...@flink.apache.org
>
> dev@beam.apache.org
>
> u...@beamho.apache.org 
>
>
> Thanks
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Udi Meiri]

Congrats Brian!


On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang  wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:
>
>> Congrats, Brian!
>>
>> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira 
>> wrote:
>>
>>> Congratulations Brian! It's well deserved.
>>>
>>> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko 
>>> wrote:
>>>
 Congratulations, Brian!

 On 15 Nov 2019, at 18:27, Rui Wang  wrote:

 Congrats!


 -Rui

 On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:

> Congratulations!
>
>
> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
> conne...@google.com> wrote:
>
>> Well done Brian!!!
>>
>> Kenn thank you for sharing
>>
>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
>> wrote:
>>
>>> Congrats Brian!
>>>
>>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
>>> wrote:
>>>
 Congratulations Brian!
 Happy to see this happening and eager to see more of your work!

 On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
 wrote:
 >
 > Congrats Brian!
 >
 > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
 wrote:
 >>
 >> Congrats Brian!
 >>
 >> On 11/15/19 9:58 AM, Reza Rokni wrote:
 >>
 >> Great news!
 >>
 >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
 wrote:
 >>>
 >>> Congratulations!
 >>>
 >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 
  Congratulations, Brian!
 
  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
 sunjincheng...@gmail.com> wrote:
 >
 > Congratulation Brian!
 >
 > Best,
 > Jincheng
 >
 > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
 >>
 >> Thanks for your contributions and congrats Brian!
 >>
 >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
 k...@apache.org> wrote:
 >>>
 >>> Hi all,
 >>>
 >>> Please join me and the rest of the Beam PMC in welcoming a
 new committer: Brian Hulette
 >>>
 >>> Brian introduced himself to dev@ earlier this year and has
 been contributing since then. His contributions to Beam include
 explorations of integration with Arrow, standardizing coders, 
 portability
 for schemas, and presentations at Beam events.
 >>>
 >>> In consideration of Brian's contributions, the Beam PMC
 trusts him with the responsibilities of a Beam committer [1].
 >>>
 >>> Thank you, Brian, for your contributions and looking
 forward to many more!
 >>>
 >>> Kenn, on behalf of the Apache Beam PMC
 >>>
 >>> [1]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
 >>
 >>
 >>
 >> --
 >>
 >> This email may be confidential and privileged. If you received
 this communication by mistake, please don't forward it to anyone else,
 please erase all copies and attachments, and please let me know that 
 it has
 gone to the wrong person.
 >>
 >> The above terms reflect a potential business arrangement, are
 provided solely as a basis for further discussion, and are not 
 intended to
 be and do not constitute a legally binding obligation. No legally 
 binding
 obligations will be created, implied, or inferred until an agreement in
 final form is executed in writing by all parties involved.

>>>

>
> --
> 
> Ruoyun  Huang
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: slides?

[Kenneth Knowles]

We have a section for this:
https://beam.apache.org/community/presentation-materials/.

Right now "Presentation Materials" has the appearance of carefully curated
stuff from a core team. That was probably true three years ago, but now it
is simply out of date. A lot of the material is so old that it is actually
incorrect. It would be good to invite more people to maintain this.

Perhaps:

 - "Presentation Materials" -> "Presentations"
 - Replace the Google Drive link with readable HTML page with a (possibly
large) archive of events and slides, with licenses so new folks can
copy/paste/modify to kick start their talk

This has the added benefit that there is a clear date and author on each
piece, so you know how old the material is and can put it in context. And
link to video of people presenting with the deck, too. That can be better
than long written speaker notes.

Kenn

On Thu, Nov 14, 2019 at 10:00 PM Austin Bennett 
wrote:

> Hi Dev and User,
>
> Wondering if people would find a benefit from collecting slides from
> Meetups/Talks?
>
> Seems that this could be appropriate on the website, for instance.  Not
> sure whether this has been asked previously, so bringing it to the group.
>
> Cheers,
> Austin
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Ruoyun Huang]

Congrats Brian!

On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:

> Congrats, Brian!
>
> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira 
> wrote:
>
>> Congratulations Brian! It's well deserved.
>>
>> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko 
>> wrote:
>>
>>> Congratulations, Brian!
>>>
>>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>>
>>> Congrats!
>>>
>>>
>>> -Rui
>>>
>>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>>
 Congratulations!


 On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done Brian!!!
>
> Kenn thank you for sharing
>
> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
>> wrote:
>>
>>> Congratulations Brian!
>>> Happy to see this happening and eager to see more of your work!
>>>
>>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>>> wrote:
>>> >
>>> > Congrats Brian!
>>> >
>>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
>>> wrote:
>>> >>
>>> >> Congrats Brian!
>>> >>
>>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>>> >>
>>> >> Great news!
>>> >>
>>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>>> wrote:
>>> >>>
>>> >>> Congratulations!
>>> >>>
>>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>> 
>>>  Congratulations, Brian!
>>> 
>>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>>> sunjincheng...@gmail.com> wrote:
>>> >
>>> > Congratulation Brian!
>>> >
>>> > Best,
>>> > Jincheng
>>> >
>>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>>> >>
>>> >> Thanks for your contributions and congrats Brian!
>>> >>
>>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>>> k...@apache.org> wrote:
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Brian Hulette
>>> >>>
>>> >>> Brian introduced himself to dev@ earlier this year and has
>>> been contributing since then. His contributions to Beam include
>>> explorations of integration with Arrow, standardizing coders, 
>>> portability
>>> for schemas, and presentations at Beam events.
>>> >>>
>>> >>> In consideration of Brian's contributions, the Beam PMC
>>> trusts him with the responsibilities of a Beam committer [1].
>>> >>>
>>> >>> Thank you, Brian, for your contributions and looking forward
>>> to many more!
>>> >>>
>>> >>> Kenn, on behalf of the Apache Beam PMC
>>> >>>
>>> >>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> This email may be confidential and privileged. If you received
>>> this communication by mistake, please don't forward it to anyone else,
>>> please erase all copies and attachments, and please let me know that it 
>>> has
>>> gone to the wrong person.
>>> >>
>>> >> The above terms reflect a potential business arrangement, are
>>> provided solely as a basis for further discussion, and are not intended 
>>> to
>>> be and do not constitute a legally binding obligation. No legally 
>>> binding
>>> obligations will be created, implied, or inferred until an agreement in
>>> final form is executed in writing by all parties involved.
>>>
>>
>>>

-- 

Ruoyun  Huang

Re: [UPDATE] Preparing for Beam 2.17.0 release

[Thomas Weise]

Any update regarding the release?

The list still shows 10 open issues:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%202.17.0%20and%20resolution%20is%20EMPTY

Is the RC blocked on those?






On Mon, Oct 28, 2019 at 12:46 PM Ahmet Altay  wrote:

>
>
> On Mon, Oct 28, 2019 at 12:44 PM Gleb Kanterov  wrote:
>
>> It looks like BigQueryIO DIRECT_READ is broken since 2.16.0, I've added a
>> ticket describing the problem and possible fix, see BEAM-8504
>>  [1].
>>
>
> Should this be added to 2.16 blog post as a known issue?
>
>
>>
>> [1]: https://issues.apache.org/jira/browse/BEAM-8504
>>
>> On Wed, Oct 23, 2019 at 9:19 PM Kenneth Knowles  wrote:
>>
>>> I opened https://github.com/apache/beam/pull/9862 to raise the
>>> documentation of Fix Version to the top level. It also includes the write
>>> up of Jira priorities, to make clear that "Blocker" priority does not refer
>>> to release blocking.
>>>
>>> On Wed, Oct 23, 2019 at 11:16 AM Kenneth Knowles 
>>> wrote:
>>>
 I've gone over the tickets and removed Fix Version from many of them
 that do not seem to be critical defects. If I removed Fix Version from a
 ticket you care about, please feel free to add it back. I am not trying to
 decide what is in/out of the release, just trying to triage the Jira data
 to match expected practices.

 It should probably be documented somewhere outside of the release
 guide. As far as I can tell, the fact that we triage them down to zero is
 the only place we mention that it is used to indicate release blockers and
 not used for feature targets.

 Kenn

 On Wed, Oct 23, 2019 at 10:40 AM Kenneth Knowles 
 wrote:

>  Wow, 28 release blocking tickets! That is the most I've ever seen, by
> far. Many appear to be feature requests, not release-blocking defects. I
> believe this is not according to our normal best practice. The release
> cadence should not wait for features in progress, with exceptions 
> discussed
> on dev@. As a matter of best practice, I think we should triage
> feature requests to not have Fix Version set until it has been discussed 
> on
> dev@.
>
> Kenn
>
> On Wed, Oct 23, 2019 at 9:55 AM Mikhail Gryzykhin 
> wrote:
>
>> Hi all,
>>
>> Beam 2.17 release branch cut is scheduled today (2019/10/23)
>> according to the release calendar [1].  I'll start working on the
>> branch cutoff and later work on cherry picking blocker fixes.
>>
>> If you have release blocking issues for 2.17 please mark their "Fix
>> Version" as 2.17.0 [2]. This tag is already created in JIRA in case you
>> would like to move any non-blocking issues to that version.
>>
>> There is a decent amount of open bugs to be resolved in 2.17.0 [2]
>> and only 4 [3] are marked as blockers. Please, review those if these bugs
>> are actually to be resolved in 2.17.0 and prioritize fixes if possible.
>>
>> Any thoughts, comments, objections?
>>
>> Regards.
>> Mikhail.
>>
>>
>> [1]
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>> [2]
>> https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.17.0
>> [3]
>> https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%202.17.0
>>
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Robin Qiu]

Congrats, Brian!

On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira 
wrote:

> Congratulations Brian! It's well deserved.
>
> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko 
> wrote:
>
>> Congratulations, Brian!
>>
>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>
>>> Congratulations!
>>>
>>>
>>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan 
>>> wrote:
>>>
 Well done Brian!!!

 Kenn thank you for sharing

 On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
> wrote:
>
>> Congratulations Brian!
>> Happy to see this happening and eager to see more of your work!
>>
>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>> wrote:
>> >
>> > Congrats Brian!
>> >
>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
>> >>
>> >> Congrats Brian!
>> >>
>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>> >>
>> >> Great news!
>> >>
>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>> wrote:
>> >>>
>> >>> Congratulations!
>> >>>
>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> 
>>  Congratulations, Brian!
>> 
>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>> sunjincheng...@gmail.com> wrote:
>> >
>> > Congratulation Brian!
>> >
>> > Best,
>> > Jincheng
>> >
>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>> >>
>> >> Thanks for your contributions and congrats Brian!
>> >>
>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Brian Hulette
>> >>>
>> >>> Brian introduced himself to dev@ earlier this year and has
>> been contributing since then. His contributions to Beam include
>> explorations of integration with Arrow, standardizing coders, portability
>> for schemas, and presentations at Beam events.
>> >>>
>> >>> In consideration of Brian's contributions, the Beam PMC
>> trusts him with the responsibilities of a Beam committer [1].
>> >>>
>> >>> Thank you, Brian, for your contributions and looking forward
>> to many more!
>> >>>
>> >>> Kenn, on behalf of the Apache Beam PMC
>> >>>
>> >>> [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> This email may be confidential and privileged. If you received
>> this communication by mistake, please don't forward it to anyone else,
>> please erase all copies and attachments, and please let me know that it 
>> has
>> gone to the wrong person.
>> >>
>> >> The above terms reflect a potential business arrangement, are
>> provided solely as a basis for further discussion, and are not intended 
>> to
>> be and do not constitute a legally binding obligation. No legally binding
>> obligations will be created, implied, or inferred until an agreement in
>> final form is executed in writing by all parties involved.
>>
>
>>

Re: Library to Parse Thrift Files for ThriftIO

[Reuven Lax]

At a quick glance, the license is Apache which is fine (though we'd have to
check dependencies as well). I do notice that git repro is no longer
maintained; is there a different one that is better supported?

Reuven

On Wed, Nov 13, 2019 at 7:04 AM Christopher Larsen <
christopher.lar...@quantiphi.com> wrote:

> Hey everyone,
>
> In regards to the library that will be used to parse the .thrift files for
> ThriftIO we are thinking about using some of the code from the library
> found here  and updating it to
> use Beam friendly packages. We would love to get the community's feedback
> on this approach.
>
> Best,
> Chris
>
> *This message contains information that may be privileged or confidential
> and is the property of the Quantiphi Inc and/or its affiliates**. It is
> intended only for the person to whom it is addressed. **If you are not
> the intended recipient, any review, dissemination, distribution, copying,
> storage or other use of all or any portion of this message is strictly
> prohibited. If you received this message in error, please immediately
> notify the sender by reply e-mail and delete this message in its *
> *entirety*
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Daniel Oliveira]

Congratulations Brian! It's well deserved.

On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko 
wrote:

> Congratulations, Brian!
>
> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>
> Congrats!
>
>
> -Rui
>
> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>
>> Congratulations!
>>
>>
>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan 
>> wrote:
>>
>>> Well done Brian!!!
>>>
>>> Kenn thank you for sharing
>>>
>>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:
>>>
 Congrats Brian!

 On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía  wrote:

> Congratulations Brian!
> Happy to see this happening and eager to see more of your work!
>
> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
> wrote:
> >
> > Congrats Brian!
> >
> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
> >>
> >> Congrats Brian!
> >>
> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
> >>
> >> Great news!
> >>
> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
> wrote:
> >>>
> >>> Congratulations!
> >>>
> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> 
>  Congratulations, Brian!
> 
>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
> sunjincheng...@gmail.com> wrote:
> >
> > Congratulation Brian!
> >
> > Best,
> > Jincheng
> >
> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
> >>
> >> Thanks for your contributions and congrats Brian!
> >>
> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
> k...@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Brian Hulette
> >>>
> >>> Brian introduced himself to dev@ earlier this year and has
> been contributing since then. His contributions to Beam include
> explorations of integration with Arrow, standardizing coders, portability
> for schemas, and presentations at Beam events.
> >>>
> >>> In consideration of Brian's contributions, the Beam PMC trusts
> him with the responsibilities of a Beam committer [1].
> >>>
> >>> Thank you, Brian, for your contributions and looking forward
> to many more!
> >>>
> >>> Kenn, on behalf of the Apache Beam PMC
> >>>
> >>> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
> >>
> >>
> >>
> >> --
> >>
> >> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
> >>
> >> The above terms reflect a potential business arrangement, are
> provided solely as a basis for further discussion, and are not intended to
> be and do not constitute a legally binding obligation. No legally binding
> obligations will be created, implied, or inferred until an agreement in
> final form is executed in writing by all parties involved.
>

>

Re: [ANNOUNCE] New committer: Brian Hulette

[Alexey Romanenko]

Congratulations, Brian!

> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
> 
> Congrats!
> 
> 
> -Rui
> 
> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  > wrote:
> Congratulations!
> 
> 
> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan  > wrote:
> Well done Brian!!! 
> 
> Kenn thank you for sharing 
> 
> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  > wrote:
> Congrats Brian!
> 
> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía  > wrote:
> Congratulations Brian!
> Happy to see this happening and eager to see more of your work!
> 
> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka  > wrote:
> >
> > Congrats Brian!
> >
> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  > > wrote:
> >>
> >> Congrats Brian!
> >>
> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
> >>
> >> Great news!
> >>
> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  >> > wrote:
> >>>
> >>> Congratulations!
> >>>
> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev  >>> > wrote:
> 
>  Congratulations, Brian!
> 
>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun   > wrote:
> >
> > Congratulation Brian!
> >
> > Best,
> > Jincheng
> >
> > Kyle Weaver mailto:kcwea...@google.com>> 
> > 于2019年11月15日周五 上午7:19写道：
> >>
> >> Thanks for your contributions and congrats Brian!
> >>
> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles  >> > wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please join me and the rest of the Beam PMC in welcoming a new 
> >>> committer: Brian Hulette
> >>>
> >>> Brian introduced himself to dev@ earlier this year and has been 
> >>> contributing since then. His contributions to Beam include 
> >>> explorations of integration with Arrow, standardizing coders, 
> >>> portability for schemas, and presentations at Beam events.
> >>>
> >>> In consideration of Brian's contributions, the Beam PMC trusts him 
> >>> with the responsibilities of a Beam committer [1].
> >>>
> >>> Thank you, Brian, for your contributions and looking forward to many 
> >>> more!
> >>>
> >>> Kenn, on behalf of the Apache Beam PMC
> >>>
> >>> [1] 
> >>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
> >>>  
> >>> 
> >>
> >>
> >>
> >> --
> >>
> >> This email may be confidential and privileged. If you received this 
> >> communication by mistake, please don't forward it to anyone else, please 
> >> erase all copies and attachments, and please let me know that it has gone 
> >> to the wrong person.
> >>
> >> The above terms reflect a potential business arrangement, are provided 
> >> solely as a basis for further discussion, and are not intended to be and 
> >> do not constitute a legally binding obligation. No legally binding 
> >> obligations will be created, implied, or inferred until an agreement in 
> >> final form is executed in writing by all parties involved.

Re: [ANNOUNCE] New committer: Brian Hulette

[Rui Wang]

Congrats!


-Rui

On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:

> Congratulations!
>
>
> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan 
> wrote:
>
>> Well done Brian!!!
>>
>> Kenn thank you for sharing
>>
>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:
>>
>>> Congrats Brian!
>>>
>>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía  wrote:
>>>
 Congratulations Brian!
 Happy to see this happening and eager to see more of your work!

 On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
 wrote:
 >
 > Congrats Brian!
 >
 > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
 >>
 >> Congrats Brian!
 >>
 >> On 11/15/19 9:58 AM, Reza Rokni wrote:
 >>
 >> Great news!
 >>
 >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
 wrote:
 >>>
 >>> Congratulations!
 >>>
 >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 
  Congratulations, Brian!
 
  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
 sunjincheng...@gmail.com> wrote:
 >
 > Congratulation Brian!
 >
 > Best,
 > Jincheng
 >
 > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
 >>
 >> Thanks for your contributions and congrats Brian!
 >>
 >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles 
 wrote:
 >>>
 >>> Hi all,
 >>>
 >>> Please join me and the rest of the Beam PMC in welcoming a new
 committer: Brian Hulette
 >>>
 >>> Brian introduced himself to dev@ earlier this year and has
 been contributing since then. His contributions to Beam include
 explorations of integration with Arrow, standardizing coders, portability
 for schemas, and presentations at Beam events.
 >>>
 >>> In consideration of Brian's contributions, the Beam PMC trusts
 him with the responsibilities of a Beam committer [1].
 >>>
 >>> Thank you, Brian, for your contributions and looking forward to
 many more!
 >>>
 >>> Kenn, on behalf of the Apache Beam PMC
 >>>
 >>> [1]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
 >>
 >>
 >>
 >> --
 >>
 >> This email may be confidential and privileged. If you received this
 communication by mistake, please don't forward it to anyone else, please
 erase all copies and attachments, and please let me know that it has gone
 to the wrong person.
 >>
 >> The above terms reflect a potential business arrangement, are
 provided solely as a basis for further discussion, and are not intended to
 be and do not constitute a legally binding obligation. No legally binding
 obligations will be created, implied, or inferred until an agreement in
 final form is executed in writing by all parties involved.

>>>

Re: [ANNOUNCE] New committer: Brian Hulette

[Thomas Weise]

Congratulations!


On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan 
wrote:

> Well done Brian!!!
>
> Kenn thank you for sharing
>
> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía  wrote:
>>
>>> Congratulations Brian!
>>> Happy to see this happening and eager to see more of your work!
>>>
>>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka  wrote:
>>> >
>>> > Congrats Brian!
>>> >
>>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
>>> >>
>>> >> Congrats Brian!
>>> >>
>>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>>> >>
>>> >> Great news!
>>> >>
>>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  wrote:
>>> >>>
>>> >>> Congratulations!
>>> >>>
>>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>> 
>>>  Congratulations, Brian!
>>> 
>>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>>> sunjincheng...@gmail.com> wrote:
>>> >
>>> > Congratulation Brian!
>>> >
>>> > Best,
>>> > Jincheng
>>> >
>>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>>> >>
>>> >> Thanks for your contributions and congrats Brian!
>>> >>
>>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles 
>>> wrote:
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Brian Hulette
>>> >>>
>>> >>> Brian introduced himself to dev@ earlier this year and has been
>>> contributing since then. His contributions to Beam include explorations of
>>> integration with Arrow, standardizing coders, portability for schemas, and
>>> presentations at Beam events.
>>> >>>
>>> >>> In consideration of Brian's contributions, the Beam PMC trusts
>>> him with the responsibilities of a Beam committer [1].
>>> >>>
>>> >>> Thank you, Brian, for your contributions and looking forward to
>>> many more!
>>> >>>
>>> >>> Kenn, on behalf of the Apache Beam PMC
>>> >>>
>>> >>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> This email may be confidential and privileged. If you received this
>>> communication by mistake, please don't forward it to anyone else, please
>>> erase all copies and attachments, and please let me know that it has gone
>>> to the wrong person.
>>> >>
>>> >> The above terms reflect a potential business arrangement, are
>>> provided solely as a basis for further discussion, and are not intended to
>>> be and do not constitute a legally binding obligation. No legally binding
>>> obligations will be created, implied, or inferred until an agreement in
>>> final form is executed in writing by all parties involved.
>>>
>>

Re: [Discuss] Beam mascot

[Maximilian Michels]

It's great we're having this discussion and we came up with a lot of 
great ideas through it. However, it is unclear how we proceed from here. 
Certainly, we can't let designers work with an open-ended discussion on 
the type of mascot.


Personally, I'm fine with _any_ kind of mascot, as long as it is made by 
a decent designer. The respectable designer I'm talking to was very 
generous to donate these sketches for free. I don't think that this is 
to be expected at all. Coming back to the original idea of hiring 
multiple designers, I don't see how we will pay those designers to all 
come up with a logo, unless we all donate money.


The sketches I've sent might not look like much because there is still a 
decent process involved in coming up with the final logo which looks 
good in different sizes, possibly with many iterations. So far I've 
tried to incorporate as many of the suggestions here, but for the sake 
of protecting the designer, I think I'll have to stop doing that. It 
simply won't work.


How to proceed from here? I think _any_ professionally executed logo 
will do, this is really more about the community agreeing for the 
greater good. Ultimately, I think the PMC might have to decide on how to 
proceed here.


Cheers,
Max

On 15.11.19 13:59, Hannah Jiang wrote:

I also vote for firefly.

On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev > wrote:


I like the firefly sketch a lot, it's my favorite so far.

On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw
mailto:rober...@google.com>> wrote:

#37 from the sketches was the cuttlefish, which would put it at
(with
4 votes) the most popular so far. I do like the firefly too.

On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas mailto:g...@apache.org>> wrote:
 >
 > Hi everyone, so exciting to see this convo taking off!
 >
 > I loved Alex's firefly! -- it can have so many cool
variations, and as a stuffed animal is very original.
 >
 > Other ideas I had are a caterpillar because it looks like a
data pipeline, lol or the beaver!
 >
 > Feedback on the current sketches.
 > - They resemble a lot either the Octocat or Totoro [1]
 > - I'd like to see something that is completely new and
original, pancakes from gRPC is an example[2]
 > - Something more caricaturesque is better, since we can dress
it up and modify it
 >
 > To move forward, it seems that the animals that were winners
in this thread are:
 >
 > Beaver (3)
 > Firefly (3)
 > Lemur or votes on sketches (3)
 > Cuttlefish (2)
 > Hedgehog (1)
 > Salmon (1)
 >
 > So let's focus the design proposals on the three winners:
beaver, firefly and lemur.
 > I'd like to see more options on beavers and fireflies, the
current sketch options I think are based on the cuttlefish and
the lemur (?)
 >
 > I think it's a good idea to get sketches from multiple
designers, since like someone else pointed out, we'll get
variations based on their personal styles, and someone else
mentioned here that we have teams/companies
 with designers in
their teams, so let's take advantage of that as well :)
 >
 > I'd suggest to fork the conversation into a call for sketched
and we vote on that.
 >
 > [1]

https://www.google.com/search?q=totoro=ACYBGNTFW6vq76cHp05g4vBaR-SVJNI1iw:1573674471669=lnms=isch=X=0ahUKEwiTwICf-uflAhUKHqwKHVtzAykQ_AUIEigB=1440=735
 > [2]

https://www.google.com/search?q=pancakes+grpc=isch=2ahUKEwixgqjV-uflAhUBOawKHX2pAfsQ2-cCegQIABAA=pancakes+grpc_l=img.3...13774.22674..22826...10.0..0.112.1818.13j6..01..gws-wiz-img.10..35i39j0j0i67j35i362i39j0i10j0i5i30j0i8i30.hNtaBfSYNv8=WV7MXfHxIYHysAX90obYDw=735=1440

Re: [ANNOUNCE] New committer: Brian Hulette

[Connell O'Callaghan]

Well done Brian!!!

Kenn thank you for sharing

On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden  wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía  wrote:
>
>> Congratulations Brian!
>> Happy to see this happening and eager to see more of your work!
>>
>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka  wrote:
>> >
>> > Congrats Brian!
>> >
>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
>> >>
>> >> Congrats Brian!
>> >>
>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>> >>
>> >> Great news!
>> >>
>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  wrote:
>> >>>
>> >>> Congratulations!
>> >>>
>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> 
>>  Congratulations, Brian!
>> 
>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>> sunjincheng...@gmail.com> wrote:
>> >
>> > Congratulation Brian!
>> >
>> > Best,
>> > Jincheng
>> >
>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>> >>
>> >> Thanks for your contributions and congrats Brian!
>> >>
>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles 
>> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Brian Hulette
>> >>>
>> >>> Brian introduced himself to dev@ earlier this year and has been
>> contributing since then. His contributions to Beam include explorations of
>> integration with Arrow, standardizing coders, portability for schemas, and
>> presentations at Beam events.
>> >>>
>> >>> In consideration of Brian's contributions, the Beam PMC trusts
>> him with the responsibilities of a Beam committer [1].
>> >>>
>> >>> Thank you, Brian, for your contributions and looking forward to
>> many more!
>> >>>
>> >>> Kenn, on behalf of the Apache Beam PMC
>> >>>
>> >>> [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> This email may be confidential and privileged. If you received this
>> communication by mistake, please don't forward it to anyone else, please
>> erase all copies and attachments, and please let me know that it has gone
>> to the wrong person.
>> >>
>> >> The above terms reflect a potential business arrangement, are provided
>> solely as a basis for further discussion, and are not intended to be and do
>> not constitute a legally binding obligation. No legally binding obligations
>> will be created, implied, or inferred until an agreement in final form is
>> executed in writing by all parties involved.
>>
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Cyrus Maden]

Congrats Brian!

On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía  wrote:

> Congratulations Brian!
> Happy to see this happening and eager to see more of your work!
>
> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka  wrote:
> >
> > Congrats Brian!
> >
> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
> >>
> >> Congrats Brian!
> >>
> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
> >>
> >> Great news!
> >>
> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  wrote:
> >>>
> >>> Congratulations!
> >>>
> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> 
>  Congratulations, Brian!
> 
>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
> sunjincheng...@gmail.com> wrote:
> >
> > Congratulation Brian!
> >
> > Best,
> > Jincheng
> >
> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
> >>
> >> Thanks for your contributions and congrats Brian!
> >>
> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles 
> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Brian Hulette
> >>>
> >>> Brian introduced himself to dev@ earlier this year and has been
> contributing since then. His contributions to Beam include explorations of
> integration with Arrow, standardizing coders, portability for schemas, and
> presentations at Beam events.
> >>>
> >>> In consideration of Brian's contributions, the Beam PMC trusts him
> with the responsibilities of a Beam committer [1].
> >>>
> >>> Thank you, Brian, for your contributions and looking forward to
> many more!
> >>>
> >>> Kenn, on behalf of the Apache Beam PMC
> >>>
> >>> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
> >>
> >>
> >>
> >> --
> >>
> >> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
> >>
> >> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>

Re: [Discuss] Beam mascot

[Hannah Jiang]

I also vote for firefly.

On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev 
wrote:

> I like the firefly sketch a lot, it's my favorite so far.
>
> On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw 
> wrote:
>
>> #37 from the sketches was the cuttlefish, which would put it at (with
>> 4 votes) the most popular so far. I do like the firefly too.
>>
>> On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas  wrote:
>> >
>> > Hi everyone, so exciting to see this convo taking off!
>> >
>> > I loved Alex's firefly! -- it can have so many cool variations, and as
>> a stuffed animal is very original.
>> >
>> > Other ideas I had are a caterpillar because it looks like a data
>> pipeline, lol or the beaver!
>> >
>> > Feedback on the current sketches.
>> > - They resemble a lot either the Octocat or Totoro [1]
>> > - I'd like to see something that is completely new and original,
>> pancakes from gRPC is an example[2]
>> > - Something more caricaturesque is better, since we can dress it up and
>> modify it
>> >
>> > To move forward, it seems that the animals that were winners in this
>> thread are:
>> >
>> > Beaver (3)
>> > Firefly (3)
>> > Lemur or votes on sketches (3)
>> > Cuttlefish (2)
>> > Hedgehog (1)
>> > Salmon (1)
>> >
>> > So let's focus the design proposals on the three winners: beaver,
>> firefly and lemur.
>> > I'd like to see more options on beavers and fireflies, the current
>> sketch options I think are based on the cuttlefish and the lemur (?)
>> >
>> > I think it's a good idea to get sketches from multiple designers, since
>> like someone else pointed out, we'll get variations based on their personal
>> styles, and someone else mentioned here that we have teams/companies
>>  with designers in their
>> teams, so let's take advantage of that as well :)
>> >
>> > I'd suggest to fork the conversation into a call for sketched and we
>> vote on that.
>> >
>> > [1]
>> https://www.google.com/search?q=totoro=ACYBGNTFW6vq76cHp05g4vBaR-SVJNI1iw:1573674471669=lnms=isch=X=0ahUKEwiTwICf-uflAhUKHqwKHVtzAykQ_AUIEigB=1440=735
>> > [2]
>> https://www.google.com/search?q=pancakes+grpc=isch=2ahUKEwixgqjV-uflAhUBOawKHX2pAfsQ2-cCegQIABAA=pancakes+grpc_l=img.3...13774.22674..22826...10.0..0.112.1818.13j6..01..gws-wiz-img.10..35i39j0j0i67j35i362i39j0i10j0i5i30j0i8i30.hNtaBfSYNv8=WV7MXfHxIYHysAX90obYDw=735=1440
>>
>

How to unsubscribe the Apache projects and jira issues notification

[P. Ramanjaneya Reddy]

Hi

Following blogs want to unsubscribe kindly guide.

I tried from google..still mails receiving

Also should unubscribe..

j...@apache.org


u...@flink.apache.org

d...@flink.apache.org

dev@beam.apache.org

u...@beamho.apache.org 


Thanks

Re: [ANNOUNCE] New committer: Brian Hulette

[Ismaël Mejía]

Congratulations Brian!
Happy to see this happening and eager to see more of your work!

On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka  wrote:
>
> Congrats Brian!
>
> On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:
>>
>> Congrats Brian!
>>
>> On 11/15/19 9:58 AM, Reza Rokni wrote:
>>
>> Great news!
>>
>> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  wrote:
>>>
>>> Congratulations!
>>>
>>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev  
>>> wrote:

 Congratulations, Brian!

 On Thu, Nov 14, 2019 at 6:25 PM jincheng sun  
 wrote:
>
> Congratulation Brian!
>
> Best,
> Jincheng
>
> Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>>
>> Thanks for your contributions and congrats Brian!
>>
>> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles  wrote:
>>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new 
>>> committer: Brian Hulette
>>>
>>> Brian introduced himself to dev@ earlier this year and has been 
>>> contributing since then. His contributions to Beam include explorations 
>>> of integration with Arrow, standardizing coders, portability for 
>>> schemas, and presentations at Beam events.
>>>
>>> In consideration of Brian's contributions, the Beam PMC trusts him with 
>>> the responsibilities of a Beam committer [1].
>>>
>>> Thank you, Brian, for your contributions and looking forward to many 
>>> more!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1] 
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>>
>>
>> --
>>
>> This email may be confidential and privileged. If you received this 
>> communication by mistake, please don't forward it to anyone else, please 
>> erase all copies and attachments, and please let me know that it has gone to 
>> the wrong person.
>>
>> The above terms reflect a potential business arrangement, are provided 
>> solely as a basis for further discussion, and are not intended to be and do 
>> not constitute a legally binding obligation. No legally binding obligations 
>> will be created, implied, or inferred until an agreement in final form is 
>> executed in writing by all parties involved.

Re: [ANNOUNCE] New committer: Brian Hulette

[Ankur Goenka]

Congrats Brian!

On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský  wrote:

> Congrats Brian!
> On 11/15/19 9:58 AM, Reza Rokni wrote:
>
> Great news!
>
> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  wrote:
>
>> Congratulations!
>>
>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> Congratulations, Brian!
>>>
>>> On Thu, Nov 14, 2019 at 6:25 PM jincheng sun 
>>> wrote:
>>>
 Congratulation Brian!

 Best,
 Jincheng

 Kyle Weaver  于2019年11月15日周五 上午7:19写道：

> Thanks for your contributions and congrats Brian!
>
> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Brian Hulette
>>
>> Brian introduced himself to dev@ earlier this year and has been
>> contributing since then. His contributions to Beam include explorations 
>> of
>> integration with Arrow, standardizing coders, portability for schemas, 
>> and
>> presentations at Beam events.
>>
>> In consideration of Brian's contributions, the Beam PMC trusts him
>> with the responsibilities of a Beam committer [1].
>>
>> Thank you, Brian, for your contributions and looking forward to many
>> more!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>
>
> --
>
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
>
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Jan Lukavský]

Congrats Brian!

On 11/15/19 9:58 AM, Reza Rokni wrote:

Great news!

On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov > wrote:


Congratulations!

On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev
mailto:valen...@google.com>> wrote:

Congratulations, Brian!

On Thu, Nov 14, 2019 at 6:25 PM jincheng sun
mailto:sunjincheng...@gmail.com>>
wrote:

Congratulation Brian!

Best,
Jincheng

Kyle Weaver mailto:kcwea...@google.com>> 于2019年11月15日周五
上午7:19写道：

Thanks for your contributions and congrats Brian!

On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles
mailto:k...@apache.org>> wrote:

Hi all,

Please join me and the rest of the Beam PMC in
welcoming a new committer: Brian Hulette

Brian introduced himself to dev@ earlier this year
and has been contributing since then. His
contributions to Beam include explorations of
integration with Arrow, standardizing coders,
portability for schemas, and presentations at Beam
events.

In consideration of Brian's contributions, the
Beam PMC trusts him with the responsibilities of a
Beam committer [1].

Thank you, Brian, for your contributions and
looking forward to many more!

Kenn, on behalf of the Apache Beam PMC

[1]

https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer



--

This email may be confidential and privileged. If you received this 
communication by mistake, please don't forward it to anyone else, 
please erase all copies and attachments, and please let me know that 
it has gone to the wrong person.


The above terms reflect a potential business arrangement, are provided 
solely as a basis for further discussion, and are not intended to be 
and do not constitute a legally binding obligation. No legally binding 
obligations will be created, implied, or inferred until an agreement 
in final form is executed in writing by all parties involved.

Re: On processing event streams

[Jan Lukavský]

Hi Kenn,

I'll quote here all the recent comments (from this and the (closed) 
thread [1]):


> Your proposed feature is sensitive to all data that is not in 
timestamp order, which is not the same as late. In Beam "late" is 
defined as "assigned to a window where the watermark has passed the end 
of the window and a 'final' aggregate has been produced". Your proposal 
is not really sensitive to this form of late data.


> I think there is some published work that will help you particularly 
in addressing out-of-order data. Note that this is not the normal notion 
of late. . Trill has a high-watermark driven sorting buffer prior to 
sending elements in order to stateful operators. It is similar to your 
sketched algorithm for emitting elements as the watermark passes. I 
believe Gearpump also uses a sorting buffer and processes in order, and 
we do have a Gearpump runner still here in our repo.


> This is a pre-watermark, pre-Beam approach to processing data. It 
drops more data and/or introduces more latency, but can lead to simpler 
or more efficient operator implementations (but not always).
> I do think it seems OK to add this to Beam in some form as a 
convenience when the user knows something about their data and/or DoFn. 
The risk of course is that users go for this when they shouldn't, 
thinking it is simpler without considering the complexity of how to 
avoid dropping too much data.


All these seem related. Let me try to explain. I'll try to keep this as 
short as possible, but because the topic is not trivial and possibly 
touches many related problems I might fail doing so. Please bear with me. :)


I'm aware of the difference between out-of-order data and late data. I 
don't think that the (generally) described scheme "unordered stream -> 
buffer -> ordered stream" has anything to do with how model (or runner) 
handles time progress. This is general idea and the respective 
properties of the this computation scheme depends on how we define 
condition when the buffer gets flushed (this can - and should be for the 
reasons you mention - be driven by watermark progress). The PR [2] 
already works on watermark progress (because it is driven by event-time 
timers). If I'm not missing something, then this approach seems to be 
very much aligned with "current Beam" approach - please correct me if 
I'm wrong.


There are two main open issues with the currently proposed approach 
(that I'm aware of). These are:
 a) as opposed to the original proposal, the current implementation 
does not enable a UDF for extraction of some fine-grained time to be 
used as sorting criterion (e.g. sequential IDs that might be present in 
data)
 b) it doesn't handle allowed lateness correctly (this might probably 
be what you are referring to)


Issue a) is sort of trivial, it is just about how to do this in a most 
compatible way in the rest of Beam APIs. Moreover, this can be trivially 
postponed and solved later, so I would ignore it for now.


Issue b) is much worse. The current approach is to force allowed 
lateness to zero, dropping all late elements. I'm not fine with that 
(and maybe as part of a review it might be concluded that a better 
approach would be to actually wait until watermark reaches T + 
allowedLateness before flushing the buffer, being consistent with 
dropping only really late elements, but introducing more latency). There 
is a 'correct' solution to this, though. That would be (very roughly 
outlined):

 a) create single input buffer, let's call this "input buffer"
 b) create two copies of the stateful DoFn consuming the sorted stream 
(scope all created states and timers with an additional dimension - one 
is "on time" and the other "late" - that is state and timers would 
reside in namespace (k, v, "on time" | "late") - let's call these two 
respective DoFns "on time" and "late"

 c) on each watermark update, do the following:
  i) take all elements with timestamp less than watermark, sort them 
and flush them out to the consuming "on time" stateful DoFn, _storing 
all output produced by the DoFn into separate state (let's call it 
"output buffer"), passing it downstream as well_
  ii) take all elements with timestamp less than watermak - 
allowedLateness, sort them and pass them to consuming "late" stateful 
DoFn, _storing all results into "late output buffer", not passing 
anything into output_
  iii) compare "output buffer" with "late output buffer" and *retract 
data elements that were part output buffer, but were not present in late 
buffer, and output data that were not in output buffer, but were present 
in late output buffer*
  iv) trim "output buffer", "late output buffer" and "input buffer" to 
contain only elements with timestamp not preceding watermark - 
allowedLateness


I have (currently) a strong belief that this is actually can serve as a 
base of fully generic solution to retraction problem (it would only 
require to actually handle all stateful operations as stateful DoFns 

Re: [ANNOUNCE] New committer: Brian Hulette

[Reza Rokni]

Great news!

On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov  wrote:

> Congratulations!
>
> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev 
> wrote:
>
>> Congratulations, Brian!
>>
>> On Thu, Nov 14, 2019 at 6:25 PM jincheng sun 
>> wrote:
>>
>>> Congratulation Brian!
>>>
>>> Best,
>>> Jincheng
>>>
>>> Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>>>
 Thanks for your contributions and congrats Brian!

 On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Brian Hulette
>
> Brian introduced himself to dev@ earlier this year and has been
> contributing since then. His contributions to Beam include explorations of
> integration with Arrow, standardizing coders, portability for schemas, and
> presentations at Beam events.
>
> In consideration of Brian's contributions, the Beam PMC trusts him
> with the responsibilities of a Beam committer [1].
>
> Thank you, Brian, for your contributions and looking forward to many
> more!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>


-- 

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

Re: Proposal: @RequiresTimeSortedInput

[Jan Lukavský]

Hi Kenn,

I would continue with this discussion on the thread [1] (as you propose 
as well) and consider all the other threads regarding this as closed.


I will take your latest note in a reply there.

Jan

[1] 
https://lists.apache.org/thread.html/e2f729c7cea22553fc34421d4547132fa1c2ec01035eb4fb1a426873@%3Cdev.beam.apache.org%3E


On 11/15/19 5:56 AM, Kenneth Knowles wrote:

Hi Jan,

Sorry for the very slow reply.

Your proposed feature is sensitive to all data that is not in 
timestamp order, which is not the same as late. In Beam "late" is 
defined as "assigned to a window where the watermark has passed the 
end of the window and a 'final' aggregate has been produced". Your 
proposal is not really sensitive to this form of late data.


I think there is some published work that will help you particularly 
in addressing out-of-order data. Note that this is not the normal 
notion of late. . Trill has a high-watermark driven sorting buffer 
prior to sending elements in order to stateful operators. It is 
similar to your sketched algorithm for emitting elements as the 
watermark passes. I believe Gearpump also uses a sorting buffer and 
processes in order, and we do have a Gearpump runner still here in our 
repo.


Kenn

On Mon, Nov 4, 2019 at 3:54 AM Jan Lukavský > wrote:


Hi,

there has been some development around this [1], which essentially
concludes that currently this feature can be safely supported only by
direct runner, flink runner (both batch and streaming, non-portable
only) and spark (batch, legacy only). This is due to the fact,
that time
sorting relies heavily on timers to be strictly ordered. Failing
to do
so might result in unpredictable data loss, due to window-cleanup of
state occurring prior to all elements being emitted (note that this
generally might happen even to current user pipelines!). I can link
issues [2], [3] and [4] to [5], but the question is, with only so few
runners being able to support this, what should be the best way to
incorporate this into any upcoming release (I'm assuming that this
will
pass a vote, which is not known yet)? I'd say that the best way
would be
the affected runners to fail to execute the pipeline until the
respective issues are resolved. Another option would be to block this
until the issues are resolved in runners, but that might delay the
availability of this feature for some unknown time.

Thanks for any opinions,

Jan

[1]

https://lists.apache.org/thread.html/71a8f48ca518f1f2e6e9b1284114624670884775d209b0097f68264b@%3Cdev.beam.apache.org%3E

[2] https://issues.apache.org/jira/browse/BEAM-8459

[3] https://issues.apache.org/jira/browse/BEAM-8460

[4] https://issues.apache.org/jira/browse/BEAM-8543.

[5] https://issues.apache.org/jira/browse/BEAM-8550

On 10/31/19 2:59 PM, Jan Lukavský wrote:
> Hi,
>
> as a follow-up from previous design draft, I'd like to promote the
> document [1] and associated PR [2] to proposal.
>
> The PR contains working implementation for:
>
>  - non-portable batch flink and batch spark (legacy)
>
>  - all non-portable streaming runners that use StatefulDoFnRunner
> (direct, samza, dataflow)
>
>  - portable flink (batch, streaming)
>
> There are still some unresolved issues:
>
>  a) no way to specify allowed lateness (currently is simply
zero, late
> data should be dropped)
>
>  b) need a way to specify user UDF for extracting timestamp
(according
> to [3] it would be useful to have that option)
>
>  c) need to add more tests (e.g. late data)
>
> The plan is to postpone resolution of issues a) and b) after the
> proposal is merged. I'd like to gather some more feedback on the
> proposal, iterate over that again, add more tests and then pass
this
> to a vote.
>
> Unrelated - during implementation a bug [4] in Samza runner was
found.
>
> Looking forward to any comments!
>
> Jan
>
> [1]
>

https://docs.google.com/document/d/1ObLVUFsf1NcG8ZuIZE4aVy2RYKx2FfyMhkZYWPnI9-c/

>
>
> [2] https://github.com/apache/beam/pull/8774
>
> [3]
>

https://lists.apache.org/thread.html/813429e78b895a336d4f5507e3b2330282e2904fa25d52d6d441741a@%3Cdev.beam.apache.org%3E
>
> [4] https://issues.apache.org/jira/browse/BEAM-8529
>
>
> On 5/23/19 4:10 PM, Jan Lukavský wrote:
>> Hi,
>>
>> I have written a very brief draft of how it might be possible to
>> implement @RequireTimeSortedInput discussed in [1]. I see the
>> document [2] a starting point for a discussion. There are several
>> open questions, which I believe can be resolved by this great
>> community. :-)
>>
>> Jan
>>
>> [1]
>>