Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-06-09 Thread Jasper Wang 
Hi team,

Just my first very small contribution:

i. By using the SDK 2.5 python dev distribution JB shared, I have just
managed to successfully launch my own custom streaming job on Dataflow.
It's working fine.

ii. this custom streaming reads data from pub/sub, conducts some simple
transformations, then injects data into BigQuery.

FYI.

Cheers.

Jasper

On Sat, Jun 9, 2018 at 9:37 AM, Boyuan Zhang  wrote:

> Hey JB,
>
> We just updated the release guide(https://github.com/
> apache/beam-site/commit/dac24e6bee4b3202fbd33543f9d66e80e8b7cd32). Please
> add your updates to release guide if you want to.
>
> Thanks so much for your help!
>
> Boyuan
>
> On Tue, Jun 5, 2018 at 10:16 AM Jean-Baptiste Onofré 
> wrote:
>
>> Thanks, yeah I saw that but I'm planning to add some additional notes.
>>
>> Regards
>> JB
>> Le 5 juin 2018, à 19:02, Boyuan Zhang  a écrit:
>>>
>>> Hey JB,
>>>
>>> We have some updates in :  https://github.com/pabloem/beam-site/blob/
>>> 372c93ecbafbf3a1440492df1e12050dfe939e91/src/contribute/release-guide.md
>>> and  https://github.com/apache/beam-site/pull/424/files
>>> .
>>> They may be helpful.
>>>
>>> Boyuan
>>>
>>> On Tue, Jun 5, 2018 at 9:50 AM Jean-Baptiste Onofré < j...@nanthrax.net>
>>> wrote:
>>>
 Was checking if there was something like the release branch of the
 maven release plugin, but there's not with the gradle one.

 I'm creating the branch by hand and I'm updating the release guide in
 the mean time.

 Regards
 JB
 Le 5 juin 2018, à 18:44, "Jean-Baptiste Onofré" < j...@nanthrax.net> a
 écrit:
>
> On the way
> Le 5 juin 2018, à 18:23, Kenneth Knowles < k...@google.com> a écrit:
>>
>> Have you cut the release branch? It is much easier to stabilize a cut
>> branch that is separated from continued development on master. I think we
>> have to cut it before continuing.
>>
>> Kenn
>>
>> On Tue, Jun 5, 2018 at 1:14 AM Jean-Baptiste Onofré < j...@nanthrax.net>
>> wrote:
>>
>>> Sorry for the noise: this build error was due to a corrupted file in
>>> my
>>> .m2/repository.
>>>
>>> The HDFS extension build is now OK. I'm launching a full build.
>>>
>>> Regards
>>> JB
>>>
>>> On 05/06/2018 07:43, Jean-Baptiste Onofré wrote:
>>> > New failure on the build:
>>> >
>>> > FAILURE: Build failed with an exception.
>>> >
>>> > * What went wrong:
>>> > Could not resolve all files for configuration
>>> > ':beam-sdks-java-io-hadoop-file-system:testCompileClasspath'.
>>> >> Could not find zookeeper-tests.jar 
>>> >> (org.apache.zookeeper:zookeeper:3.4.6).
>>>
>>> >   Searched in the following locations:
>>> >
>>> > file:/home/jbonofre/.m2/repository/org/apache/
>>> zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
>>> >
>>> > I'm fixing the HDFS extension.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 05/06/2018 07:18, Jean-Baptiste Onofré wrote:
>>> >> Hi,
>>> >>
>>> >> yes, it's release blocker: the build is not fully stable. I'm
>>> trying to
>>> >> build the release for one week and it fails with different
>>> errors.
>>> >>
>>> >> I have a new build in progress. I hope it will be good. I keep
>>> you posted.
>>> >>
>>> >> Regards
>>> >> JB
>>> >>
>>> >> On 05/06/2018 01:38, Scott Wegner wrote:
>>> >>> Hey JB, you mentioned some build issues on Slack [1]. Is this
>>> blocking
>>> >>> the release? Let me know if there's anything I can help with.
>>> >>>
>>> >>> [1]  https://the-asf.slack.com/archives/C9H0YNP3P/
>>> p1528133545000136
>>> >>>
>>> >>> On Sun, Jun 3, 2018 at 10:58 PM Jean-Baptiste Onofré <
>>> j...@nanthrax.net
>>> >>> > wrote:
>>> >>>
>>> >>> Hi guys,
>>> >>>
>>> >>> just to let you know that the build is now OK. I'm
>>> completing the Jira
>>> >>> triage this morning (my time) and cut the release branch
>>> (starting the
>>> >>> release process). I will validate the release guide in the
>>> mean time.
>>> >>>
>>> >>> Thanks,
>>> >>> Regards
>>> >>> JB
>>> >>>
>>> >>> On 06/04/2018 10:48, Jean-Baptiste Onofré wrote:
>>> >>> > Hi guys,
>>> >>> >
>>> >>> > Apache Beam 2.4.0 has been released on March 20th.
>>> >>> >
>>> >>> > According to our cycle of release (roughly 6 weeks), we
>>> should
>>> >>> think about 2.5.0.
>>> >>> >
>>> >>> > I'm volunteer to tackle this release.
>>> >>> >
>>> >>> > I'm proposing the following items:
>>> >>> >
>>> >>> > 1. We start the Jira triage now, up to Tuesday
>>> >>> > 2. I would like to cut the release on Tuesday night

Re: Beam Cookbook?

2018-06-09 Thread Davor Bonaci 
Hi Austin --
It would be great to see this materialize.

I've been pursued by publishers a lot in the last year, so I might be able
to facilitate introductions if you need them. You'll probably want to have
a few sample draft chapters, however.

I'm aware of 3 different groups creating related content in this space, but
I think nobody is doing it from this angle. They are lurking on the mailing
list, so they may have reached out already.

All the best, and I wish you finish it with the same enthusiasm as you
started!

Davor

On Thu, Jun 7, 2018 at 11:44 AM, Austin Bennett  wrote:

> I'm looking at assembling a physical book along the lines of "Apache Beam
> Cookbook", though might take a different approach to topic (if realize
> there is a better hole to fill or something that needs more attention
> before that).
>
> I believe many could benefit from more substantive write-ups and
> explanations on use-cases, and specific bits in code (ex: to accomplish x
> you might want to use recipe Y, pay special attention to this function,
> with associated paragraphs of text and noting specific lines in the code,
> etc etc).  While this can be done on a website and GitHub, I do believe the
> more concrete nature of a book (esp. with reputable publisher) gives
> additional signaling to others that the subject is sufficiently mature.  My
> aim will be for this book to be freely available at least in an e-book
> version, an example of that I have is: https://www.confluent.io/
> resources/kafka-the-definitive-guide/ and surely you've come across other
> examples.
>
> I see many cookbook examples of code already exist, but the associated
> writeup I know could be useful to others; as well as the overall
> presentation/bundling to make it even easier to find and use.
>
> Wondering thoughts from the group and if there are others with a strong
> interest in collaborating on such an undertaking.
>
>
>

Re: [DISCUSS] Use Confluence wiki for non-user-facing stuff

2018-06-09 Thread Jean-Baptiste Onofré 
Sure, let me change the permissions.

Regards
JB

On 09/06/2018 15:57, Kenneth Knowles wrote:
> Yea, we have one but it seems to not be set up quite right. JB you are
> the admin from the original incubation. Can you check the permissions?
> Also make some more admins (maybe all the PMC to start with). I would
> like to help out.
> 
> https://cwiki.apache.org/confluence/display/BEAM/
> https://issues.apache.org/jira/browse/INFRA-11181
> 
> 
> Kenn
> 
> On Fri, Jun 8, 2018 at 10:12 PM Jean-Baptiste Onofré  > wrote:
> 
> +1
> 
> That's funny, because AFAIR, it's what we discussed in the early stage
> of the project.
> We can also link wiki pages on the website if we want to provide
> details.
> AFAIR, we already have a confluence wiki space for Beam.
> 
> Regards
> JB
> 
> On 07/06/2018 22:23, Kenneth Knowles wrote:
> > Hi all,
> >
> > I've been in half a dozen conversations recently about whether to
> have a
> > wiki and what to use it for. Some things I've heard:
> >
> >  - "why is all this stuff that users don't care about here?"
> >  - "can we have a lighter weight place to put technical references for
> > contributors"
> >
> > So I want to consider as a community starting up our wiki. Ideas for
> > what could go there:
> >
> >  - Collection of links to design docs like
> > https://beam.apache.org/contribute/design-documents/
> >  - Specialized walkthroughs
> > like https://beam.apache.org/contribute/docker-images/
> >  - Best-effort notes that just try to help out
> > like https://beam.apache.org/contribute/intellij/
> >  - Docs on in-progress stuff
> > like https://beam.apache.org/documentation/runners/jstorm/
> >  - Expanded instructions for committers, more
> > than https://beam.apache.org/contribute/committer-guide/
> >  - BIPs / summaries of collections of JIRA
> >  - Docs sitting in markdown in the repo
> > like https://github.com/apache/beam/blob/master/sdks/CONTAINERS.md
> > and https://github.com/apache/beam-site/blob/asf-site/README.md (which
> > will soon not be a toplevel README)
> >
> > What do you think?
> >
> > (a) should we do it?
> > (b) what should go there?
> > (c) what should not go there?
> >
> > Kenn
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [DISCUSS] Use Confluence wiki for non-user-facing stuff

2018-06-09 Thread Kenneth Knowles 
Yea, we have one but it seems to not be set up quite right. JB you are the
admin from the original incubation. Can you check the permissions? Also
make some more admins (maybe all the PMC to start with). I would like to
help out.

https://cwiki.apache.org/confluence/display/BEAM/
https://issues.apache.org/jira/browse/INFRA-11181


Kenn

On Fri, Jun 8, 2018 at 10:12 PM Jean-Baptiste Onofré 
wrote:

> +1
>
> That's funny, because AFAIR, it's what we discussed in the early stage
> of the project.
> We can also link wiki pages on the website if we want to provide details.
> AFAIR, we already have a confluence wiki space for Beam.
>
> Regards
> JB
>
> On 07/06/2018 22:23, Kenneth Knowles wrote:
> > Hi all,
> >
> > I've been in half a dozen conversations recently about whether to have a
> > wiki and what to use it for. Some things I've heard:
> >
> >  - "why is all this stuff that users don't care about here?"
> >  - "can we have a lighter weight place to put technical references for
> > contributors"
> >
> > So I want to consider as a community starting up our wiki. Ideas for
> > what could go there:
> >
> >  - Collection of links to design docs like
> > https://beam.apache.org/contribute/design-documents/
> >  - Specialized walkthroughs
> > like https://beam.apache.org/contribute/docker-images/
> >  - Best-effort notes that just try to help out
> > like https://beam.apache.org/contribute/intellij/
> >  - Docs on in-progress stuff
> > like https://beam.apache.org/documentation/runners/jstorm/
> >  - Expanded instructions for committers, more
> > than https://beam.apache.org/contribute/committer-guide/
> >  - BIPs / summaries of collections of JIRA
> >  - Docs sitting in markdown in the repo
> > like https://github.com/apache/beam/blob/master/sdks/CONTAINERS.md
> > and https://github.com/apache/beam-site/blob/asf-site/README.md (which
> > will soon not be a toplevel README)
> >
> > What do you think?
> >
> > (a) should we do it?
> > (b) what should go there?
> > (c) what should not go there?
> >
> > Kenn
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>