Re: git commit message

2017-04-30 Thread Jean-Baptiste Onofré 
Ah, as the subject of the thread is "git commit message", I thought you talked 
about the commit message.


The commit message should look like "[BEAM-xx] message" for the squashed 
content, and "[BEAM-xx] This closes #" for the merge commit.


Regarding the pull request title, I highly recommend to use the Jira ID and 
quick abstract of the changes in the PR.


Regards
JB

On 04/30/2017 10:40 PM, Ted Yu wrote:

I was actually talking about title of the PR.

From the link, there is guideline which some of the recent commits didn't
follow:

The title of the pull request should be strictly in the following format:


On Sun, Apr 30, 2017 at 1:22 PM, Chamikara Jayalath 
wrote:


Do we have a convention on the commit message ? Seems like Contributor's
guide only talks about the title of the PR not the commit message. I might
be missing something.

https://beam.apache.org/contribute/contribution-guide/
#create-a-pull-request

Thanks,
Cham

On Sun, Apr 30, 2017 at 12:41 PM Jean-Baptiste Onofré 
wrote:


Hi,

IMHO, it's very rare. That's true we did couple of "fast" commit to fix
issues
(I remember at the beginning of the project).

However, almost all changes go via PR and most of with a Jira.

So, if maybe you found couple of, again, it's rare and we follow the
Jira/PR/review/commit path.

Regards
JB

On 04/30/2017 04:59 PM, Ted Yu wrote:

Hi,
When I went over git commit messages, I found some without either of

the

following:

JIRA number
PR number

It would be nice for other people to get background / goal / discussion
w.r.t. any commit if either (or both) of the above is present.

My two cents.



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com







--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[PROPOSAL] Running Splittable DoFn via Source API

2017-04-30 Thread Eugene Kirpichov 
Hey all,

TL;DR: Development of SDF ecosystem (transitioning existing connectors to
SDF, building libraries, battle-testing the API) is currently blocked on
having SDF supported at full parity with Source API in all Beam runners,
which will take a long time.

But we can unblock all this work and start doing it very soon, by running a
special case of SDF on top of the Source API.

I think this is very exciting. Please comment on the following short
proposal!

https://s.apache.org/sdf-via-source

After getting some consensus on the doc and in this thread, I will start
filing a network of JIRAs and follow up with next steps.

Re: [JENKINS] - Removal of top level views

2017-04-30 Thread Jason Kuster 
I'm happy to do it, or help out if someone else is interested.

On Sun, Apr 30, 2017 at 9:27 PM, Davor Bonaci  wrote:

> We need to move our top-level view [1] into the A-D view [2].
>
> Any volunteers willing to code this view in the Jenkins DSL?
>
> Davor
>
> [1] https://builds.apache.org/view/Beam/
> [2] https://builds.apache.org/view/A-D/view/Beam/
>
> -- Forwarded message --
> From: Gavin McDonald 
> Date: Sun, Apr 30, 2017 at 1:19 AM
> Subject: [JENKINS] - Removal of top level views
> To: bui...@apache.org
> Cc: us...@infra.apache.org
>
>
> Hi All,
>
> We are starting to get complaints about the amount of top level views in
> our Jenkins instance.
>
> I believe it is time for cleanup of those views once more.
>
> A few years ago, I cleaned up a heap of views and got quite some complaints
> from projects
> about doing so, without consultation, and the fact that this breaks links
> some projects use.
>
> Since that time however, projects have continued to create new Top Level
> views and now
> we are at a stage where again there are too many - there are currently 22
> project type views.
>
> There are many more projects and I’d hate for this trend to continue
> upwards and clutter the
> main Jenkins landing page even more.
>
> I have created an INFRA ticket to track this at :
>
> https://issues.apache.org/jira/browse/INFRA-14002
>
> The views I propose to remove are :
>
> Airavata, Apache Commons, Apex, Beam Brooklyn, Calciate, CarbonData, DL,
> Groovy, Hawq, Infra Jobs Monitor View, Lucene, OWB, Olingo, OpenJPA, POI,
> Shiro, Tamaya, Tika, XMLGraphics. cloudstack, directory
>
> Also: All the single letter viewss from A to Z. They are not much used and
> it is preferred to use 'A-D', 'E-G', 'H-L', 'M-R' and 'S-Z'
>
> Two Views unsure what to do with are: 'Incubator Projects' and 'PreCommit
> Builds'
>
>
> Some views have already been removed, I removed one called ‘jobs I have
> access to’ or something like that, the Maven project also removed their top
> level views so my thanks to them.
> The Infra jobs monitor view is one I created, my fault and Im going to move
> it under H-L where it belongs.
>
> ACTION NEEDED:- Please, could all project who are listed above as having a
> top level view, please move it to a view under the appropriate letters
> group, or remove it.
>
> The two Im not sure about ‘Incubator Projects’ and ‘PreCommit Builds’ ,
> ‘Incubator Projects’ may be a benefit at top level if used correctly.
> ‘PreCommit Builds’ is not one owned by any one project, Im unsure whether
> this could stay top level and be a benefit.
>
> I’ll be moving the Infra views shortly.
>
> If, after a week, there are still projects at the top level view, I’ll move
> onto the next step of mailing the projects dev lists directly, though that
> should not be required if they have the representation on this list.
>
> Thanks in advance.
>
> I’ll keep the INFRA-14002 ticket updated with progress.
>
> Gav… (ASF Infra.)
>



-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

Fwd: [JENKINS] - Removal of top level views

2017-04-30 Thread Davor Bonaci 
We need to move our top-level view [1] into the A-D view [2].

Any volunteers willing to code this view in the Jenkins DSL?

Davor

[1] https://builds.apache.org/view/Beam/
[2] https://builds.apache.org/view/A-D/view/Beam/

-- Forwarded message --
From: Gavin McDonald 
Date: Sun, Apr 30, 2017 at 1:19 AM
Subject: [JENKINS] - Removal of top level views
To: bui...@apache.org
Cc: us...@infra.apache.org


Hi All,

We are starting to get complaints about the amount of top level views in
our Jenkins instance.

I believe it is time for cleanup of those views once more.

A few years ago, I cleaned up a heap of views and got quite some complaints
from projects
about doing so, without consultation, and the fact that this breaks links
some projects use.

Since that time however, projects have continued to create new Top Level
views and now
we are at a stage where again there are too many - there are currently 22
project type views.

There are many more projects and I’d hate for this trend to continue
upwards and clutter the
main Jenkins landing page even more.

I have created an INFRA ticket to track this at :

https://issues.apache.org/jira/browse/INFRA-14002

The views I propose to remove are :

Airavata, Apache Commons, Apex, Beam Brooklyn, Calciate, CarbonData, DL,
Groovy, Hawq, Infra Jobs Monitor View, Lucene, OWB, Olingo, OpenJPA, POI,
Shiro, Tamaya, Tika, XMLGraphics. cloudstack, directory

Also: All the single letter viewss from A to Z. They are not much used and
it is preferred to use 'A-D', 'E-G', 'H-L', 'M-R' and 'S-Z'

Two Views unsure what to do with are: 'Incubator Projects' and 'PreCommit
Builds'


Some views have already been removed, I removed one called ‘jobs I have
access to’ or something like that, the Maven project also removed their top
level views so my thanks to them.
The Infra jobs monitor view is one I created, my fault and Im going to move
it under H-L where it belongs.

ACTION NEEDED:- Please, could all project who are listed above as having a
top level view, please move it to a view under the appropriate letters
group, or remove it.

The two Im not sure about ‘Incubator Projects’ and ‘PreCommit Builds’ ,
‘Incubator Projects’ may be a benefit at top level if used correctly.
‘PreCommit Builds’ is not one owned by any one project, Im unsure whether
this could stay top level and be a benefit.

I’ll be moving the Infra views shortly.

If, after a week, there are still projects at the top level view, I’ll move
onto the next step of mailing the projects dev lists directly, though that
should not be required if they have the representation on this list.

Thanks in advance.

I’ll keep the INFRA-14002 ticket updated with progress.

Gav… (ASF Infra.)

Re: git commit message

2017-04-30 Thread Ted Yu 
I was actually talking about title of the PR.

>From the link, there is guideline which some of the recent commits didn't
follow:

The title of the pull request should be strictly in the following format:


On Sun, Apr 30, 2017 at 1:22 PM, Chamikara Jayalath 
wrote:

> Do we have a convention on the commit message ? Seems like Contributor's
> guide only talks about the title of the PR not the commit message. I might
> be missing something.
>
> https://beam.apache.org/contribute/contribution-guide/
> #create-a-pull-request
>
> Thanks,
> Cham
>
> On Sun, Apr 30, 2017 at 12:41 PM Jean-Baptiste Onofré 
> wrote:
>
> > Hi,
> >
> > IMHO, it's very rare. That's true we did couple of "fast" commit to fix
> > issues
> > (I remember at the beginning of the project).
> >
> > However, almost all changes go via PR and most of with a Jira.
> >
> > So, if maybe you found couple of, again, it's rare and we follow the
> > Jira/PR/review/commit path.
> >
> > Regards
> > JB
> >
> > On 04/30/2017 04:59 PM, Ted Yu wrote:
> > > Hi,
> > > When I went over git commit messages, I found some without either of
> the
> > > following:
> > >
> > > JIRA number
> > > PR number
> > >
> > > It would be nice for other people to get background / goal / discussion
> > > w.r.t. any commit if either (or both) of the above is present.
> > >
> > > My two cents.
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

Re: git commit message

2017-04-30 Thread Chamikara Jayalath 
Do we have a convention on the commit message ? Seems like Contributor's
guide only talks about the title of the PR not the commit message. I might
be missing something.

https://beam.apache.org/contribute/contribution-guide/#create-a-pull-request

Thanks,
Cham

On Sun, Apr 30, 2017 at 12:41 PM Jean-Baptiste Onofré 
wrote:

> Hi,
>
> IMHO, it's very rare. That's true we did couple of "fast" commit to fix
> issues
> (I remember at the beginning of the project).
>
> However, almost all changes go via PR and most of with a Jira.
>
> So, if maybe you found couple of, again, it's rare and we follow the
> Jira/PR/review/commit path.
>
> Regards
> JB
>
> On 04/30/2017 04:59 PM, Ted Yu wrote:
> > Hi,
> > When I went over git commit messages, I found some without either of the
> > following:
> >
> > JIRA number
> > PR number
> >
> > It would be nice for other people to get background / goal / discussion
> > w.r.t. any commit if either (or both) of the above is present.
> >
> > My two cents.
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: git commit message

2017-04-30 Thread Jean-Baptiste Onofré 

Hi,

IMHO, it's very rare. That's true we did couple of "fast" commit to fix issues 
(I remember at the beginning of the project).


However, almost all changes go via PR and most of with a Jira.

So, if maybe you found couple of, again, it's rare and we follow the 
Jira/PR/review/commit path.


Regards
JB

On 04/30/2017 04:59 PM, Ted Yu wrote:

Hi,
When I went over git commit messages, I found some without either of the
following:

JIRA number
PR number

It would be nice for other people to get background / goal / discussion
w.r.t. any commit if either (or both) of the above is present.

My two cents.



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Status of our CI tools

2017-04-30 Thread Chamikara Jayalath 
+1

So far, I've been able to commit to Python SDK without caring much about
Travis. At least for now, it's not too hard to do testing locally in rare
cases where Jenkins is not stable. Hopefully Jenkins will become more
stable/faster over time.

- Cham

On Sun, Apr 30, 2017 at 10:19 AM Dan Halperin 
wrote:

> I think the confusion to new users is much worse than any temporary loss of
> functionality here. +1 * 100!
>
> On Fri, Apr 28, 2017 at 11:00 PM, Mingmin Xu  wrote:
>
> > +1
> > Have ignored TravisCI for some time as the failures are not related with
> > code/test issues.
> >
> > I still hope TravisCI could work with Beam code repository some day, to
> > run tests before creating a PR.
> >
> > Mingmin
> >
> > > On Apr 28, 2017, at 10:26 PM, Aljoscha Krettek 
> > wrote:
> > >
> > > Big +1
> > >
> > >> On 29. Apr 2017, at 07:21, Robert Bradshaw
> 
> > wrote:
> > >>
> > >> On Fri, Apr 28, 2017 at 9:56 PM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > wrote:
> > >>> +1
> > >>>
> > >>> Travis is useless and our Jenkins is good IMHO !
> > >>
> > >> Travis is really useful for the Python SDK, but I'm hopeful that soon
> > >> Jenkins will be stable and quick enough that I won't miss it, and
> > >> having only one CI to deal with should simplify things.
> > >>
> > >> - Robert
> > >
> >
>

Re: How to create an unbounded PCollection?

2017-04-30 Thread Shen Li 
Sorry about the duplication. Please ignore the above email.

Shen

On Sat, Apr 29, 2017 at 3:43 PM, Shen LI  wrote:

> It seems that Create.of(Iterable) can only create a BoundedSource. Is there
> a convenient way to read from an unbounded Iterable object without writing
> application code to wrap it into an UnboundedSource object?
>
> Thanks,
>
> Shen
>

How to create an unbounded PCollection?

2017-04-30 Thread Shen LI 
It seems that Create.of(Iterable) can only create a BoundedSource. Is there
a convenient way to read from an unbounded Iterable object without writing
application code to wrap it into an UnboundedSource object?

Thanks,

Shen

Re: Status of our CI tools

2017-04-30 Thread Dan Halperin 
I think the confusion to new users is much worse than any temporary loss of
functionality here. +1 * 100!

On Fri, Apr 28, 2017 at 11:00 PM, Mingmin Xu  wrote:

> +1
> Have ignored TravisCI for some time as the failures are not related with
> code/test issues.
>
> I still hope TravisCI could work with Beam code repository some day, to
> run tests before creating a PR.
>
> Mingmin
>
> > On Apr 28, 2017, at 10:26 PM, Aljoscha Krettek 
> wrote:
> >
> > Big +1
> >
> >> On 29. Apr 2017, at 07:21, Robert Bradshaw 
> wrote:
> >>
> >> On Fri, Apr 28, 2017 at 9:56 PM, Jean-Baptiste Onofré 
> wrote:
> >>> +1
> >>>
> >>> Travis is useless and our Jenkins is good IMHO !
> >>
> >> Travis is really useful for the Python SDK, but I'm hopeful that soon
> >> Jenkins will be stable and quick enough that I won't miss it, and
> >> having only one CI to deal with should simplify things.
> >>
> >> - Robert
> >
>

Re: What's the easiest way for an application to convert an Iterable to an UnboundedSource

2017-04-30 Thread Dan Halperin 
Hi Shen,

Most runners are expected to use `UnboundedReadFromBoundedSource` (in
`runners-core-construction`) to convert a BoundedSource to an
UnboundedSource if that is the semantics they need.

As Eugene says, I suspect you can also get similar behavior with a
SplittableDoFn.

Dan

On Sat, Apr 29, 2017 at 10:56 PM, Jean-Baptiste Onofré 
wrote:

> Yes, sorry, TestStream (a bit early in the morning ;)).
>
> Thanks to have corrected me ;)
>
> Regards
> JB
>
>
> On 04/30/2017 07:17 AM, Eugene Kirpichov wrote:
>
>> Clarification: likely you meant TestStream
>> > src/main/java/org/apache/beam/sdk/testing/TestStream.java>
>> ?
>>
>> On Sat, Apr 29, 2017 at 10:03 PM Jean-Baptiste Onofré 
>> wrote:
>>
>> Hi,
>>>
>>> In addition of Eugene and Jesse's answers, for testing purpose, it's
>>> possible to
>>> use CreateStream in some runners. It allows us to test streaming.
>>>
>>> Regards
>>> JB
>>>
>>> On 04/29/2017 09:50 PM, Shen Li wrote:
>>>
 It seems that Create.of(Iterable) can only create a BoundedSource. Is

>>> there
>>>
 a convenient way to read from an unbounded Iterable object without

>>> writing
>>>
 application code to wrap it into an UnboundedSource object?


 Thanks,

 Shen


>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

git commit message

2017-04-30 Thread Ted Yu 
Hi,
When I went over git commit messages, I found some without either of the
following:

JIRA number
PR number

It would be nice for other people to get background / goal / discussion
w.r.t. any commit if either (or both) of the above is present.

My two cents.

Jenkins build is unstable: beam_Release_NightlySnapshot #400

2017-04-30 Thread Apache Jenkins Server 
See