Re: Switching to Java 8

[Jean-Baptiste Onofré]

Hi

I created the PR about build during the weekend. I'm working on the examples 
merge PR and also polishing the first one. I will add you as reviewer.

Regards
JB

Le 16 janv. 2018 à 07:35, à 07:35, Eugene Kirpichov  a 
écrit:
>Hi JB - any updates here?
>
>On Tue, Jan 9, 2018, 2:51 AM Jean-Baptiste Onofré 
>wrote:
>
>> Actually, it's part of the build and I will "expand" the java version
>in
>> the
>> enforcer.
>>
>> Regards
>> JB
>>
>> On 01/09/2018 11:46 AM, Etienne Chauchot wrote:
>> > Hi,
>> >
>> > +1 as well, excellent news !
>> >
>> > I would add also: remove (AFAIK in some IOs) the enforcer
>configuration
>> (like
>> > [1]) that were put when java 8 was needed in a java 7 build.
>> >
>> > [1]
>> >
>> > 
>> > [1.8,)
>> > 
>> >
>> >
>> > Etienne
>> >
>> >
>> > Le 08/01/2018 à 14:02, Jean-Baptiste Onofré a écrit :
>> >> I created https://issues.apache.org/jira/browse/BEAM-3426 as
>umbrella
>> Jira and
>> >> created the sub-tasks related to build and examples.
>> >>
>> >> Feel free to add the relevant sub-tasks there.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 01/08/2018 11:33 AM, Ismaël Mejía wrote:
>> >>> Excellent news ! Probably a good idea to fill JIRAs to all of
>those. I
>> >>> would add:
>> >>>
>> >>> - Remove the references in the website to Java 7
>> >>> - Remove Java 7 and any related task from the CI
>> >>> - Update the docker dev build images (I will take this one since
>> >>> reproducible build is my pet project)
>> >>> - Upgrade the IOs who were still in older versions because of
>client
>> >>> compatibility. I remember SolfIO was one case but probably there
>are
>> >>> others.
>> >>>
>> >>>
>> >>> On Mon, Jan 8, 2018 at 7:49 AM, Jean-Baptiste Onofré
>
>> wrote:
>>  Yes, that's the plan: build first, example "merge" after.
>> 
>>  Regards
>>  JB
>> 
>>  On 01/08/2018 07:43 AM, Eugene Kirpichov wrote:
>> >
>> > Sounds great, thanks! Probably best done as 2 separate steps,
>because
>> > after updating the build scripts, everything else can begin in
>> parallel?
>> >
>> > On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré <
>> j...@nanthrax.net
>> > > wrote:
>> >
>> >  Hi Eugene,
>> >
>> >  I'm taking the build update: Maven/Gradle with enforcer +
>merge
>> of the
>> > examples
>> >  all together.
>> >
>> >  Regards
>> >  JB
>> >
>> >  On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:
>> >   > The vote on user@ about switching to Java 8 has
>concluded,
>> > affirmatively.
>> >   >
>> >   > What needs to be done to complete the switch? I can see
>at
>> least
>> > the
>> >  following:
>> >   > - Change maven and gradle scripts to use 1.8 source and
>> target
>> > version
>> >   > - Fix resulting compilation/test errors (Java8 has
>slightly
>> > different type
>> >   > checking, more minor issues may arise)
>> >   > - Remove all special-casing of java8 in build scripts
>> >   > - Merge all modules like "java8 examples" and "java8
>tests"
>> into
>> > respective
>> >   > non-"java8" modules
>> >   > - Organize an effort to modernize code to use Java 8
>> constructs
>> > where
>> >   > appropriate. Especially important to modernize
>examples. To
>> a large
>> >  extent this
>> >   > can probably be automated with an IDE.
>> >   >
>> >   > Anything else?
>> >   >
>> >
>> >  --
>> >  Jean-Baptiste Onofré
>> >  jbono...@apache.org 
>> >  http://blog.nanthrax.net
>> >  Talend - http://www.talend.com
>> >
>> 
>>  --
>>  Jean-Baptiste Onofré
>>  jbono...@apache.org
>>  http://blog.nanthrax.net
>>  Talend - http://www.talend.com
>> >>
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>

Re: Switching to Java 8

[Eugene Kirpichov]

Hi JB - any updates here?

On Tue, Jan 9, 2018, 2:51 AM Jean-Baptiste Onofré  wrote:

> Actually, it's part of the build and I will "expand" the java version in
> the
> enforcer.
>
> Regards
> JB
>
> On 01/09/2018 11:46 AM, Etienne Chauchot wrote:
> > Hi,
> >
> > +1 as well, excellent news !
> >
> > I would add also: remove (AFAIK in some IOs) the enforcer configuration
> (like
> > [1]) that were put when java 8 was needed in a java 7 build.
> >
> > [1]
> >
> > 
> > [1.8,)
> > 
> >
> >
> > Etienne
> >
> >
> > Le 08/01/2018 à 14:02, Jean-Baptiste Onofré a écrit :
> >> I created https://issues.apache.org/jira/browse/BEAM-3426 as umbrella
> Jira and
> >> created the sub-tasks related to build and examples.
> >>
> >> Feel free to add the relevant sub-tasks there.
> >>
> >> Regards
> >> JB
> >>
> >> On 01/08/2018 11:33 AM, Ismaël Mejía wrote:
> >>> Excellent news ! Probably a good idea to fill JIRAs to all of those. I
> >>> would add:
> >>>
> >>> - Remove the references in the website to Java 7
> >>> - Remove Java 7 and any related task from the CI
> >>> - Update the docker dev build images (I will take this one since
> >>> reproducible build is my pet project)
> >>> - Upgrade the IOs who were still in older versions because of client
> >>> compatibility. I remember SolfIO was one case but probably there are
> >>> others.
> >>>
> >>>
> >>> On Mon, Jan 8, 2018 at 7:49 AM, Jean-Baptiste Onofré 
> wrote:
>  Yes, that's the plan: build first, example "merge" after.
> 
>  Regards
>  JB
> 
>  On 01/08/2018 07:43 AM, Eugene Kirpichov wrote:
> >
> > Sounds great, thanks! Probably best done as 2 separate steps, because
> > after updating the build scripts, everything else can begin in
> parallel?
> >
> > On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré <
> j...@nanthrax.net
> > > wrote:
> >
> >  Hi Eugene,
> >
> >  I'm taking the build update: Maven/Gradle with enforcer + merge
> of the
> > examples
> >  all together.
> >
> >  Regards
> >  JB
> >
> >  On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:
> >   > The vote on user@ about switching to Java 8 has concluded,
> > affirmatively.
> >   >
> >   > What needs to be done to complete the switch? I can see at
> least
> > the
> >  following:
> >   > - Change maven and gradle scripts to use 1.8 source and
> target
> > version
> >   > - Fix resulting compilation/test errors (Java8 has slightly
> > different type
> >   > checking, more minor issues may arise)
> >   > - Remove all special-casing of java8 in build scripts
> >   > - Merge all modules like "java8 examples" and "java8 tests"
> into
> > respective
> >   > non-"java8" modules
> >   > - Organize an effort to modernize code to use Java 8
> constructs
> > where
> >   > appropriate. Especially important to modernize examples. To
> a large
> >  extent this
> >   > can probably be automated with an IDE.
> >   >
> >   > Anything else?
> >   >
> >
> >  --
> >  Jean-Baptiste Onofré
> >  jbono...@apache.org 
> >  http://blog.nanthrax.net
> >  Talend - http://www.talend.com
> >
> 
>  --
>  Jean-Baptiste Onofré
>  jbono...@apache.org
>  http://blog.nanthrax.net
>  Talend - http://www.talend.com
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5661

[Kenneth Knowles]

In case anyone is following, this is
https://issues.apache.org/jira/browse/BEAM-3438

Kenn

On Mon, Jan 15, 2018 at 5:08 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  MavenInstall/5661/display/redirect>
>
>

Re: [DISCUSS] State of the project: Community growth

[Austin Bennett]

Think that Beam's great, so happy to grow community -- by my individual
involvement and to encourage others.

1). Users:  I think that this can follow from the others being done well
2).  Contributors: some other projects mark very simple contributions for
newcomers to find as onramping.  Hadn't noticed that here in the little I
had explored.
3-5). Community/Event/Brand:  Beam is inherently collaborative given its
model and what I understand of purpose (different runners, batch/stream,
etc).  An easy place to start will be places that play very well and/or
integrate with.



On Mon, Jan 15, 2018 at 3:27 PM, Griselda Cuevas  wrote:

>
>
>
>
>
>
>
> *Hi Everyone, Thanks Davor for starting the discussion around the state of
> the Apache Beam in 2017[1]. In this fork of that conversation, I’d like to
> continue the dialogue around how should we grow our community in alignment
> to the project's vision, goals & values.Here are some areas and prompts to
> guide the conversation. If there are other ideas related to community
> growth missing here, please add them in this thread as well. 1. New users:
> How could the community help potential and new users learn about Beam? How
> can we support questions from users better?2. New contributors: What are
> our contributions Best Practices and how can we share them with new
> members? How could the community help new contributors ramp up faster in
> our Project’s dev environment? How could we empower new developers to
> contribute to Beam’s core features? 3. Community engagement: What type of
> activities and efforts would you like to see to increase our community
> engagement?4. Events: What events should we attend? What events should we
> sponsor? How could we support community led events better?5. Brand
> building: What efforts do you think we should do to build our brand? What
> use cases, ideas, talks, etc. should we collaborate on to give the project
> more visibility?If there’s interest, I’d also be happy to host a virtual
> meeting for anyone who’d prefer that avenue of discussion and will make
> sure that any new ideas or details are brought back to the discussion
> threads after that. Express interest in a virtual meeting in this thread so
> I can coordinate.Thanks, and let’s make this an exciting community!Gris
> Cuevas[1]https://lists.apache.org/thread.html/f750f288af8dab3f468b869bf5a3f473094f4764db419567f33805d0@%3Cdev.beam.apache.org%3E
> *
>

[DISCUSS] State of the project: Community growth

[Griselda Cuevas]

*Hi Everyone, Thanks Davor for starting the discussion around the state of
the Apache Beam in 2017[1]. In this fork of that conversation, I’d like to
continue the dialogue around how should we grow our community in alignment
to the project's vision, goals & values.Here are some areas and prompts to
guide the conversation. If there are other ideas related to community
growth missing here, please add them in this thread as well. 1. New users:
How could the community help potential and new users learn about Beam? How
can we support questions from users better?2. New contributors: What are
our contributions Best Practices and how can we share them with new
members? How could the community help new contributors ramp up faster in
our Project’s dev environment? How could we empower new developers to
contribute to Beam’s core features? 3. Community engagement: What type of
activities and efforts would you like to see to increase our community
engagement?4. Events: What events should we attend? What events should we
sponsor? How could we support community led events better?5. Brand
building: What efforts do you think we should do to build our brand? What
use cases, ideas, talks, etc. should we collaborate on to give the project
more visibility?If there’s interest, I’d also be happy to host a virtual
meeting for anyone who’d prefer that avenue of discussion and will make
sure that any new ideas or details are brought back to the discussion
threads after that. Express interest in a virtual meeting in this thread so
I can coordinate.Thanks, and let’s make this an exciting community!Gris
Cuevas[1]https://lists.apache.org/thread.html/f750f288af8dab3f468b869bf5a3f473094f4764db419567f33805d0@%3Cdev.beam.apache.org%3E
*

Re: [DISCUSS] State of the project

[Jesse Anderson]

I think a focus on the runners is what's key to Beam's adoption. The
runners are the foundation on which Beam sits. If the runners don't work
properly, Beam won't work.

A focus on improved unit tests is a good start, but isn't what's needed.
Compatibility matrices will help see how your runner of choice stacks up,
but that requires too much knowledge of Beam's internals to be
interpretable.

Imagine you're an (enterprise) architect looking at adopting Beam. What do
you look at or what do you look for before going deeper? What would make
you stick your neck out to adopt Beam? For my experience, there are
several/pass fails along the way.

Here are a few of the common ones I've seen:

   - Will this dramatically improve the problems I'm trying to solve? (not
   writing APIs/better programming model/Beam's better handling of windowing)
   - Can I get commercial support for Beam? (This is changing soon)
   - Are other people using Beam with the configuration and use case as me?
   (e.g. I'm using Spark with Beam to process imagery. Are others doing this
   in production?)
   - Is there good documentation and books on the subject? (Tyler's and
   others' book will improve this)
   - Can I get my team trained on this new technology? (I have Beam
   training and Google has some cursory training)

I think the one the community can improve on the most is the social proof
of Beam. I've tried to do this (
http://www.jesse-anderson.com/2017/06/beam-2-0-q-and-a/ and
http://www.jesse-anderson.com/2016/07/question-and-answers-with-the-apache-beam-team/).
We need to get the message out more about people using Beam in production,
which configuration they have, and what their results were. I think we have
the social proof on Dataflow, but not as much on Spark/Flink/Apex.

I think it's important to note that these checks don't look at the hardcore
language or API semantics that we're working on. These are much later stage
issues, if they're ever used at all.

In my experience with other open source adoption at enterprises, it starts
with architects and works its way around the organization from there.

Thanks,

Jesse

On Mon, Jan 15, 2018 at 8:14 AM Ted Yu  wrote:

> bq. are hard to detect in our unit-test framework
>
> Looks like more integration tests would help discover bug / regression
> more quickly. If committer reviewing the PR has concern in this regard, the
> concern should be stated on the PR so that the contributor (and reviewer)
> can spend more time in solidifying the solution.
>
> bq. I've gone and fixed these issues myself when merging
>
> We can make stricter checkstyle rules so that the code wouldn't pass build
> without addressing commonly known issues.
>
> Cheers
>
> On Sun, Jan 14, 2018 at 12:37 PM, Reuven Lax  wrote:
>
>> I agree with the sentiment, but I don't completely agree with the
>> criteria.
>>
>> I think we need to be much better about reviewing PRs. Some PRs languish
>> for too long before the reviewer gets to it (and I've been guilty of this
>> too), which does not send a good message. Also new PRs sometimes languish
>> because there is no reviewer assigned; maybe we could write a gitbot to
>> automatically assign a reviewer to every new PR?
>>
>> Also, I think that the bar for merging a PR from a contributor should not
>> be "the PR is perfect." It's perfectly fine to merge a PR that still has
>> some issues (especially if the issues are stylistic). In the past when I've
>> done this, I've gone and fixed these issues myself when merging. It was a
>> bit more work for me to fix these things myself, but it was a small price
>> to pay in order to portray Beam as a welcoming place for contributions.
>>
>> On the other hand, "the build does not break" is - in my opinion - too
>> weak of a criterion for merging. A few reasons for this:
>>
>>   * Beam is a data-processing framework, and data integrity is paramount.
>> If a reviewer sees an issue that could lead to data loss (or duplication,
>> or corruption), I don't think that PR should be merged. Historically many
>> such issues only actually manifest at scale, and are hard to detect in our
>> unit-test framework. (we also need to invest in more at-scale tests to
>> catch such issues).
>>
>>   * Beam guarantees backwards compatibility for users (except across
>> major versions). If a bad API gets merged and released (and the chances of
>> "forgetting" about it before the release is cut is unfortunately high), we
>> are stuck with it. This is less of an issue for many other open-source
>> projects that do not make such a compatibility guarantee, as they are able
>> to simply remove or fix the API in the next version.
>>
>> I think we still need honest review of PRs, with the criteria being
>> stronger than "the build doesn't break." However reviewers also need to be
>> reasonable about what they ask for.
>>
>> Reuven
>>
>> On Sun, Jan 14, 2018 at 11:19 AM, Ted Yu  wrote:
>>
>>> bq. if a 

Re: [DISCUSS] State of the project

[Ted Yu]

bq. are hard to detect in our unit-test framework

Looks like more integration tests would help discover bug / regression more
quickly. If committer reviewing the PR has concern in this regard, the
concern should be stated on the PR so that the contributor (and reviewer)
can spend more time in solidifying the solution.

bq. I've gone and fixed these issues myself when merging

We can make stricter checkstyle rules so that the code wouldn't pass build
without addressing commonly known issues.

Cheers

On Sun, Jan 14, 2018 at 12:37 PM, Reuven Lax  wrote:

> I agree with the sentiment, but I don't completely agree with the
> criteria.
>
> I think we need to be much better about reviewing PRs. Some PRs languish
> for too long before the reviewer gets to it (and I've been guilty of this
> too), which does not send a good message. Also new PRs sometimes languish
> because there is no reviewer assigned; maybe we could write a gitbot to
> automatically assign a reviewer to every new PR?
>
> Also, I think that the bar for merging a PR from a contributor should not
> be "the PR is perfect." It's perfectly fine to merge a PR that still has
> some issues (especially if the issues are stylistic). In the past when I've
> done this, I've gone and fixed these issues myself when merging. It was a
> bit more work for me to fix these things myself, but it was a small price
> to pay in order to portray Beam as a welcoming place for contributions.
>
> On the other hand, "the build does not break" is - in my opinion - too
> weak of a criterion for merging. A few reasons for this:
>
>   * Beam is a data-processing framework, and data integrity is paramount.
> If a reviewer sees an issue that could lead to data loss (or duplication,
> or corruption), I don't think that PR should be merged. Historically many
> such issues only actually manifest at scale, and are hard to detect in our
> unit-test framework. (we also need to invest in more at-scale tests to
> catch such issues).
>
>   * Beam guarantees backwards compatibility for users (except across major
> versions). If a bad API gets merged and released (and the chances of
> "forgetting" about it before the release is cut is unfortunately high), we
> are stuck with it. This is less of an issue for many other open-source
> projects that do not make such a compatibility guarantee, as they are able
> to simply remove or fix the API in the next version.
>
> I think we still need honest review of PRs, with the criteria being
> stronger than "the build doesn't break." However reviewers also need to be
> reasonable about what they ask for.
>
> Reuven
>
> On Sun, Jan 14, 2018 at 11:19 AM, Ted Yu  wrote:
>
>> bq. if a PR is basically right (it does what it should) without breaking
>> the build, then it has to be merged fast
>>
>> +1 on above.
>> This would give contributors positive feedback.
>>
>> On Sun, Jan 14, 2018 at 8:13 AM, Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi Davor,
>>>
>>> Thanks a lot for this e-mail.
>>>
>>> I would like to emphasize two areas where we have to improve:
>>>
>>> 1. Apache way and community. We still have to focus and being dedicated
>>> on our communities (both user & dev). Helping, encouraging, growing our
>>> communities is key for the project. Building bridges between communities is
>>> also very important. We have to be more "accessible": sometime simplifying
>>> our discussions, showing more interest and open minded in the proposals
>>> would help as well. I think we do a good job already: we just have to
>>> improve.
>>>
>>> 2. Execution: a successful project is a project with a regular activity
>>> in term of releases, fixes, improvements.
>>> Regarding the PR, I think today we have a PR opened for long. And I
>>> think for three reasons:
>>> - some are not ready, not good enough, no question on these ones
>>> - some needs reviewer and speed up: we have to be careful on the open
>>> PRs and review asap
>>> - some are under review but we have a lot of "ping pong" and long
>>> discussion, not always justified. I already said that on the mailing list
>>> but, as for other Apache projects, if a PR is basically right (it does what
>>> it should) without breaking the build, then it has to be merged fast. If it
>>> requires additional changes (tests, polishing, improvements, ...), then it
>>> can be addressed in new PRs.
>>> As already mentioned in the Beam 2.3.0 thread, we have to adopt a
>>> regular schedule for releases. It's a best effort to have a release every 2
>>> months, whatever the release will contain. That's essential to maintain a
>>> good activity in the project and for the third party projects using Beam.
>>>
>>> Again, don't get me wrong: we already do a good job ! It's just area
>>> where I think we have to improve.
>>>
>>> Anyway, thanks for all the hard work we are doing all together !
>>>
>>> Regards
>>> JB
>>>
>>>
>>> On 13/01/2018 05:12, Davor Bonaci wrote:
>>>
 Hi everyone 

Jenkins build is back to normal : beam_SeedJob #895

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_SeedJob #894

[Apache Jenkins Server]

See 

--
GitHub pull request #4416 of commit 0855d1581e8e08c9c3024a1e8b7ad071c3848b17, 
no merge conflicts.
Setting status of 0855d1581e8e08c9c3024a1e8b7ad071c3848b17 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/894/ and message: 'Build started 
sha1 is merged.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4416/*:refs/remotes/origin/pr/4416/*
 > git rev-parse refs/remotes/origin/pr/4416/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4416/merge^{commit} # timeout=10
Checking out Revision f674a61c1ad9a3e973804d6682d8790fa06b5bfc 
(refs/remotes/origin/pr/4416/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f674a61c1ad9a3e973804d6682d8790fa06b5bfc
Commit message: "Merge 0855d1581e8e08c9c3024a1e8b7ad071c3848b17 into 
0012ab1d19287421b55468122c806a51360de31a"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_beam_Java_Build.groovy
Processing DSL script job_beam_Java_CodeHealth.groovy
Processing DSL script job_beam_Java_IntegrationTest.groovy
Processing DSL script job_beam_Java_UnitTest.groovy
Processing DSL script job_beam_PerformanceTests_Dataflow.groovy
ERROR: (common_job_properties.groovy, line 274) No signature of method: static 
common_job_properties.parameters() is applicable for argument types: 
(common_job_properties$_buildPerformanceTest_closure11) values: 
[common_job_properties$_buildPerformanceTest_closure11@124a273b]

Build failed in Jenkins: beam_Release_NightlySnapshot #653

[Apache Jenkins Server]

See 


--
[...truncated 3.07 MB...]
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.hk2:hk2-locator:jar:2.4.0-b34 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.jersey.core:jersey-common:jar:2.22.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
javax.annotation:javax.annotation-api:jar:1.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2 from the shaded 
jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.hk2:osgi-resource-locator:jar:1.0.1 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.jersey.core:jersey-server:jar:2.22.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2 from the 
shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2 from 
the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding io.netty:netty-all:jar:4.0.43.Final 
from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding io.netty:netty:jar:3.9.9.Final from 
the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
io.dropwizard.metrics:metrics-json:jar:3.1.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
io.dropwizard.metrics:metrics-graphite:jar:3.1.2 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding org.apache.ivy:ivy:jar:2.4.0 from the 
shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding net.razorvine:pyrolite:jar:4.13 from 
the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding net.sf.py4j:py4j:jar:0.10.4 from the 
shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.apache.spark:spark-tags_2.11:jar:2.2.1 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.apache.commons:commons-crypto:jar:1.0.0 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
2018-01-15T08:30:17.227 [INFO] Excluding 
org.apache.spark:spark-streaming_2.11:jar:2.2.1 from the shaded jar.
2018-01-15T08:30:18.916 [INFO] Replacing original artifact with shaded artifact.
2018-01-15T08:30:19.022 [INFO] 
2018-01-15T08:30:19.022 [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar 
(attach-javadocs) @ beam-sdks-java-javadoc ---
2018-01-15T08:30:19.026 [INFO] Not executing Javadoc as the project is not a 
Java classpath-capable package
2018-01-15T08:30:19.133 [INFO] 
2018-01-15T08:30:19.133 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-javadoc ---
2018-01-15T08:30:19.242 [INFO] 
2018-01-15T08:30:19.242 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-javadoc ---
2018-01-15T08:30:19.349 [INFO] 
2018-01-15T08:30:19.349 [INFO] --- 
reproducible-build-maven-plugin:0.4:strip-jar (default) @ 
beam-sdks-java-javadoc ---
2018-01-15T08:30:19.350 [INFO] Stripping 

2018-01-15T08:30:19.517 [INFO] 
2018-01-15T08:30:19.517 [INFO] --- maven-dependency-plugin:3.0.1:analyze-only 
(default) @ beam-sdks-java-javadoc ---
2018-01-15T08:30:19.519 [INFO] Skipping plugin execution
2018-01-15T08:30:19.625 [INFO] 
2018-01-15T08:30:19.625 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-sdks-java-javadoc ---
2018-01-15T08:30:19.626 [INFO] Installing 

 to 

2018-01-15T08:30:19.628 [INFO] Installing 

 to 

2018-01-15T08:30:19.760 [INFO] 
2018-01-15T08:30:19.760 [INFO] --- maven-deploy-plugin:2.8.2:deploy 
(default-deploy) @ beam-sdks-java-javadoc ---
2018-01-15T08:30:19.763 [INFO] Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-javadoc/2.3.0-SNAPSHOT/maven-metadata.xml
2018-01-15T08:30:19.998 [INFO] Downloaded: 

bigquery issue

[Chaim Turkel]

Hi,
  I have a fairly simple pipeline that create daily snapshots of my
data, and it sometimes fails, but the reason is not obvious:


(863777e448a29a5c): Workflow failed. Causes: (863777e448a298ff):
S41:Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Read+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionsReshuffle/GroupByKey/GroupByWindow+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionsReshuffle/ExpandIterable+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables)+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/extract
table name 
+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/GroupByKey+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/Combine.GroupedValues/Partial+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/GroupByKey/Reify+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/GroupByKey/Write+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/WithKeys/AddKeys/Map+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/Window.Into()/Window.Assign+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/GroupByKey/Reify+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/GroupByKey/Write
failed., (a412b0e93a586d57): A work item was attempted 4 times without
success. Each time the worker eventually lost contact with the
service. The work item was attempted on:
dailysnapshotoptions-chai-01142358-ef6c-harness-qw6s,
dailysnapshotoptions-chai-01142358-ef6c-harness-j09b,
dailysnapshotoptions-chai-01142358-ef6c-harness-3t9m,
dailysnapshotoptions-chai-01142358-ef6c-harness-372t

is there any way to get more information?

the job is:


https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-01-14_23_58_54-6125672650598375925?project=ordinal-ember-163410=782381653268


chaim

-- 


Loans are funded by FinWise Bank, a Utah-chartered bank located in Sandy, 
Utah, member FDIC, Equal Opportunity Lender. Merchant Cash Advances are 
made by Behalf. For more information on ECOA, click here 
. For important information about 
opening a new account, review Patriot Act procedures here 
. Visit Legal 
 to review our comprehensive program terms, 
conditions, and disclosures.