Re: Java precommit and postcommit tests are failing

[Rui Wang]

It is at least frequent (and event consistent) based on the list of failed
PR which run java checks.


Also, this issue is being tracked by
https://issues.apache.org/jira/browse/BEAM-4847.


-Rui

On Mon, Jul 23, 2018 at 9:55 PM Jean-Baptiste Onofré 
wrote:

> Hi Rui
>
> I will. I guess it randomly happens, right ?
>
> Regards
> JB
> Le 24 juil. 2018, à 02:37, Rui Wang  a écrit:
>>
>> Hi community,
>>
>> Seems like both Java precommit and postcommit tests are failing due to GC
>> issues. I created a JIRA under test-failures component (
>> https://issues.apache.org/jira/browse/BEAM-4848). Could someone who is
>> familiar with Jenkins take a look at this?
>>
>>
>> Thanks,
>> Rui
>>
>

Re: Java precommit and postcommit tests are failing

[Jean-Baptiste Onofré]

Hi Rui

I will. I guess it randomly happens, right ?

Regards
JB

Le 24 juil. 2018 à 02:37, à 02:37, Rui Wang  a écrit:
>Hi community,
>
>Seems like both Java precommit and postcommit tests are failing due to
>GC
>issues. I created a JIRA under test-failures component (
>https://issues.apache.org/jira/browse/BEAM-4848). Could someone who is
>familiar with Jenkins take a look at this?
>
>
>Thanks,
>Rui

Re: CODEOWNERS for apache/beam repo

[Jean-Baptiste Onofré]

It looks interesting but I would like to see the complete video and explanation 
about prow. Especially what we concretely need.

Regards
JB

Le 24 juil. 2018 à 04:17, à 04:17, Udi Meiri  a écrit:
>I was recently told about Prow
>, which
>automates testing and merging for Kubernetes and other projects.
>It also automates assigning reviewers and suggesting approvers. Example
> PR, video
>explanation
>
>I propose trying out Prow, since is it's a maintained and it uses
>OWNERS
>files to explicitly define both who should be reviewing and who should
>approve a PR.
>
>I'm not suggesting we use it to replace Jenkins or do our merges for
>us.
>
>
>On Tue, Jul 17, 2018 at 11:04 AM Udi Meiri  wrote:
>
>> +1 to generating the file.
>> I'll go ahead and file a PR to remove CODEOWNERS
>>
>> On Tue, Jul 17, 2018 at 9:28 AM Holden Karau 
>wrote:
>>
>>> So it doesn’t support doing that right now, although if we find it’s
>a
>>> problem we can specify an exclude file with folks who haven’t
>contributed
>>> in the past year. Would people want me to generate that first?
>>>
>>> On Tue, Jul 17, 2018 at 10:22 AM Ismaël Mejía 
>wrote:
>>>
 Is there a way to put inactive people as not reviewers for the
>blame
 case? I think it can be useful considering that a good amount of
>our
 committers are not active at the moment and auto-assigning reviews
>to
 them seem like a waste of energy/time.
 On Tue, Jul 17, 2018 at 1:58 AM Eugene Kirpichov
>
 wrote:
 >
 > We did not, but I think we should. So far, in 100% of the PRs
>I've
 authored, the default functionality of CODEOWNERS did the wrong
>thing and I
 had to fix something up manually.
 >
 > On Mon, Jul 16, 2018 at 3:42 PM Andrew Pilloud
>
 wrote:
 >>
 >> This sounds like a good plan. Did we want to rename the
>CODEOWNERS
 file to disable github's mass adding of reviewers while we figure
>this out?
 >>
 >> Andrew
 >>
 >> On Mon, Jul 16, 2018 at 10:20 AM Jean-Baptiste Onofré <
 j...@nanthrax.net> wrote:
 >>>
 >>> +1
 >>>
 >>> Le 16 juil. 2018, à 19:17, Holden Karau
> a
 écrit:
 
  Ok if no one objects I'll create the INFRA ticket after OSCON
>and
 we can test it for a week and decide if it helps or hinders.
 
  On Mon, Jul 16, 2018, 7:12 PM Jean-Baptiste Onofré <
 j...@nanthrax.net> wrote:
 >
 > Agree to test it for a week.
 >
 > Regards
 > JB
 > Le 16 juil. 2018, à 18:59, Holden Karau <
>holden.ka...@gmail.com>
 a écrit:
 >>
 >> Would folks be OK with me asking infra to turn on blame
>based
 suggestions for Beam and trying it out for a week?
 >>
 >> On Mon, Jul 16, 2018, 6:53 PM Rafael Fernandez <
 rfern...@google.com> wrote:
 >>>
 >>> +1 using blame -- nifty :)
 >>>
 >>> On Mon, Jul 16, 2018 at 2:31 AM Huygaa Batsaikhan <
 bat...@google.com> wrote:
 
  +1. This is great.
 
  On Sat, Jul 14, 2018 at 7:44 AM Udi Meiri <
>eh...@google.com>
 wrote:
 >
 > Mention bot looks cool, as it tries to guess the reviewer
 using blame.
 > I've written a quick and dirty script that uses only
 CODEOWNERS.
 >
 > Its output looks like:
 > $ python suggest_reviewers.py --pr 5940
 > INFO:root:Selected reviewer @lukecwik for:

>/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
 (path_pattern: /runners/core-construction-java*)
 > INFO:root:Selected reviewer @lukecwik for:

>/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
 (path_pattern: /runners/core-construction-java*)
 > INFO:root:Selected reviewer @echauchot for:

>/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java
 (path_pattern: /runners/core-java*)
 > INFO:root:Selected reviewer @lukecwik for:
 /runners/flink/build.gradle (path_pattern: */build.gradle*)
 > INFO:root:Selected reviewer @lukecwik for:

>/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java
 (path_pattern: *.java)
 > INFO:root:Selected reviewer @pabloem for:

>/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 (path_pattern: /runners/google-cloud-dataflow-java*)
 > INFO:root:Selected reviewer @lukecwik for:

>/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
 (path_pattern: /sdks/java/core*)
 > 

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

I was recently told about Prow
, which
automates testing and merging for Kubernetes and other projects.
It also automates assigning reviewers and suggesting approvers. Example
 PR, video explanation

I propose trying out Prow, since is it's a maintained and it uses OWNERS
files to explicitly define both who should be reviewing and who should
approve a PR.

I'm not suggesting we use it to replace Jenkins or do our merges for us.


On Tue, Jul 17, 2018 at 11:04 AM Udi Meiri  wrote:

> +1 to generating the file.
> I'll go ahead and file a PR to remove CODEOWNERS
>
> On Tue, Jul 17, 2018 at 9:28 AM Holden Karau  wrote:
>
>> So it doesn’t support doing that right now, although if we find it’s a
>> problem we can specify an exclude file with folks who haven’t contributed
>> in the past year. Would people want me to generate that first?
>>
>> On Tue, Jul 17, 2018 at 10:22 AM Ismaël Mejía  wrote:
>>
>>> Is there a way to put inactive people as not reviewers for the blame
>>> case? I think it can be useful considering that a good amount of our
>>> committers are not active at the moment and auto-assigning reviews to
>>> them seem like a waste of energy/time.
>>> On Tue, Jul 17, 2018 at 1:58 AM Eugene Kirpichov 
>>> wrote:
>>> >
>>> > We did not, but I think we should. So far, in 100% of the PRs I've
>>> authored, the default functionality of CODEOWNERS did the wrong thing and I
>>> had to fix something up manually.
>>> >
>>> > On Mon, Jul 16, 2018 at 3:42 PM Andrew Pilloud 
>>> wrote:
>>> >>
>>> >> This sounds like a good plan. Did we want to rename the CODEOWNERS
>>> file to disable github's mass adding of reviewers while we figure this out?
>>> >>
>>> >> Andrew
>>> >>
>>> >> On Mon, Jul 16, 2018 at 10:20 AM Jean-Baptiste Onofré <
>>> j...@nanthrax.net> wrote:
>>> >>>
>>> >>> +1
>>> >>>
>>> >>> Le 16 juil. 2018, à 19:17, Holden Karau  a
>>> écrit:
>>> 
>>>  Ok if no one objects I'll create the INFRA ticket after OSCON and
>>> we can test it for a week and decide if it helps or hinders.
>>> 
>>>  On Mon, Jul 16, 2018, 7:12 PM Jean-Baptiste Onofré <
>>> j...@nanthrax.net> wrote:
>>> >
>>> > Agree to test it for a week.
>>> >
>>> > Regards
>>> > JB
>>> > Le 16 juil. 2018, à 18:59, Holden Karau < holden.ka...@gmail.com>
>>> a écrit:
>>> >>
>>> >> Would folks be OK with me asking infra to turn on blame based
>>> suggestions for Beam and trying it out for a week?
>>> >>
>>> >> On Mon, Jul 16, 2018, 6:53 PM Rafael Fernandez <
>>> rfern...@google.com> wrote:
>>> >>>
>>> >>> +1 using blame -- nifty :)
>>> >>>
>>> >>> On Mon, Jul 16, 2018 at 2:31 AM Huygaa Batsaikhan <
>>> bat...@google.com> wrote:
>>> 
>>>  +1. This is great.
>>> 
>>>  On Sat, Jul 14, 2018 at 7:44 AM Udi Meiri < eh...@google.com>
>>> wrote:
>>> >
>>> > Mention bot looks cool, as it tries to guess the reviewer
>>> using blame.
>>> > I've written a quick and dirty script that uses only
>>> CODEOWNERS.
>>> >
>>> > Its output looks like:
>>> > $ python suggest_reviewers.py --pr 5940
>>> > INFO:root:Selected reviewer @lukecwik for:
>>> /runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
>>> (path_pattern: /runners/core-construction-java*)
>>> > INFO:root:Selected reviewer @lukecwik for:
>>> /runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
>>> (path_pattern: /runners/core-construction-java*)
>>> > INFO:root:Selected reviewer @echauchot for:
>>> /runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java
>>> (path_pattern: /runners/core-java*)
>>> > INFO:root:Selected reviewer @lukecwik for:
>>> /runners/flink/build.gradle (path_pattern: */build.gradle*)
>>> > INFO:root:Selected reviewer @lukecwik for:
>>> /runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java
>>> (path_pattern: *.java)
>>> > INFO:root:Selected reviewer @pabloem for:
>>> /runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
>>> (path_pattern: /runners/google-cloud-dataflow-java*)
>>> > INFO:root:Selected reviewer @lukecwik for:
>>> /sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
>>> (path_pattern: /sdks/java/core*)
>>> > Suggested reviewers: @echauchot, @lukecwik, @pabloem
>>> >
>>> > Script is in: https://github.com/apache/beam/pull/5951
>>> >
>>> >
>>> > What does the community think? Do you prefer blame-based or
>>> rules-based reviewer suggestions?
>>> >
>>> > On Fri, Jul 13, 2018 at 11:13 AM 

Java precommit and postcommit tests are failing

[Rui Wang]

Hi community,

Seems like both Java precommit and postcommit tests are failing due to GC
issues. I created a JIRA under test-failures component (
https://issues.apache.org/jira/browse/BEAM-4848). Could someone who is
familiar with Jenkins take a look at this?


Thanks,
Rui

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #112

[Apache Jenkins Server]

See 


--
[...truncated 17.45 MB...]
Task ':beam-sdks-python-container:buildLinuxAmd64' is not up-to-date because:
  No history is available.
:beam-sdks-python-container:buildLinuxAmd64 (Thread[Task worker for ':' Thread 
44,5,main]) completed. Took 2.698 secs.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 
44,5,main]) started.

> Task :beam-sdks-python-container:build
Caching disabled for task ':beam-sdks-python-container:build': Caching has not 
been enabled for the task
Task ':beam-sdks-python-container:build' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 
44,5,main]) completed. Took 0.002 secs.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 44,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:jar
Build cache key for task ':beam-vendor-sdks-java-extensions-protobuf:jar' is 
318184f3acfe150a3a38d605826c992a
Caching disabled for task ':beam-vendor-sdks-java-extensions-protobuf:jar': 
Caching has not been enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:jar' is not up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 44,5,main]) completed. Took 0.007 secs.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 44,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:compileTestJava' as 
it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 44,5,main]) completed. Took 0.001 secs.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 44,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:processTestResources' 
as it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 44,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 3,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:testClasses UP-TO-DATE
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:testClasses' as it 
has no actions.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 3,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 3,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:packageTests
Build cache key for task 
':beam-vendor-sdks-java-extensions-protobuf:packageTests' is 
725c57f0b1c0d6e4c572de9d9db06024
Caching disabled for task 
':beam-vendor-sdks-java-extensions-protobuf:packageTests': Caching has not been 
enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:packageTests' is not 
up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 3,5,main]) completed. Took 0.003 secs.
:beam-vendor-sdks-java-extensions-protobuf:assemble (Thread[Task worker for ':' 
Thread 3,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:assemble
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:assemble' as it has 
no actions.
:beam-vendor-sdks-java-extensions-protobuf:assemble (Thread[Task worker for ':' 
Thread 3,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleMain (Thread[Task worker 
for ':' Thread 3,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:checkstyleMain NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:checkstyleMain' as it 
has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleMain (Thread[Task worker 
for ':' Thread 3,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleTest (Thread[Task worker 
for ':' Thread 31,5,main]) started.

> Task 

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #111

[Apache Jenkins Server]

See 


--
[...truncated 18.34 MB...]
Task ':beam-sdks-python-container:buildLinuxAmd64' is not up-to-date because:
  No history is available.
:beam-sdks-python-container:buildLinuxAmd64 (Thread[Task worker for ':' Thread 
24,5,main]) completed. Took 2.79 secs.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 
24,5,main]) started.

> Task :beam-sdks-python-container:build
Caching disabled for task ':beam-sdks-python-container:build': Caching has not 
been enabled for the task
Task ':beam-sdks-python-container:build' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 
24,5,main]) completed. Took 0.001 secs.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 24,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:jar
Build cache key for task ':beam-vendor-sdks-java-extensions-protobuf:jar' is 
318184f3acfe150a3a38d605826c992a
Caching disabled for task ':beam-vendor-sdks-java-extensions-protobuf:jar': 
Caching has not been enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:jar' is not up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 24,5,main]) completed. Took 0.009 secs.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 24,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:compileTestJava' as 
it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 24,5,main]) completed. Took 0.001 secs.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 18,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:processTestResources' 
as it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 18,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 18,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:testClasses UP-TO-DATE
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:testClasses' as it 
has no actions.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 18,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 39,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:packageTests
Build cache key for task 
':beam-vendor-sdks-java-extensions-protobuf:packageTests' is 
725c57f0b1c0d6e4c572de9d9db06024
Caching disabled for task 
':beam-vendor-sdks-java-extensions-protobuf:packageTests': Caching has not been 
enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:packageTests' is not 
up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 39,5,main]) completed. Took 0.003 secs.
:beam-vendor-sdks-java-extensions-protobuf:assemble (Thread[Task worker for ':' 
Thread 39,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:assemble
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:assemble' as it has 
no actions.
:beam-vendor-sdks-java-extensions-protobuf:assemble (Thread[Task worker for ':' 
Thread 39,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleMain (Thread[Task worker 
for ':' Thread 39,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:checkstyleMain NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:checkstyleMain' as it 
has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleMain (Thread[Task worker 
for ':' Thread 39,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleTest (Thread[Task worker 
for ':' Thread 39,5,main]) started.

> Task 

Re: SQS source

[Raghu Angadi]

On Mon, Jul 23, 2018 at 2:25 PM John Rudolf Lewis 
wrote:

> So I guess I can add a timestamp to the message attributes when i receive
> it from SQS since there is no such built in property.
> But what triggers finilizeCheckpoint to be called? So far in my testing, I
> never see that method get called, and hence, my messages keep getting
> redelivered.
>

Are you testing with direct runner? It should be called after first stage
processes (i.e. the checkpoint mark is durably committed by the runner).

Raghu.


>
>
> On Thu, Jul 19, 2018 at 5:26 PM, Raghu Angadi  wrote:
>
>> A timestamp for a message is fundamental to an element in a PCollection.
>> What do you mean by not knowing timestamp of a message?
>> There is finalizeCheckpoint API[1] in UnboundedSource. Does that help?
>> PubSub is also very similar, a message need to be acked with in a timeout,
>> otherwise it will be redelivered to one of the consumer. Pubsub messages
>> are acked inside finalize().
>>
>> [1]:
>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L129
>>
>>
>> On Thu, Jul 19, 2018 at 3:28 PM John Rudolf Lewis 
>> wrote:
>>
>>> hmm... made lots of progress on this today. But need help understanding
>>> something
>>>
>>> UnboundedSource seems to assume that there is some guarantee of message
>>> ordering, and that you can get the timestamp of the current message. Using
>>> UnboundedSource.CheckpointMark to help advance the offset. Seems to work ok
>>> for any source that supports those assumptions. But SQS does not work this
>>> way.
>>>
>>> With a standard SQS queue, there is no guarantee of ordering and there
>>> is no timestamp for a message.  With SQS, one needs to call the delete api
>>> using the receipt handle from the message to acknowledge receipt of a
>>> message and prevent its redelivery after the visibility timeout has expired.
>>>
>>> I'm not sure how to adapt these two patterns and would welcome
>>> suggestions.
>>>
>>>
>>>
>>> On Thu, Jul 19, 2018 at 7:40 AM, Jean-Baptiste Onofré 
>>> wrote:
>>>
 Thx John !

 Regards
 JB

 On 19/07/2018 16:39, John Rudolf Lewis wrote:
 > Thank you.
 >
 > I've created a jira ticket to add SQS and have assigned it to
 > myself: https://issues.apache.org/jira/browse/BEAM-4828
 >
 > Modified the documentation to show it as in-progress:
 > https://github.com/apache/beam/pull/5995
 >
 > And will be starting my work
 > here: https://github.com/JohnRudolfLewis/beam/tree/Add-SqsIO
 >
 > On Thu, Jul 19, 2018 at 1:43 AM, Jean-Baptiste Onofré <
 j...@nanthrax.net
 > > wrote:
 >
 > Agree with Ismaël.
 >
 > I would be more than happy to help on this one (as I contributed
 on AMQP
 > and JMS IOs ;)).
 >
 > Regards
 > JB
 >
 > On 19/07/2018 10:39, Ismaël Mejía wrote:
 > > Thanks for your interest John, it would be a really nice
 contribution
 > > to add SQS support.
 > >
 > > Some context on the kinesis stuff:
 > >
 > > The reason why kinesis is still in a separate module is more
 related
 > > to a licensing problem. Kinesis uses some native libraries that
 are
 > > published under a not 100% apache compatible license and we are
 not
 > > allowed to shade and republish them but it seems there is a
 workaround
 > > now, for more details see
 > > https://issues.apache.org/jira/browse/BEAM-3549
 > 
 > > In any case if to use SQS you only need the Apache licensed
 aws-sdk
 > > deps it is ok (and a good idea) if you put it in the
 > > amazon-web-services module.
 > >
 > > The kinesis connector is way more complex for multiple reasons,
 first,
 > > the raw version of the amazon client libraries is not so
 ‘friendly’
 > > and the guys who created KinesisIO had to do some workarounds to
 > > provide accurate checkpointing/watermarks. So since SQS is a way
 > > simpler system you should probably be ok basing it in simpler
 sources
 > > like AMQP or JMS.
 > >
 > > If you feel like to, please create the JIRA and don’t hesitate
 to ask
 > > questions if you find issues or if you need some review.
 > >
 > > On Thu, Jul 19, 2018 at 12:55 AM Lukasz Cwik >>> > > wrote:
 > >>
 > >>
 > >>
 > >> On Wed, Jul 18, 2018 at 3:30 PM John Rudolf Lewis
 > mailto:johnrle...@gmail.com>> wrote:
 > >>>
 > >>> I need an SQS source for my project that is using beam. A
 brief
 > search did not turn up any in-progress work in this area. Please
 > point me to the right repo if I missed it.
 

Re: SQS source

[John Rudolf Lewis]

So I guess I can add a timestamp to the message attributes when i receive
it from SQS since there is no such built in property.

But what triggers finilizeCheckpoint to be called? So far in my testing, I
never see that method get called, and hence, my messages keep getting
redelivered.

On Thu, Jul 19, 2018 at 5:26 PM, Raghu Angadi  wrote:

> A timestamp for a message is fundamental to an element in a PCollection.
> What do you mean by not knowing timestamp of a message?
> There is finalizeCheckpoint API[1] in UnboundedSource. Does that help?
> PubSub is also very similar, a message need to be acked with in a timeout,
> otherwise it will be redelivered to one of the consumer. Pubsub messages
> are acked inside finalize().
>
> [1]: https://github.com/apache/beam/blob/master/sdks/
> java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L129
>
>
> On Thu, Jul 19, 2018 at 3:28 PM John Rudolf Lewis 
> wrote:
>
>> hmm... made lots of progress on this today. But need help understanding
>> something
>>
>> UnboundedSource seems to assume that there is some guarantee of message
>> ordering, and that you can get the timestamp of the current message. Using
>> UnboundedSource.CheckpointMark to help advance the offset. Seems to work ok
>> for any source that supports those assumptions. But SQS does not work this
>> way.
>>
>> With a standard SQS queue, there is no guarantee of ordering and there is
>> no timestamp for a message.  With SQS, one needs to call the delete api
>> using the receipt handle from the message to acknowledge receipt of a
>> message and prevent its redelivery after the visibility timeout has expired.
>>
>> I'm not sure how to adapt these two patterns and would welcome
>> suggestions.
>>
>>
>>
>> On Thu, Jul 19, 2018 at 7:40 AM, Jean-Baptiste Onofré 
>> wrote:
>>
>>> Thx John !
>>>
>>> Regards
>>> JB
>>>
>>> On 19/07/2018 16:39, John Rudolf Lewis wrote:
>>> > Thank you.
>>> >
>>> > I've created a jira ticket to add SQS and have assigned it to
>>> > myself: https://issues.apache.org/jira/browse/BEAM-4828
>>> >
>>> > Modified the documentation to show it as in-progress:
>>> > https://github.com/apache/beam/pull/5995
>>> >
>>> > And will be starting my work
>>> > here: https://github.com/JohnRudolfLewis/beam/tree/Add-SqsIO
>>> >
>>> > On Thu, Jul 19, 2018 at 1:43 AM, Jean-Baptiste Onofré >> > > wrote:
>>> >
>>> > Agree with Ismaël.
>>> >
>>> > I would be more than happy to help on this one (as I contributed
>>> on AMQP
>>> > and JMS IOs ;)).
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 19/07/2018 10:39, Ismaël Mejía wrote:
>>> > > Thanks for your interest John, it would be a really nice
>>> contribution
>>> > > to add SQS support.
>>> > >
>>> > > Some context on the kinesis stuff:
>>> > >
>>> > > The reason why kinesis is still in a separate module is more
>>> related
>>> > > to a licensing problem. Kinesis uses some native libraries that
>>> are
>>> > > published under a not 100% apache compatible license and we are
>>> not
>>> > > allowed to shade and republish them but it seems there is a
>>> workaround
>>> > > now, for more details see
>>> > > https://issues.apache.org/jira/browse/BEAM-3549
>>> > 
>>> > > In any case if to use SQS you only need the Apache licensed
>>> aws-sdk
>>> > > deps it is ok (and a good idea) if you put it in the
>>> > > amazon-web-services module.
>>> > >
>>> > > The kinesis connector is way more complex for multiple reasons,
>>> first,
>>> > > the raw version of the amazon client libraries is not so
>>> ‘friendly’
>>> > > and the guys who created KinesisIO had to do some workarounds to
>>> > > provide accurate checkpointing/watermarks. So since SQS is a way
>>> > > simpler system you should probably be ok basing it in simpler
>>> sources
>>> > > like AMQP or JMS.
>>> > >
>>> > > If you feel like to, please create the JIRA and don’t hesitate
>>> to ask
>>> > > questions if you find issues or if you need some review.
>>> > >
>>> > > On Thu, Jul 19, 2018 at 12:55 AM Lukasz Cwik >> > > wrote:
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Wed, Jul 18, 2018 at 3:30 PM John Rudolf Lewis
>>> > mailto:johnrle...@gmail.com>> wrote:
>>> > >>>
>>> > >>> I need an SQS source for my project that is using beam. A brief
>>> > search did not turn up any in-progress work in this area. Please
>>> > point me to the right repo if I missed it.
>>> > >>
>>> > >>
>>> > >> To my knowledge there is none and nobody has marked it in
>>> > progress on https://beam.apache.org/documentation/io/built-in/
>>> > . It would be
>>> > good to create a JIRA issue on https://issues.apache.org/ and
>>> send a
>>> > PR to add SQS to the inprogress list 

Re: [DISCUSS] Gradle tips in Contribution Guide

[Alexey Romanenko]

+1 Agree to extend "Building & testing" section in "Beam Contribution Guide”, 
we need to have all commands there that are likely needed for development. 
All “advanced” gradle tricks, like what Luke’s document contains, probably 
could be put on wiki.

In the same time, I agree with JB to improve README.md but, since it’s mostly 
for users, then we need to add more user-oriented commands.


> On 23 Jul 2018, at 12:47, Łukasz Gajowy  wrote:
> 
> +1 to putting a link in README.md to more detailed building instructions. 
> 
> There already is a "Building & testing" section in "Beam Contribution 
> Guide"[1]. Maybe it's a good idea to just extend it with the contents from 
> the above-mentioned documents?
> 
> [1] https://beam.apache.org/contribute/#building--testing 
>  
> 
> pt., 20 lip 2018 o 18:15 Thomas Weise  > napisał(a):
> Yep, at least a link should be in README.md. Whether all details should be 
> captured there may be another matter, I thought of wiki/Confluence because it 
> is easier to edit.
> 
> Thomas
> 
> On Fri, Jul 20, 2018 at 8:59 AM Jean-Baptiste Onofré  > wrote:
> Hi,
> 
> as discussed together yesterday, it's a good idea.
> 
> IMHO, it should be directly in README.md. Let me explain, you are a
> contributor, you cloned the repository locally and you want to build.
> 
> So, you will probably start with ./gradlew projects and ./gradlew tasks.
> 
> However, pretty soon, you will look for some "basic" tricks: executing a
> test, publishing SNAPSHOT in local repository, ...
> And you are offline (coding during a flight, ...).
> 
> That's why I think README.md or BUILDING.md is a great location because
> it comes with the sources you want to build.
> 
> Regards
> JB
> 
> On 20/07/2018 16:41, Alexey Romanenko wrote:
> > Hi all,
> > 
> > Since we totally have moved to Gradle as Beam build system (after final
> > removing all maven poms recently), don’t you think that we need to add a
> > dedicated page or, at least, a paragraph, into Contribution Guide where
> > to add more examples of some tips while using Gradle? 
> > 
> > Comparing to Maven, I can guess that not so many people are so well
> > aware about how to use Gradle (me is a good example for that =)). For
> > instance, it can be quite useful to add some information, like, how to
> > build and publish maven artefacts locally (that can be used for testing
> > when you wish to build and run pipeline with custom jars), running
> > integration tests, remote debugging and etc. 
> > 
> > Ismaël Mejía and Luke Ćwik already started to write up such things in
> > these docs:
> > https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/
> >  
> > 
> > https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs
> >  
> > 
> > 
> > So, based on these docs, check that they are up-to-date with last
> > changes and create such new page on website which should help contributors.
> > 
> > What do you think?
> > 
> > Alexey

Re: [Vote] Dev wiki engine

[Mikhail Gryzykhin]

That's unanimous.

I believe all Committers have access to wiki.
https://cwiki.apache.org/confluence/display/BEAM/

Contributors can request access if required.

Thank you everyone for participation.
--Mikhail

Have feedback ?


On Fri, Jul 20, 2018 at 1:32 AM Alexey Romanenko 
wrote:

> +1 for Confluence
>
> On 20 Jul 2018, at 09:50, Łukasz Gajowy  wrote:
>
> +1 on confluence
>
> pt., 20 lip 2018 o 09:33 Etienne Chauchot 
> napisał(a):
>
>> +1 on confluence.
>> Le jeudi 19 juillet 2018 à 19:43 -0700, Rafael Fernandez a écrit :
>>
>> -1 .md! :-)
>>
>> On Thu, Jul 19, 2018 at 6:38 PM Gaurav Thakur 
>> wrote:
>>
>> +1 for confluence
>>
>> On Fri, Jul 20, 2018 at 11:42 AM Thomas Weise  wrote:
>>
>> +1 for Confluence
>>
>>
>> On Thu, Jul 19, 2018 at 4:00 PM Kai Jiang  wrote:
>>
>> +1 Apache Confluence
>>
>> On Thu, Jul 19, 2018, 15:18 Lukasz Cwik  wrote:
>>
>> +1 for confluence.
>>
>> On Thu, Jul 19, 2018 at 3:17 PM Anton Kedin  wrote:
>>
>> +1 for Confluence
>>
>>
>> On Thu, Jul 19, 2018 at 2:56 PM Andrew Pilloud 
>> wrote:
>>
>> +1 Apache Confluence
>>
>> Because .md files in code repo require code review and commit.
>>
>> On Thu, Jul 19, 2018, 2:22 PM Mikhail Gryzykhin 
>> wrote:
>>
>> Hi everyone,
>>
>> There is a long lasting discussion on starting Beam Dev Wiki
>> 
>> ongoing. Seems that the only question remaining is to decide on what engine
>> to use for wiki. So far it seems that we have two suggestions: confluence
>> and .md files in repo.
>>
>> Quick summary can also be found in following doc
>> 
>> .
>>
>> I suggest to vote on which approach to use:
>> 1. Apache Confluence
>> 2. .md files in code repository (Those can be rendered by Github)
>>
>> --Mikhail
>>
>>
>

Re: FYI: Jenkins Upgrade

[Jean-Baptiste Onofré]

Hi Jason,

yeah, it seems overall better.

Thanks !

Regards
JB

On 23/07/2018 18:33, Jason Kuster wrote:
> Hi all,
> 
> As an FYI: the Jenkins master got an upgrade to a much beefier machine
> over the weekend. Per Infra (on bui...@apache.org
> ): > Jenkins has been migrated to a brand new
> 48 core machine with 128GB RAM and 3.5TB of NVMe storage. Initial
> testing looks good, and builds are proceeding as expected. We hope this
> will provide a significant improvement in user experience on the
> front-end, and increase the throughput of builds. > > Please note that
> the 3.5TB of NVMe is less than what the previous jenkins master had. It
> is critical that all projects aggressively trim their retained builds in
> order for this shared resource to continue to function optimally.
> Anecdotally, the Jenkins UI feels much faster to me at this point, which
> is exciting!
> If anyone sees issues with e.g. scripts, access, builds queueing, or
> other Jenkins-related things please ping on this thread and we can put
> together a set of escalations to Infra.
> 
> As I recall we are not one of the projects causing problems with
> retained builds (I think we generally keep artifacts for <1mo) but it's
> worth noting the paragraph about storage space on Jenkins. If storage
> constraints give Infra problems they may implement a global retention
> window which would affect us at that point.
> 
> Best,
> 
> Jason
> 
> -- 
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
> 
> See something? Say something. go/jasonkuster-feedback

FYI: Jenkins Upgrade

[Jason Kuster]

Hi all,

As an FYI: the Jenkins master got an upgrade to a much beefier machine over
the weekend. Per Infra (on bui...@apache.org): > Jenkins has been migrated
to a brand new 48 core machine with 128GB RAM and 3.5TB of NVMe storage.
Initial testing looks good, and builds are proceeding as expected. We hope
this will provide a significant improvement in user experience on the
front-end, and increase the throughput of builds. > > Please note that the
3.5TB of NVMe is less than what the previous jenkins master had. It is
critical that all projects aggressively trim their retained builds in order
for this shared resource to continue to function optimally.
Anecdotally, the Jenkins UI feels much faster to me at this point, which is
exciting!
If anyone sees issues with e.g. scripts, access, builds queueing, or other
Jenkins-related things please ping on this thread and we can put together a
set of escalations to Infra.

As I recall we are not one of the projects causing problems with retained
builds (I think we generally keep artifacts for <1mo) but it's worth noting
the paragraph about storage space on Jenkins. If storage constraints give
Infra problems they may implement a global retention window which would
affect us at that point.

Best,

Jason

-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #110

[Apache Jenkins Server]

See 


Changes:

[carlos.alonso] Adds the BigQueryInsertError model class

[carlos.alonso] Adds BigQueryInsertErrorCoder and tests

[carlos.alonso] Adds extended errors info to WriteResult

[carlos.alonso] Makes streaming methods generic but limited to the implemented 
error

[carlos.alonso] Removes ErrorContainers auxiliary class. Adds 
BigQueryServicesImpl test

[carlos.alonso] Makes BigQueryIO.Write to be configurable to decide which kind 
of errors

[carlos.alonso] Removes unneeded coder registration

[carlos.alonso] Simplifies BigQueryInsertError codification

[carlos.alonso] A few style improvements

--
[...truncated 17.58 MB...]
:beam-sdks-python-container:installDependencies (Thread[Task worker for ':' 
Thread 42,5,main]) started.

> Task :beam-sdks-python-container:installDependencies
Caching disabled for task ':beam-sdks-python-container:installDependencies': 
Caching has not been enabled for the task
Task ':beam-sdks-python-container:installDependencies' is not up-to-date 
because:
  Task has not declared any outputs despite executing actions.
Cache 

 not found, skip.
:beam-sdks-python-container:installDependencies (Thread[Task worker for ':' 
Thread 42,5,main]) completed. Took 0.463 secs.
:beam-sdks-python-container:buildLinuxAmd64 (Thread[Task worker for ':' Thread 
42,5,main]) started.

> Task :beam-sdks-python-container:buildLinuxAmd64
Build cache key for task ':beam-sdks-python-container:buildLinuxAmd64' is 
3e31a02da0011ff4929871d0b8933414
Caching disabled for task ':beam-sdks-python-container:buildLinuxAmd64': 
Caching has not been enabled for the task
Task ':beam-sdks-python-container:buildLinuxAmd64' is not up-to-date because:
  No history is available.
:beam-sdks-python-container:buildLinuxAmd64 (Thread[Task worker for ':' Thread 
42,5,main]) completed. Took 2.488 secs.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 
42,5,main]) started.

> Task :beam-sdks-python-container:build
Caching disabled for task ':beam-sdks-python-container:build': Caching has not 
been enabled for the task
Task ':beam-sdks-python-container:build' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 
42,5,main]) completed. Took 0.001 secs.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 42,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:jar
Build cache key for task ':beam-vendor-sdks-java-extensions-protobuf:jar' is 
318184f3acfe150a3a38d605826c992a
Caching disabled for task ':beam-vendor-sdks-java-extensions-protobuf:jar': 
Caching has not been enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:jar' is not up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 42,5,main]) completed. Took 0.007 secs.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 42,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:compileTestJava' as 
it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 42,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 42,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:processTestResources' 
as it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 42,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 42,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:testClasses UP-TO-DATE
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:testClasses' as it 
has no actions.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 42,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 25,5,main]) started.

> Task