[jira] [Updated] (BEAM-4409) NoSuchMethodException reading from JmsIO

[Edward Pricer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Pricer updated BEAM-4409:

Description: 
Running with the DirectRunner, and reading from a queue with JmsIO as an 
unbounded source will produce a NoSuchMethodException. This occurs as the 
UnboundedReadEvaluatorFactory.UnboundedReadEvaluator attempts to clone the 
JmsCheckpointMark with the default (Avro) coder.

The following trivial code on the reader side reproduces the error 
(DirectRunner must be in path). The messages on the queue for this test case 
were simple TextMessages. I found this exception is triggered more readily when 
messages are published rapidly (~200/second)
{code:java}
Pipeline p = 
Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());

// read from the queue
ConnectionFactory factory = new
ActiveMQConnectionFactory("tcp://localhost:61616");

PCollection inputStrings = p.apply("Read from queue",
JmsIO.readMessage() .withConnectionFactory(factory)
.withQueue("somequeue") .withCoder(StringUtf8Coder.of())
.withMessageMapper((JmsIO.MessageMapper) message ->
((TextMessage) message).getText()));

// decode 
PCollection asStrings = inputStrings.apply("Decode Message", 
ParDo.of(new DoFn() { @ProcessElement public
void processElement(ProcessContext context) {
System.out.println(context.element());
context.output(context.element()); } })); 
p.run();
{code}
Stack trace:
{code:java}
Exception in thread "main" java.lang.RuntimeException: 
java.lang.NoSuchMethodException: javax.jms.Message.() at 
org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353) at 
org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369) at 
org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901) at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
 at 
org.apache.avro.reflect.ReflectDatumReader.readCollection(ReflectDatumReader.java:219)
 at 
org.apache.avro.reflect.ReflectDatumReader.readArray(ReflectDatumReader.java:137)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:177)
 at 
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:302)
 at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
 at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145) at 
org.apache.beam.sdk.coders.AvroCoder.decode(AvroCoder.java:318) at 
org.apache.beam.sdk.coders.Coder.decode(Coder.java:170) at 
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:122) 
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:105) 
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:99) 
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:148) at 
org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.getReader(UnboundedReadEvaluatorFactory.java:194)
 at 
org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:124)
 at 
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:161)
 at 
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:125)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748) Caused by: 
java.lang.NoSuchMethodException: javax.jms.Message.() at 
java.lang.Class.getConstructor0(Class.java:3082) at 
java.lang.Class.getDeclaredConstructor(Class.java:2178) at 
org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
{code}
 

And a more contrived example of how to produce the exception:
{code:java}
package org.apache.beam.sdk.io.jms; 
import org.apache.activemq.command.ActiveMQTextMessage; 
import org.apache.beam.sdk.coders.Coder; import 
org.apache.beam.sdk.util.CoderUtils; 
final class CoderErrorExample { public static void main(String[] args) throws 
Exception { 
  Coder coder = new JmsIO.UnboundedJmsSource(null).getCheckpointMarkCoder();
  JmsCheckpointMark checkpointMark = new JmsCheckpointMark(); 
  checkpointMark.addMessage(new ActiveMQTextMessage());
  CoderUtils.clone(coder, checkpointMark); // from 
org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory.UnboundedReadEvaluator#getReader
} 

[jira] [Created] (BEAM-4409) NoSuchMethodException reading from JmsIO

[Edward Pricer (JIRA)]

Edward Pricer created BEAM-4409:
---

 Summary: NoSuchMethodException reading from JmsIO
 Key: BEAM-4409
 URL: https://issues.apache.org/jira/browse/BEAM-4409
 Project: Beam
  Issue Type: Bug
  Components: io-java-jms
Affects Versions: 2.4.0
 Environment: Linux, Java 1.8, Beam 2.4, Direct Runner, ActiveMQ
Reporter: Edward Pricer
Assignee: Jean-Baptiste Onofré


Running with the DirectRunner, and reading from a queue with JmsIO as an 
unbounded source will produce a NoSuchMethodException. This occurs as the 
UnboundedReadEvaluatorFactory.UnboundedReadEvaluator attempts to clone the 
JmsCheckpointMark with the default (Avro) coder.

The following trivial code on the reader side reproduces the error 
(DirectRunner must be in path). The messages on the queue for this test case 
were simple TextMessages. I found this exception is triggered more readily when 
messages are published rapidly (~200/second)
{code:java}
Pipeline p = 
Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());

// read from the queue
ConnectionFactory factory = new
ActiveMQConnectionFactory("tcp://localhost:61616");

PCollection inputStrings = p.apply("Read from queue",
JmsIO.readMessage() .withConnectionFactory(factory)
.withQueue("somequeue") .withCoder(StringUtf8Coder.of())
.withMessageMapper((JmsIO.MessageMapper) message ->
((TextMessage) message).getText()));

// decode PCollection asStrings = inputStrings.apply("Decode
Message", ParDo.of(new DoFn() { @ProcessElement public
void processElement(ProcessContext context) {
System.out.println(context.element());
context.output(context.element()); } })); 
p.run();
{code}
Stack trace:
{code:java}
Exception in thread "main" java.lang.RuntimeException: 
java.lang.NoSuchMethodException: javax.jms.Message.() at 
org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353) at 
org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369) at 
org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901) at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
 at 
org.apache.avro.reflect.ReflectDatumReader.readCollection(ReflectDatumReader.java:219)
 at 
org.apache.avro.reflect.ReflectDatumReader.readArray(ReflectDatumReader.java:137)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:177)
 at 
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:302)
 at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
 at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145) at 
org.apache.beam.sdk.coders.AvroCoder.decode(AvroCoder.java:318) at 
org.apache.beam.sdk.coders.Coder.decode(Coder.java:170) at 
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:122) 
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:105) 
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:99) 
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:148) at 
org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.getReader(UnboundedReadEvaluatorFactory.java:194)
 at 
org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:124)
 at 
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:161)
 at 
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:125)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748) Caused by: 
java.lang.NoSuchMethodException: javax.jms.Message.() at 
java.lang.Class.getConstructor0(Class.java:3082) at 
java.lang.Class.getDeclaredConstructor(Class.java:2178) at 
org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
{code}
 

And a more contrived example of how to produce the exception:
{code:java}
package org.apache.beam.sdk.io.jms; 
import org.apache.activemq.command.ActiveMQTextMessage; 
import org.apache.beam.sdk.coders.Coder; import 
org.apache.beam.sdk.util.CoderUtils; 
final class CoderErrorExample { public static void main(String[] args) throws 
Exception { 
  Coder coder = new 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #342

[Apache Jenkins Server]

See 


Changes:

[robertwb] Slowly ramp up wait time in state sampler.

[tgroh] Move the Local Artifact Services to the DirectRunner

[tgroh] Migrate the ReferenceRunnerJobServer to runners-direct

[tgroh] Fix License Headers

[apilloud] [SQL] Fix transitive dependencies

[apilloud] [SQL] Run tests against actual jar

[tgroh] Update Docker Command

--
[...truncated 18.06 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 25, 2018 2:12:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 25, 2018 2:12:49 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525021245-6e73552b/output/results/staging/
May 25, 2018 2:12:49 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash 0yY4SgYX_KT_-h7hSmBFvw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525021245-6e73552b/output/results/staging/pipeline-0yY4SgYX_KT_-h7hSmBFvw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 2:12:50 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-24_19_12_49-15493961418752181912?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-24_19_12_49-15493961418752181912

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 2:12:50 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-24_19_12_49-15493961418752181912

[jira] [Updated] (BEAM-4407) Artifact stager should validate the manifest at pipeline submission time.

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-4407:

Component/s: (was: sdk-go)

> Artifact stager should validate the manifest at pipeline submission time.
> -
>
> Key: BEAM-4407
> URL: https://issues.apache.org/jira/browse/BEAM-4407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Henning Rohde
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4408) Go SDK artifact staging client should validate the manifest

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-4408:
---

 Summary: Go SDK artifact staging client should validate the 
manifest
 Key: BEAM-4408
 URL: https://issues.apache.org/jira/browse/BEAM-4408
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go, sdk-java-core, sdk-py-core
Reporter: Henning Rohde
Assignee: Henning Rohde


One step of executing a Beam pipeline is staging pipeline dependencies to the 
runner, for Python SDK example see: 
[https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
 and it's implementations. 

As a part of this process, we create a manifest of all staged artifacts. We 
should identify the requirements that constitute correctness of the manifest 
(for example, artifacts do not repeat twice, see also: 
https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
 and verify these requirements on the SDK side at pipeline submission, to fail 
faster.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4407) Artifact stager should validate the manifest at pipeline submission time.

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490064#comment-16490064
 ] 

Henning Rohde commented on BEAM-4407:
-

Moved the Go part to a separate bug

> Artifact stager should validate the manifest at pipeline submission time.
> -
>
> Key: BEAM-4407
> URL: https://issues.apache.org/jira/browse/BEAM-4407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4408) Go SDK artifact staging client should validate the manifest

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-4408:
---

Assignee: (was: Henning Rohde)

> Go SDK artifact staging client should validate the manifest
> ---
>
> Key: BEAM-4408
> URL: https://issues.apache.org/jira/browse/BEAM-4408
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4407) Artifact stager should validate the manifest at pipeline submission time.

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490064#comment-16490064
 ] 

Henning Rohde edited comment on BEAM-4407 at 5/25/18 12:56 AM:
---

Moved the Go part to a separate bug. We only stage 1 artifact at this point, so 
there is less urgency.


was (Author: herohde):
Moved the Go part to a separate bug

> Artifact stager should validate the manifest at pipeline submission time.
> -
>
> Key: BEAM-4407
> URL: https://issues.apache.org/jira/browse/BEAM-4407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4408) Go SDK artifact staging client should validate the manifest

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-4408:

Component/s: (was: sdk-py-core)
 (was: sdk-java-core)

> Go SDK artifact staging client should validate the manifest
> ---
>
> Key: BEAM-4408
> URL: https://issues.apache.org/jira/browse/BEAM-4408
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4407) Artifact stager should validate the manifest at pipeline submission time.

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-4407:
---

Assignee: (was: Henning Rohde)

> Artifact stager should validate the manifest at pipeline submission time.
> -
>
> Key: BEAM-4407
> URL: https://issues.apache.org/jira/browse/BEAM-4407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #341

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update GroupAlsoByWindowEvaluatorFactory

[boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar

[boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

--
[...truncated 18.05 MB...]
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 25, 2018 12:09:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 25, 2018 12:09:49 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525000946-a8a470c0/output/results/staging/
May 25, 2018 12:09:49 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash sd06lC0hzqLaAFuIa3_jJw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525000946-a8a470c0/output/results/staging/pipeline-sd06lC0hzqLaAFuIa3_jJw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 12:09:50 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-24_17_09_49-4612987860916041288?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #211

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update GroupAlsoByWindowEvaluatorFactory

[schroederc] Use ElementEncoders for beam.Create

[boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar

[robertwb] Slowly ramp up wait time in state sampler.

[tgroh] Move the Local Artifact Services to the DirectRunner

[tgroh] Migrate the ReferenceRunnerJobServer to runners-direct

[tgroh] Fix License Headers

[schroederc] Add test case for proto oneof fields in create_test.go

[apilloud] [SQL] Fix transitive dependencies

[apilloud] [SQL] Run tests against actual jar

[boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

[tgroh] Update Docker Command

--
[...truncated 469.52 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05241707-5wxt-harness-z15w.c.apache-beam-testing.internal/10.128.0.5
 to 174.230.224.35.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #216

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update GroupAlsoByWindowEvaluatorFactory

[schroederc] Use ElementEncoders for beam.Create

[boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar

[robertwb] Slowly ramp up wait time in state sampler.

[tgroh] Move the Local Artifact Services to the DirectRunner

[tgroh] Migrate the ReferenceRunnerJobServer to runners-direct

[tgroh] Fix License Headers

[schroederc] Add test case for proto oneof fields in create_test.go

[apilloud] [SQL] Fix transitive dependencies

[apilloud] [SQL] Run tests against actual jar

[boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

[tgroh] Update Docker Command

--
[...truncated 193.36 KB...]
INFO: No server chosen by WritableServerSelector from cluster description 
ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, 
all=[ServerDescription{address=104.197.70.76:27017, type=UNKNOWN, 
state=CONNECTING}]}. Waiting for 3 ms before timing out
May 25, 2018 12:15:18 AM com.mongodb.diagnostics.logging.SLF4JLogger info
INFO: Opened connection [connectionId{localValue:1, serverValue:1}] to 
104.197.70.76:27017
May 25, 2018 12:15:18 AM com.mongodb.diagnostics.logging.SLF4JLogger info
INFO: Monitor thread successfully connected to server with description 
ServerDescription{address=104.197.70.76:27017, type=STANDALONE, 
state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 5]}, 
minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, 
roundTripTimeNanos=2178560}
May 25, 2018 12:15:18 AM com.mongodb.diagnostics.logging.SLF4JLogger info
INFO: Opened connection [connectionId{localValue:2, serverValue:2}] to 
104.197.70.76:27017

Gradle Test Executor 1 finished executing tests.

> Task :beam-sdks-java-io-mongodb:integrationTest FAILED

org.apache.beam.sdk.io.mongodb.MongoDBIOIT > testWriteAndRead FAILED
java.lang.RuntimeException: com.mongodb.MongoSocketReadException: 
Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #210

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #635

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update GroupAlsoByWindowEvaluatorFactory

[schroederc] Use ElementEncoders for beam.Create

[boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar

[robertwb] Slowly ramp up wait time in state sampler.

[tgroh] Move the Local Artifact Services to the DirectRunner

[tgroh] Migrate the ReferenceRunnerJobServer to runners-direct

[tgroh] Fix License Headers

[schroederc] Add test case for proto oneof fields in create_test.go

[apilloud] [SQL] Fix transitive dependencies

[apilloud] [SQL] Run tests against actual jar

[boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

[tgroh] Update Docker Command

--
[...truncated 98.45 KB...]
:beam-runners-direct-java:shadowJar (Thread[Task worker for ':' Thread 
6,5,main]) completed. Took 0.016 secs.

> Task :beam-sdks-java-core:shadowTestJar UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:shadowTestJar' is 
eb085dcff49ef2078526803f1f8d900b
Caching disabled for task ':beam-sdks-java-core:shadowTestJar': Caching has not 
been enabled for the task
Skipping task ':beam-sdks-java-core:shadowTestJar' as it is up-to-date.
:beam-sdks-java-core:shadowTestJar (Thread[Task worker for ':' Thread 
12,5,main]) completed. Took 0.031 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
(Thread[Task worker for ':' Thread 12,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava' is 
6e111914758aae2d217f9875cd75a8dc
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
(Thread[Task worker for ':' Thread 12,5,main]) completed. Took 0.042 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:testClasses (Thread[Task 
worker for ':' Thread 12,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:testClasses 
> UP-TO-DATE
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:testClasses' as it has 
no actions.
:beam-sdks-java-extensions-google-cloud-platform-core:testClasses (Thread[Task 
worker for ':' Thread 12,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 12,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' is 
94355c32ac5642f3cf936a9c0fa1ff12
Caching disabled for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar': Caching 
has not been enabled for the task
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 12,5,main]) completed. Took 0.02 secs.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 12,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
65e46ecc339810ed9a2145cfdde7dee4
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 12,5,main]) completed. Took 0.072 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
bf59dc38c7c6b3694e2163ba969761f0
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.033 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 12,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #307

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update GroupAlsoByWindowEvaluatorFactory

[schroederc] Use ElementEncoders for beam.Create

[boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar

[robertwb] Slowly ramp up wait time in state sampler.

[tgroh] Move the Local Artifact Services to the DirectRunner

[tgroh] Migrate the ReferenceRunnerJobServer to runners-direct

[tgroh] Fix License Headers

[schroederc] Add test case for proto oneof fields in create_test.go

[apilloud] [SQL] Fix transitive dependencies

[apilloud] [SQL] Run tests against actual jar

[boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

[tgroh] Update Docker Command

--
[...truncated 106.43 KB...]
> Task :beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' is 
c4c060029c61f3973bb64e8fa3c78094
Caching disabled for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar': Caching 
has not been enabled for the task
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 8,5,main]) completed. Took 0.017 secs.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 8,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
9f35653cbb89cff366817a6194c660e5
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 8,5,main]) completed. Took 0.048 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 8,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 8,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 14,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
e0763f4811f609bc600bae6dd8ee5524
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 14,5,main]) completed. Took 0.032 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 14,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
c9ad3e35b9a0bb890c21584b38004f9f
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 14,5,main]) completed. Took 0.046 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 14,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 14,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 14,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
b00305420443493e2775726e17dfb9b4
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 14,5,main]) completed. Took 0.033 secs.
:beam-sdks-java-io-hadoop-input-format:compileTestJava (Thread[Task worker for 
':' Thread 14,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-hadoop-input-format:compileTestJava' is 
e9485deaa4365cf8e38a1ffb8154b259
Skipping task 

[jira] [Resolved] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud resolved BEAM-4357.
--
Resolution: Fixed

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (0e17676 -> 419cff6)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 0e17676  Prepare repository for deployment.
 add 475e323  Fix typo in becoming-a-committer docs
 add 1e2ed9a  This closes #447
 new 419cff6  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/become-a-committer/index.html | 2 +-
 src/contribute/become-a-committer.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 419cff6561853206dd54a021e9c3d2b25c7ebfbc
Author: Mergebot 
AuthorDate: Thu May 24 17:00:08 2018 -0700

Prepare repository for deployment.
---
 content/contribute/become-a-committer/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/content/contribute/become-a-committer/index.html 
b/content/contribute/become-a-committer/index.html
index e0cfe9c..5f8c08d 100644
--- a/content/contribute/become-a-committer/index.html
+++ b/content/contribute/become-a-committer/index.html
@@ -219,7 +219,7 @@ when to ask someone else to make the judgment
 
   They have a proven commitment to the project
   They share their intentions with the community
-  They accept and integrate community feedback in their on plans, designs,
+  They accept and integrate community feedback in their plans, designs,
 code, etc.
   They earnestly try to make Beam better with their contributions
   In particular, if a code contributor:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (79fbaed -> 1e2ed9a)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 79fbaed  This closes #448
 add 0e17676  Prepare repository for deployment.
 new 475e323  Fix typo in becoming-a-committer docs
 new 1e2ed9a  This closes #447

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/testing/index.html | 17 +
 src/contribute/become-a-committer.md  |  2 +-
 2 files changed, 2 insertions(+), 17 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #447

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 1e2ed9abc20fbecd0884b844eb0acaceafcd4164
Merge: 0e17676 475e323
Author: Mergebot 
AuthorDate: Thu May 24 16:57:33 2018 -0700

This closes #447

 src/contribute/become-a-committer.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Fix typo in becoming-a-committer docs

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 475e323dc37a5fbe314061305f7fb84095b3c3c9
Author: Scott Wegner 
AuthorDate: Tue May 22 14:25:35 2018 -0700

Fix typo in becoming-a-committer docs
---
 src/contribute/become-a-committer.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/contribute/become-a-committer.md 
b/src/contribute/become-a-committer.md
index 0d48917..8b34a86 100644
--- a/src/contribute/become-a-committer.md
+++ b/src/contribute/become-a-committer.md
@@ -76,7 +76,7 @@ See the [ASF 
documentation](https://www.apache.org/dev/committers.html#committer
 
  - They have a proven commitment to the project
  - They share their intentions with the community
- - They accept and integrate community feedback in their on plans, designs,
+ - They accept and integrate community feedback in their plans, designs,
code, etc.
  - They earnestly try to make Beam better with their contributions
  - In particular, if a code contributor:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch asf-site updated (4ecc2ab -> 0e17676)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 4ecc2ab  Prepare repository for deployment.
 add abfa977  Remove DoFnTester from testing guide.
 add 79fbaed  This closes #448
 new 0e17676  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/testing/index.html | 17 +
 src/contribute/testing.md | 15 ---
 2 files changed, 1 insertion(+), 31 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 0e176768f6b1cbe7515ec49bc6ad8874e07a2e04
Author: Mergebot 
AuthorDate: Thu May 24 16:56:40 2018 -0700

Prepare repository for deployment.
---
 content/contribute/testing/index.html | 17 +
 1 file changed, 1 insertion(+), 16 deletions(-)

diff --git a/content/contribute/testing/index.html 
b/content/contribute/testing/index.html
index ca0747b..2a1db6c 100644
--- a/content/contribute/testing/index.html
+++ b/content/contribute/testing/index.html
@@ -228,21 +228,6 @@ details on those testing types.
Runs In

   
-  
-   DoFn
-   
-   Correctness on one/few bundles
-   
-   DoFnTester
-   
-   https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java#L1325;>BigQueryIOTest
-   
-   Unit
-   
-   Precommit, Postcommit
-   
-  
-  
BoundedSource

Correctly Reads Input
@@ -255,7 +240,7 @@ details on those testing types.

Precommit, Postcommit

-  
+  /tr
   



-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (44ba321 -> 79fbaed)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 44ba321  This closes #444
 discard 20912e6  Add Go quickstart link to the footer
 new abfa977  Remove DoFnTester from testing guide.
 new 79fbaed  This closes #448

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (44ba321)
\
 N -- N -- N   refs/heads/mergebot (79fbaed)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/_includes/footer.html |  1 -
 src/contribute/testing.md | 15 ---
 2 files changed, 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Remove DoFnTester from testing guide.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit abfa977175a8e9d93e37328f6cc2005b14134dcd
Author: Scott Wegner 
AuthorDate: Tue May 22 14:37:58 2018 -0700

Remove DoFnTester from testing guide.

This documentation had a dead-link, and on investigating DoFnTester
isn't actually well-supported. So simply deleting the row is the best
path forward.
---
 src/contribute/testing.md | 15 ---
 1 file changed, 15 deletions(-)

diff --git a/src/contribute/testing.md b/src/contribute/testing.md
index 2dcc7fe..649967b 100644
--- a/src/contribute/testing.md
+++ b/src/contribute/testing.md
@@ -44,21 +44,6 @@ details on those testing types.
Runs In

   
-  
-   DoFn
-   
-   Correctness on one/few bundles
-   
-   DoFnTester
-   
-   https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java#L1325;>BigQueryIOTest
-   
-   Unit
-   
-   Precommit, Postcommit
-   
-  
-  
BoundedSource

Correctly Reads Input

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #448

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 79fbaedb3b95a6fa31aba461aab6fe364264550c
Merge: 4ecc2ab abfa977
Author: Mergebot 
AuthorDate: Thu May 24 16:53:57 2018 -0700

This closes #448

 src/contribute/testing.md | 15 ---
 1 file changed, 15 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #444

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 44ba3217d01b462995f66fda9ca2e2fc8d6b35be
Merge: 4ecc2ab 20912e6
Author: Mergebot 
AuthorDate: Thu May 24 16:51:00 2018 -0700

This closes #444

 src/_includes/footer.html | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (31b5675 -> 44ba321)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 31b5675  This closes #437
 add 4ecc2ab  Prepare repository for deployment.
 new 20912e6  Add Go quickstart link to the footer
 new 44ba321  This closes #444

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../go => dsls/sql/aggregate-functions}/index.html |   83 +-
 .../python => dsls/sql/data-types}/index.html  |  141 ++-
 content/documentation/dsls/sql/index.html  | 1135 
 .../{sdks/java => dsls/sql/joins}/index.html   |  108 +-
 .../{sdks/java => dsls/sql/overview}/index.html|   79 +-
 .../dsls/sql/scalar-functions/index.html   |  603 +++
 .../sql/statements/create-table}/index.html|   75 +-
 .../java => dsls/sql/statements/select}/index.html |  116 +-
 .../dsls/sql/user-defined-functions/index.html |  338 ++
 .../documentation/dsls/sql/walkthrough/index.html  |  385 +++
 .../sql/windowing-and-triggering}/index.html   |  102 +-
 .../sdks/feature-comparison/index.html |   29 +-
 content/documentation/sdks/go/index.html   |   29 +-
 .../documentation/sdks/java-extensions/index.html  |   29 +-
 .../documentation/sdks/java-thirdparty/index.html  |   29 +-
 content/documentation/sdks/java/index.html |   29 +-
 content/documentation/sdks/java/nexmark/index.html |   29 +-
 .../documentation/sdks/python-custom-io/index.html |   29 +-
 .../sdks/python-pipeline-dependencies/index.html   |   29 +-
 .../documentation/sdks/python-streaming/index.html |   29 +-
 .../sdks/python-type-safety/index.html |   29 +-
 content/documentation/sdks/python/index.html   |   29 +-
 src/_includes/footer.html  |1 +
 23 files changed, 2096 insertions(+), 1389 deletions(-)
 copy content/documentation/{sdks/go => 
dsls/sql/aggregate-functions}/index.html (80%)
 copy content/documentation/{sdks/python => dsls/sql/data-types}/index.html 
(74%)
 delete mode 100644 content/documentation/dsls/sql/index.html
 copy content/documentation/{sdks/java => dsls/sql/joins}/index.html (69%)
 copy content/documentation/{sdks/java => dsls/sql/overview}/index.html (77%)
 create mode 100644 content/documentation/dsls/sql/scalar-functions/index.html
 copy content/documentation/{sdks/java => 
dsls/sql/statements/create-table}/index.html (80%)
 copy content/documentation/{sdks/java => 
dsls/sql/statements/select}/index.html (74%)
 create mode 100644 
content/documentation/dsls/sql/user-defined-functions/index.html
 create mode 100644 content/documentation/dsls/sql/walkthrough/index.html
 copy content/documentation/{sdks/java => 
dsls/sql/windowing-and-triggering}/index.html (72%)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Add Go quickstart link to the footer

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 20912e6a260bd22f1bb273df222b772b7bc05235
Author: Henning Rohde 
AuthorDate: Fri May 18 17:01:21 2018 -0700

Add Go quickstart link to the footer
---
 src/_includes/footer.html | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/_includes/footer.html b/src/_includes/footer.html
index bab23db..5b2f129 100644
--- a/src/_includes/footer.html
+++ b/src/_includes/footer.html
@@ -14,6 +14,7 @@
 Overview
 Quickstart 
(Java)
 Quickstart 
(Python)
+Quickstart 
(Go)
 Downloads
   
   

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105782
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 23:35
Start Date: 24/May/18 23:35
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on a change in pull request #5467: 
[BEAM-4328]: beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#discussion_r190758729
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -64,6 +64,8 @@ dependencies {
   shadow library.java.slf4j_api
 
 Review comment:
   Fixed shaded projects list: 
https://github.com/apache/beam/pull/5467/commits/0354ebd43bcd10a90f9369596e275010cc51e6ff


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105782)
Time Spent: 2h 50m  (was: 2h 40m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4407) Artifact stager should validate the manifest at pipeline submission time.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489983#comment-16489983
 ] 

Valentyn Tymofieiev commented on BEAM-4407:
---

cc: [~angoenka], [~altay], [~herohde]

> Artifact stager should validate the manifest at pipeline submission time.
> -
>
> Key: BEAM-4407
> URL: https://issues.apache.org/jira/browse/BEAM-4407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go, sdk-java-core, sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Henning Rohde
>Priority: Minor
>
> One step of executing a Beam pipeline is staging pipeline dependencies to the 
> runner, for Python SDK example see: 
> [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
>  and it's implementations. 
> As a part of this process, we create a manifest of all staged artifacts. We 
> should identify the requirements that constitute correctness of the manifest 
> (for example, artifacts do not repeat twice, see also: 
> https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
>  and verify these requirements on the SDK side at pipeline submission, to 
> fail faster.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=105781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105781
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 24/May/18 23:33
Start Date: 24/May/18 23:33
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5443: [BEAM-4357]  Fix 
SQL shadow jar
URL: https://github.com/apache/beam/pull/5443
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build_rules.gradle b/build_rules.gradle
index 0132c062c56..465d5283b71 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -564,6 +564,11 @@ ext.applyJavaNature = {
 classifier = "shaded-tests"
 from sourceSets.test.output
 configurations = [project.configurations.testRuntime]
+
+exclude "META-INF/INDEX.LIST"
+exclude "META-INF/*.SF"
+exclude "META-INF/*.DSA"
+exclude "META-INF/*.RSA"
   } << configuration.shadowClosure)
 
   // Ensure that shaded jar and test-jar are part of the their own 
configuration artifact sets
diff --git a/sdks/java/extensions/sql/build.gradle 
b/sdks/java/extensions/sql/build.gradle
index 1f9005cfa1f..51d2f78c7ea 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -19,7 +19,25 @@ import groovy.json.JsonOutput
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature(enableSpotless: true)
+applyJavaNature(enableSpotless: true, shadowClosure: DEFAULT_SHADOW_CLOSURE << 
{
+  dependencies {
+include(dependency(library.java.protobuf_java))
+include(dependency(library.java.protobuf_java_util))
+include(dependency("org.apache.calcite:.*"))
+include(dependency("org.apache.calcite.avatica:.*"))
+include(dependency("org.codehaus.janino:.*"))
+  }
+  relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
+  relocate "org.apache.calcite", getJavaRelocatedPath("org.apache.calcite")
+
+  // Looking up the compiler factory in Calcite depends on having a properties
+  // file in the right location. We package one that is shading compatible
+  // in src/main/resources. Note that if this shaded path changes, that
+  // files name and contents need to be updated as well. TODO, swap to use
+  // getJavaRelocatedPath once the Maven build is no longer also shading this
+  // module.
+  relocate "org.codehaus", 
"org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus"
+})
 apply plugin: 'ca.coglinc.javacc'
 
 description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL"
@@ -31,6 +49,9 @@ configurations {
   // TODO: Migrate to a FMPP plugin once one exists
   fmppTask
   fmppTemplates
+
+  // Run tests from shaded jar
+  shadowTestRuntimeClasspath
 }
 
 def calcite_version = "1.16.0"
@@ -45,23 +66,31 @@ dependencies {
   compile "org.apache.calcite:calcite-core:$calcite_version"
   compile "org.apache.calcite:calcite-linq4j:$calcite_version"
   compile "org.apache.calcite.avatica:avatica-core:$avatica_version"
-  compile "com.alibaba:fastjson:1.2.12"
   compileOnly library.java.findbugs_jsr305
   compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow project(path: ":beam-sdks-java-extensions-join-library", 
configuration: "shadow")
   shadow library.java.slf4j_api
   shadow library.java.commons_csv
+  shadow library.java.commons_lang3
+  shadow library.java.jackson_databind
   shadow library.java.joda_time
+  shadow "com.alibaba:fastjson:1.2.12"
   shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-kafka", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-google-cloud-platform", 
configuration: "shadow")
   provided library.java.kafka_clients
-  testCompile library.java.slf4j_jdk14
-  testCompile library.java.junit
-  testCompile library.java.hamcrest_core
-  testCompile library.java.mockito_core
-  testCompile library.java.quickcheck_core
+  shadowTest library.java.slf4j_jdk14
+  shadowTest library.java.junit
+  shadowTest library.java.hamcrest_core
+  shadowTest library.java.mockito_core
+  shadowTest library.java.quickcheck_core
+  shadowTestRuntimeClasspath project(path: project.path, configuration: 
"shadowTest")
+  shadowTestRuntimeClasspath project(path: project.path, configuration: 
"provided")
+}
+
+test {
+  classpath = configurations.shadowTestRuntimeClasspath
 }
 
 // Copy Caclcite templates and our own template into the build directory
@@ -103,26 +132,6 @@ compileJavacc {
   arguments = [grammar_encoding: "UTF-8", static: "false", lookahead: "2"]
 }
 
-shadowJar {

[beam] branch master updated (b0b7e3b -> 5ca33d4)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b0b7e3b  Merge pull request #5370: Move the ReferenceRunner Job 
Service, Artifact Services to the DirectRunner module
 add 9ea90eb  [SQL] Fix transitive dependencies
 add 96f2d27  [SQL] Run tests against actual jar
 new 5ca33d4  Merge pull request #5443: [BEAM-4357]  Fix SQL shadow jar

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build_rules.gradle|  5 +++
 sdks/java/extensions/sql/build.gradle | 64 ---
 2 files changed, 41 insertions(+), 28 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #5443: [BEAM-4357] Fix SQL shadow jar

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5ca33d4798d4e3877821ef900e716863520b3fef
Merge: b0b7e3b 96f2d27
Author: Kenn Knowles 
AuthorDate: Thu May 24 16:33:27 2018 -0700

Merge pull request #5443: [BEAM-4357]  Fix SQL shadow jar

 build_rules.gradle|  5 +++
 sdks/java/extensions/sql/build.gradle | 64 ---
 2 files changed, 41 insertions(+), 28 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Created] (BEAM-4407) Artifact stager should validate the manifest at pipeline submission time.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-4407:
-

 Summary: Artifact stager should validate the manifest at pipeline 
submission time.
 Key: BEAM-4407
 URL: https://issues.apache.org/jira/browse/BEAM-4407
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go, sdk-java-core, sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Henning Rohde


One step of executing a Beam pipeline is staging pipeline dependencies to the 
runner, for Python SDK example see: 
[https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83]
 and it's implementations. 

As a part of this process, we create a manifest of all staged artifacts. We 
should identify the requirements that constitute correctness of the manifest 
(for example, artifacts do not repeat twice, see also: 
https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122),
 and verify these requirements on the SDK side at pipeline submission, to fail 
faster.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=105776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105776
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 24/May/18 23:24
Start Date: 24/May/18 23:24
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5445: [BEAM-2915] Add 
support for handling bag user state to the java-fn-execution library to support 
runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-391895687
 
 
   I was able to fix a few minor issues with the payload representation and 
tests. PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105776)
Time Spent: 10m
Remaining Estimate: 0h

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105774
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 24/May/18 23:20
Start Date: 24/May/18 23:20
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #5370: [BEAM-2885] Move the 
ReferenceRunner Job Service, Artifact Services to the DirectRunner module
URL: https://github.com/apache/beam/pull/5370
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/direct-java/build.gradle b/runners/direct-java/build.gradle
index af99b5b5bc2..893f21149b5 100644
--- a/runners/direct-java/build.gradle
+++ b/runners/direct-java/build.gradle
@@ -64,6 +64,7 @@ dependencies {
   shadow library.java.slf4j_api
   shadow library.java.hamcrest_core
   shadow library.java.junit
+  shadow library.java.args4j
   testRuntime project(path: ":beam-sdks-java-harness")
   shadowTest project(path: ":beam-sdks-java-core", configuration: "shadowTest")
   shadowTest project(path: ":beam-runners-core-java", configuration: 
"shadowTest")
diff --git 
a/runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalArtifactStagingLocation.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
similarity index 97%
rename from 
runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalArtifactStagingLocation.java
rename to 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
index e11125f75a3..146a091fbbe 100644
--- 
a/runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalArtifactStagingLocation.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-package org.apache.beam.artifact.local;
+package org.apache.beam.runners.direct.portable.artifact;
 
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkState;
@@ -29,7 +29,7 @@
  * A location where the results of an {@link 
LocalFileSystemArtifactStagerService} are stored and
  * where the retrieval service retrieves them from.
  */
-public class LocalArtifactStagingLocation {
+class LocalArtifactStagingLocation {
   /**
* Create a new {@link LocalArtifactStagingLocation} rooted at the specified 
location, creating
* any directories or subdirectories as necessary.
diff --git 
a/runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactRetrievalService.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java
similarity index 98%
rename from 
runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactRetrievalService.java
rename to 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java
index 73afcdcba96..7372ce7fb39 100644
--- 
a/runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactRetrievalService.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-package org.apache.beam.artifact.local;
+package org.apache.beam.runners.direct.portable.artifact;
 
 import com.google.protobuf.ByteString;
 import io.grpc.Status;
diff --git 
a/runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactStagerService.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
similarity index 95%
rename from 
runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactStagerService.java
rename to 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
index 049d6147633..6e72737bb84 100644
--- 
a/runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactStagerService.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 

[beam] branch master updated (3173f01 -> b0b7e3b)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3173f01  Merge pull request #5469: Update Docker Command
 add b5deb31  Move the Local Artifact Services to the DirectRunner
 add 01f7ce1  Migrate the ReferenceRunnerJobServer to runners-direct
 add 316caeb  Fix License Headers
 new b0b7e3b  Merge pull request #5370: Move the ReferenceRunner Job 
Service, Artifact Services to the DirectRunner module

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/direct-java/build.gradle   |   1 +
 .../artifact}/LocalArtifactStagingLocation.java|   4 +-
 .../LocalFileSystemArtifactRetrievalService.java   |   2 +-
 .../LocalFileSystemArtifactStagerService.java  |  14 +-
 .../direct/portable/artifact}/package-info.java|   6 +-
 .../runners/direct/portable}/job/PreparingJob.java |   7 +-
 .../portable}/job/ReferenceRunnerJobServer.java|  39 +++---
 .../portable}/job/ReferenceRunnerJobService.java   |  13 +-
 .../runners/direct/portable}/job/package-info.java |   2 +-
 .../runners/direct/DirectRunnerApiSurfaceTest.java |   4 +-
 .../LocalArtifactStagingLocationTest.java  |   2 +-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |   4 +-
 .../LocalFileSystemArtifactStagerServiceTest.java  |   4 +-
 .../job/ReferenceRunnerJobServiceTest.java |   3 +-
 runners/local-artifact-service-java/build.gradle   |  44 ---
 runners/local-artifact-service-java/pom.xml| 137 ---
 .../apache/beam/artifact/local/package-info.java   |  22 
 runners/reference/job-server/build.gradle  |  38 --
 runners/reference/job-server/pom.xml   | 145 -
 runners/reference/pom.xml  |   1 -
 sdks/java/javadoc/build.gradle |   2 -
 settings.gradle|   2 -
 22 files changed, 58 insertions(+), 438 deletions(-)
 rename 
runners/{local-artifact-service-java/src/main/java/org/apache/beam/artifact/local
 => 
direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact}/LocalArtifactStagingLocation.java
 (97%)
 rename 
runners/{local-artifact-service-java/src/main/java/org/apache/beam/artifact/local
 => 
direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact}/LocalFileSystemArtifactRetrievalService.java
 (98%)
 rename 
runners/{local-artifact-service-java/src/main/java/org/apache/beam/artifact/local
 => 
direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact}/LocalFileSystemArtifactStagerService.java
 (95%)
 copy 
runners/{apex/src/main/java/org/apache/beam/runners/apex/translation/utils => 
direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact}/package-info.java
 (86%)
 rename 
runners/{reference/job-server/src/main/java/org/apache/beam/runners/reference 
=> 
direct-java/src/main/java/org/apache/beam/runners/direct/portable}/job/PreparingJob.java
 (92%)
 rename 
runners/{reference/job-server/src/main/java/org/apache/beam/runners/reference 
=> 
direct-java/src/main/java/org/apache/beam/runners/direct/portable}/job/ReferenceRunnerJobServer.java
 (72%)
 rename 
runners/{reference/job-server/src/main/java/org/apache/beam/runners/reference 
=> 
direct-java/src/main/java/org/apache/beam/runners/direct/portable}/job/ReferenceRunnerJobService.java
 (92%)
 rename 
runners/{reference/job-server/src/main/java/org/apache/beam/runners/reference 
=> 
direct-java/src/main/java/org/apache/beam/runners/direct/portable}/job/package-info.java
 (94%)
 rename 
runners/{local-artifact-service-java/src/test/java/org/apache/beam/artifact/local
 => 
direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact}/LocalArtifactStagingLocationTest.java
 (98%)
 rename 
runners/{local-artifact-service-java/src/test/java/org/apache/beam/artifact/local
 => 
direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact}/LocalFileSystemArtifactRetrievalServiceTest.java
 (98%)
 rename 
runners/{local-artifact-service-java/src/test/java/org/apache/beam/artifact/local
 => 
direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact}/LocalFileSystemArtifactStagerServiceTest.java
 (98%)
 rename 
runners/{reference/job-server/src/test/java/org/apache/beam/runners/reference 
=> 
direct-java/src/test/java/org/apache/beam/runners/direct/portable}/job/ReferenceRunnerJobServiceTest.java
 (99%)
 delete mode 100644 runners/local-artifact-service-java/build.gradle
 delete mode 100644 runners/local-artifact-service-java/pom.xml
 delete mode 100644 
runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/package-info.java
 

[beam] 01/01: Merge pull request #5370: Move the ReferenceRunner Job Service, Artifact Services to the DirectRunner module

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b0b7e3bf4941f874d360923ffd1d03e38befc589
Merge: 3173f01 316caeb
Author: Thomas Groh 
AuthorDate: Thu May 24 16:20:08 2018 -0700

Merge pull request #5370: Move the ReferenceRunner Job Service, Artifact 
Services to the DirectRunner module

[BEAM-2885]

 runners/direct-java/build.gradle   |   1 +
 .../artifact}/LocalArtifactStagingLocation.java|   4 +-
 .../LocalFileSystemArtifactRetrievalService.java   |   2 +-
 .../LocalFileSystemArtifactStagerService.java  |  14 +-
 .../direct/portable/artifact}/package-info.java|   6 +-
 .../runners/direct/portable}/job/PreparingJob.java |   7 +-
 .../portable}/job/ReferenceRunnerJobServer.java|  39 +++---
 .../portable}/job/ReferenceRunnerJobService.java   |  13 +-
 .../runners/direct/portable}/job/package-info.java |   2 +-
 .../runners/direct/DirectRunnerApiSurfaceTest.java |   4 +-
 .../LocalArtifactStagingLocationTest.java  |   2 +-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |   4 +-
 .../LocalFileSystemArtifactStagerServiceTest.java  |   4 +-
 .../job/ReferenceRunnerJobServiceTest.java |   3 +-
 runners/local-artifact-service-java/build.gradle   |  44 ---
 runners/local-artifact-service-java/pom.xml| 137 ---
 runners/reference/job-server/build.gradle  |  38 --
 runners/reference/job-server/pom.xml   | 145 -
 runners/reference/pom.xml  |   1 -
 sdks/java/javadoc/build.gradle |   2 -
 settings.gradle|   2 -
 21 files changed, 58 insertions(+), 416 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Resolved] (BEAM-4393) Missing copyright notices for shaded packages

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4393.
-
Resolution: Fixed

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #541

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105771
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 23:14
Start Date: 24/May/18 23:14
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on a change in pull request #5467: 
[BEAM-4328]: beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#discussion_r190755681
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -64,6 +64,8 @@ dependencies {
   shadow library.java.slf4j_api
 
 Review comment:
   I checked beam-runners-direct-java shadowJar, it doesn't include shaded 
byte_buddy from sdks-java-core


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105771)
Time Spent: 2h 40m  (was: 2.5h)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4406) Increment major version used by Dataflow runner for portability

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-4406:
-

 Summary: Increment major version used by Dataflow runner for 
portability
 Key: BEAM-4406
 URL: https://issues.apache.org/jira/browse/BEAM-4406
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


The major version for portable pipelines in Java and Python is set to 1. I'd 
like to set them to 7 so they have parity with the major version of native 
pipelines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105770
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 22:54
Start Date: 24/May/18 22:54
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5467: [BEAM-4328]: 
beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#issuecomment-391890503
 
 
   If looking a the stacktrace, it seems like, direct runner called some guava 
method, these guava methods called byte_buddy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105770)
Time Spent: 2.5h  (was: 2h 20m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4402) Run tests against shaded JAR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4402?focusedWorklogId=105765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105765
 ]

ASF GitHub Bot logged work on BEAM-4402:


Author: ASF GitHub Bot
Created on: 24/May/18 22:42
Start Date: 24/May/18 22:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5471: 
[BEAM-4402] Test shadowJar everywhere
URL: https://github.com/apache/beam/pull/5471#discussion_r190750275
 
 

 ##
 File path: sdks/java/io/kinesis/build.gradle
 ##
 @@ -17,7 +17,7 @@
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature(failOnWarning: true)
+applyJavaNature(failOnWarning: true, shadowClosure: {})
 
 Review comment:
   The default if nothing is included and nothing is excluded means to include 
everything which is not what I think we want.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105765)
Time Spent: 0.5h  (was: 20m)

> Run tests against shaded JAR
> 
>
> Key: BEAM-4402
> URL: https://issues.apache.org/jira/browse/BEAM-4402
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We should run our tests against shaded jars.
> See https://github.com/apache/beam/pull/5471



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4402) Run tests against shaded JAR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4402?focusedWorklogId=105766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105766
 ]

ASF GitHub Bot logged work on BEAM-4402:


Author: ASF GitHub Bot
Created on: 24/May/18 22:42
Start Date: 24/May/18 22:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5471: 
[BEAM-4402] Test shadowJar everywhere
URL: https://github.com/apache/beam/pull/5471#discussion_r190750951
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -621,6 +629,20 @@ artifactId=${project.name}
 }
 artifacts.archives javadocJar
 
+// Run tests on artifacts
+configurations {
+  shadowTestRuntimeClasspath
+}
+
+dependencies {
+  shadowTestRuntimeClasspath project(path: project.path, configuration: 
"shadowTest")
+  shadowTestRuntimeClasspath project(path: project.path, configuration: 
"provided")
 
 Review comment:
   There is also a `runtime` configuration that should be added to the 
`shadowTestRuntimeClasspath`, similar to `provided`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105766)
Time Spent: 40m  (was: 0.5h)

> Run tests against shaded JAR
> 
>
> Key: BEAM-4402
> URL: https://issues.apache.org/jira/browse/BEAM-4402
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We should run our tests against shaded jars.
> See https://github.com/apache/beam/pull/5471



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4402) Run tests against shaded JAR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4402?focusedWorklogId=105764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105764
 ]

ASF GitHub Bot logged work on BEAM-4402:


Author: ASF GitHub Bot
Created on: 24/May/18 22:42
Start Date: 24/May/18 22:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5471: 
[BEAM-4402] Test shadowJar everywhere
URL: https://github.com/apache/beam/pull/5471#discussion_r190750782
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -621,6 +629,20 @@ artifactId=${project.name}
 }
 artifacts.archives javadocJar
 
+// Run tests on artifacts
 
 Review comment:
   Update comment to: `// Use a configuration and dependency set which 
represents the execution classpath using shaded artifacts for tests.`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105764)
Time Spent: 20m  (was: 10m)

> Run tests against shaded JAR
> 
>
> Key: BEAM-4402
> URL: https://issues.apache.org/jira/browse/BEAM-4402
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We should run our tests against shaded jars.
> See https://github.com/apache/beam/pull/5471



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105760
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 22:32
Start Date: 24/May/18 22:32
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5467: 
[BEAM-4328]: beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#discussion_r190749530
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -64,6 +64,8 @@ dependencies {
   shadow library.java.slf4j_api
 
 Review comment:
   shadow means its not shaded, so its dependencies should automatically become 
dependencies of this project.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105760)
Time Spent: 2h 10m  (was: 2h)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4312) Enforce ErrorProne analysis in Gearpump runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4312?focusedWorklogId=105759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105759
 ]

ASF GitHub Bot logged work on BEAM-4312:


Author: ASF GitHub Bot
Created on: 24/May/18 22:31
Start Date: 24/May/18 22:31
Worklog Time Spent: 10m 
  Work Description: manuzhang commented on issue #5474: [BEAM-4312] Enforce 
ErrorProne analysis in gearpump runner
URL: https://github.com/apache/beam/pull/5474#issuecomment-391883899
 
 
   Thanks. It seems there is a new primitive transform that Gearpump hasn't 
supported yet
   
   ```
   java.lang.IllegalStateException: no translator registered for Impulse
at 
org.apache.beam.runners.gearpump.translators.GearpumpPipelineTranslator.visitPrimitiveTransform(GearpumpPipelineTranslator.java:106)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:668)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at 
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
   ```
   
   Shall we exclude it ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105759)
Time Spent: 50m  (was: 40m)

> Enforce ErrorProne analysis in Gearpump runner project
> --
>
> Key: BEAM-4312
> URL: https://issues.apache.org/jira/browse/BEAM-4312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-gearpump}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-gearpump:assemble}}
> # Fix each ErrorProne warning from the {{runners/gearpump}} project.
> # In {{runners/gearpump/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4255) GloballyAsSingletonView is writing multiple values to the SingletonView

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4255?focusedWorklogId=105763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105763
 ]

ASF GitHub Bot logged work on BEAM-4255:


Author: ASF GitHub Bot
Created on: 24/May/18 22:37
Start Date: 24/May/18 22:37
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #5311: [BEAM-4255] Set 
singleton view writers' accumulation mode to discarding
URL: https://github.com/apache/beam/pull/5311#issuecomment-391887091
 
 
   Using singletonView with Combine is generally flawed. Not continuing this PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105763)
Time Spent: 1h 10m  (was: 1h)

> GloballyAsSingletonView is writing multiple values to the SingletonView
> ---
>
> Key: BEAM-4255
> URL: https://issues.apache.org/jira/browse/BEAM-4255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 3.0.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The code snippet below is writing multiple values to the sideInput. Resulting 
> in exception: "PCollection with more than one element accessed as a singleton 
> view."
> PCollectionView globalView = input
>  .apply(Window.into(new GlobalWindows())
>     .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
>     .accumulatingFiredPanes())
>  .apply(Sum.integersGlobally().asSingletonView())
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #539

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update GroupAlsoByWindowEvaluatorFactory

--
[...truncated 18.30 MB...]
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 24, 2018 10:26:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-24_15_26_46-9391827849725810598?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-24_15_26_46-9391827849725810598

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 24, 2018 10:26:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-24_15_26_46-9391827849725810598
May 24, 2018 10:26:47 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-24_15_26_46-9391827849725810598 with 0 
expected assertions.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:46.615Z: Autoscaling is enabled for job 
2018-05-24_15_26_46-9391827849725810598. The number of workers will be between 
1 and 1000.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:46.654Z: Autoscaling was automatically enabled for 
job 2018-05-24_15_26_46-9391827849725810598.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:49.125Z: Checking required Cloud APIs are enabled.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:49.254Z: Checking permissions granted to controller 
Service Account.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:52.864Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:53.293Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:53.550Z: Expanding GroupByKey operations into 
optimizable parts.
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:53.589Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:53.896Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:53.942Z: Elided trivial flatten 
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:53.987Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:54.033Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:54.087Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 24, 2018 10:27:00 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:26:54.130Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-4255) GloballyAsSingletonView is writing multiple values to the SingletonView

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4255?focusedWorklogId=105762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105762
 ]

ASF GitHub Bot logged work on BEAM-4255:


Author: ASF GitHub Bot
Created on: 24/May/18 22:36
Start Date: 24/May/18 22:36
Worklog Time Spent: 10m 
  Work Description: huygaa11 closed pull request #5311: [BEAM-4255] Set 
singleton view writers' accumulation mode to discarding
URL: https://github.com/apache/beam/pull/5311
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
index 6278d312d46..87e5b3db3b2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
@@ -1283,6 +1283,10 @@ private GloballyAsSingletonView(
   insertDefault,
   insertDefault ? fn.defaultValue() : null,
   combined.getCoder());
+
+  // GBK's inside view writers for singleton view should discard previous 
values.
+  
materializationInput.setWindowingStrategyInternal(materializationInput.getWindowingStrategy()
+  
.withMode(WindowingStrategy.AccumulationMode.DISCARDING_FIRED_PANES));
   materializationInput.apply(CreatePCollectionView.of(view));
   return view;
 }
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
index e34e98f1070..06a31cc6cb1 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
@@ -1299,6 +1299,38 @@ public void processElement(ProcessContext c) {
   pipeline.run();
 }
 
+/**
+ * This test is added to catch an issue that singleton side input with 
accumulating triggers
+ * were writing multiple values as singleton view. For more info:
+ * https://issues.apache.org/jira/browse/BEAM-4255
+ */
+@Test
+@Category({ ValidatesRunner.class })
+public void testCombineGloballyAsSingletonViewWithTrigger() {
+  final PCollectionView view = pipeline
+  .apply("CreateSideInput",
+  Create.timestamped(
+  TimestampedValue.of(1, new Instant(100)),
+  TimestampedValue.of(2, new Instant(100)),
+  TimestampedValue.of(3, new Instant(200)),
+  TimestampedValue.of(4, new Instant(200
+  .apply(Window.into(new GlobalWindows())
+  .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(2)))
+  .accumulatingFiredPanes())
+  .apply(Sum.integersGlobally().asSingletonView());
+
+  PCollection output = pipeline
+  .apply("CreateVoidMainInput", Create.of((Void) null))
+  .apply("OutputSideInput", ParDo.of(new DoFn() {
+@ProcessElement
+public void processElement(ProcessContext c) {
+  c.output(c.sideInput(view));
+}
+  }).withSideInputs(view));
+
+  pipeline.run();
+}
+
 @Test
 @Category(ValidatesRunner.class)
 public void testWindowedCombineGloballyAsSingletonView() {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105762)
Time Spent: 1h  (was: 50m)

> GloballyAsSingletonView is writing multiple values to the SingletonView
> ---
>
> Key: BEAM-4255
> URL: https://issues.apache.org/jira/browse/BEAM-4255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 3.0.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The code snippet below is writing multiple values to the sideInput. Resulting 
> in exception: "PCollection with more than one element accessed as a singleton 
> view."
> PCollectionView globalView = input
>  .apply(Window.into(new GlobalWindows())
>     

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105761
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 22:36
Start Date: 24/May/18 22:36
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5467: [BEAM-4328]: 
beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#issuecomment-391886913
 
 
   sdks/java/core is meant to shade it so bytebuddy should be repackaged 
underneath beam_sdks_java_core.
   
   Note that bytebuddy isn't imported anywhere within the direct-java module: 
https://github.com/apache/beam/search?q=bytebuddy_q=bytebuddy


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105761)
Time Spent: 2h 20m  (was: 2h 10m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5466 from robertwb/fast-state-sampler

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 73a26d54d3a0ba61f4574f69c86249463b84cdd7
Merge: 1e2218a 7ff8fea
Author: Robert Bradshaw 
AuthorDate: Thu May 24 15:29:36 2018 -0700

Merge pull request #5466 from robertwb/fast-state-sampler

Slowly ramp up wait time in state sampler.

 .../apache_beam/runners/worker/statesampler_fast.pyx  | 19 +--
 1 file changed, 17 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #540

[Apache Jenkins Server]

See 


Changes:

[boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

--
[...truncated 17.28 MB...]
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:47.474Z: Autoscaling is enabled for job 
2018-05-24_15_28_47-7054387351049985998. The number of workers will be between 
1 and 1000.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:47.520Z: Autoscaling was automatically enabled for 
job 2018-05-24_15_28_47-7054387351049985998.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:50.316Z: Checking required Cloud APIs are enabled.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:50.408Z: Checking permissions granted to controller 
Service Account.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:54.030Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:54.526Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:54.775Z: Expanding GroupByKey operations into 
optimizable parts.
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:54.820Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.100Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.130Z: Elided trivial flatten 
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.177Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.221Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.270Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.319Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.365Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:28:55.412Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 24, 2018 10:28:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #538

[Apache Jenkins Server]

See 


Changes:

[boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar

--
[...truncated 17.65 MB...]
May 24, 2018 10:25:30 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-24_15_25_29-6163476401828408823 with 0 
expected assertions.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:29.372Z: Autoscaling is enabled for job 
2018-05-24_15_25_29-6163476401828408823. The number of workers will be between 
1 and 1000.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:29.419Z: Autoscaling was automatically enabled for 
job 2018-05-24_15_25_29-6163476401828408823.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:31.895Z: Checking required Cloud APIs are enabled.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:32.085Z: Checking permissions granted to controller 
Service Account.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:35.555Z: Worker configuration: n1-standard-1 in 
us-central1-a.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:35.931Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.217Z: Expanding GroupByKey operations into 
optimizable parts.
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.247Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.532Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.578Z: Elided trivial flatten 
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.628Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.671Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.729Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.816Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.871Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 24, 2018 10:25:40 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-24T22:25:36.916Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

[beam] branch master updated (1e2218a -> 73a26d5)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1e2218a  Merge pull request #5472: [BEAM-4393]: Copy root repo's 
LICENSE & NOTICE into shadowJar
 add 7ff8fea  Slowly ramp up wait time in state sampler.
 new 73a26d5  Merge pull request #5466 from robertwb/fast-state-sampler

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache_beam/runners/worker/statesampler_fast.pyx  | 19 +--
 1 file changed, 17 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105756
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 22:25
Start Date: 24/May/18 22:25
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5467: [BEAM-4328]: 
beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#issuecomment-391884784
 
 
   Here is a build scan with gcp-io failure: 
https://scans.gradle.com/s/ocreh4atnld5a/tests/rn3sxlthlhuvc-d2gqd4tdbjfgq. As 
you can see, direct-runner uses byte_buddy as a runtime(?) dependency and it 
cannot find it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105756)
Time Spent: 2h  (was: 1h 50m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4312) Enforce ErrorProne analysis in Gearpump runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4312?focusedWorklogId=105755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105755
 ]

ASF GitHub Bot logged work on BEAM-4312:


Author: ASF GitHub Bot
Created on: 24/May/18 22:20
Start Date: 24/May/18 22:20
Worklog Time Spent: 10m 
  Work Description: manuzhang commented on issue #5474: [BEAM-4312] Enforce 
ErrorProne analysis in gearpump runner
URL: https://github.com/apache/beam/pull/5474#issuecomment-391883899
 
 
   Thanks. It seems there is a new primitive transform that Gearpump hasn't 
supported yet
   
   ```
   java.lang.IllegalStateException: no translator registered for Impulse
at 
org.apache.beam.runners.gearpump.translators.GearpumpPipelineTranslator.visitPrimitiveTransform(GearpumpPipelineTranslator.java:106)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:668)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at 
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105755)
Time Spent: 40m  (was: 0.5h)

> Enforce ErrorProne analysis in Gearpump runner project
> --
>
> Key: BEAM-4312
> URL: https://issues.apache.org/jira/browse/BEAM-4312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-gearpump}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-gearpump:assemble}}
> # Fix each ErrorProne warning from the {{runners/gearpump}} project.
> # In {{runners/gearpump/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105754
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 22:12
Start Date: 24/May/18 22:12
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on a change in pull request #5467: 
[BEAM-4328]: beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#discussion_r190745896
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -64,6 +64,8 @@ dependencies {
   shadow library.java.slf4j_api
 
 Review comment:
   But we list java-sdks-core as shadow dependency, which means we don't want 
to include this into our project right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105754)
Time Spent: 1h 50m  (was: 1h 40m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #3

[Apache Jenkins Server]

See 


--
[...truncated 9.91 MB...]
INFO: received start, clock: -9223372036854775808, sessionId: 15
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 16
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 17
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 8
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 21
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 18
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 19
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 20
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 22
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 23
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 24
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 25
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 26
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 27
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 28
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 30
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 29
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 10:10:33 PM org.apache.gearpump.streaming.task.TaskActor 
org$apache$gearpump$streaming$task$TaskActor$$onStartTask
INFO: received start, clock: -9223372036854775808, sessionId: 31
May 24, 2018 10:10:33 PM 
org.apache.gearpump.streaming.source.DataSourceTask onStart
INFO: opening data source at -9223372036854775808
May 24, 2018 

[jira] [Resolved] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[Boyuan Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyuan Zhang resolved BEAM-4392.

   Resolution: Fixed
Fix Version/s: Not applicable

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105752
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 22:09
Start Date: 24/May/18 22:09
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5467: 
[BEAM-4328]: beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#discussion_r190745152
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -64,6 +64,8 @@ dependencies {
   shadow library.java.slf4j_api
 
 Review comment:
   It seems like our list of shaded projects doesn't match the list defined up 
above, looks like we are missing `:beam-runners-java-fn-execution`, 
`:beam-sdks-java-fn-execution`. This seems worthwhile to fix regardless of the 
bytebuddy/commons_lang3


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105752)
Time Spent: 1h 40m  (was: 1.5h)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4312) Enforce ErrorProne analysis in Gearpump runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4312?focusedWorklogId=105751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105751
 ]

ASF GitHub Bot logged work on BEAM-4312:


Author: ASF GitHub Bot
Created on: 24/May/18 22:07
Start Date: 24/May/18 22:07
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5474: [BEAM-4312] 
Enforce ErrorProne analysis in gearpump runner
URL: https://github.com/apache/beam/pull/5474#issuecomment-391881140
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105751)
Time Spent: 0.5h  (was: 20m)

> Enforce ErrorProne analysis in Gearpump runner project
> --
>
> Key: BEAM-4312
> URL: https://issues.apache.org/jira/browse/BEAM-4312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-gearpump}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-gearpump:assemble}}
> # Fix each ErrorProne warning from the {{runners/gearpump}} project.
> # In {{runners/gearpump/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4405) Support SortValues GroupByKey extension

[Cody Schroeder (JIRA)]

Cody Schroeder created BEAM-4405:


 Summary: Support SortValues GroupByKey extension
 Key: BEAM-4405
 URL: https://issues.apache.org/jira/browse/BEAM-4405
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Cody Schroeder
Assignee: Henning Rohde


A port of the Java 
[SortValues|https://beam.apache.org/documentation/sdks/java-extensions/] 
extension would be helpful in the Go SDK.

Brief summary of the extension: for a GroupByKey or CoGroupByKey operation, 
allow the user to specify that the value streams per key should be sorted 
(either by their byte encoding or by a secondary sort key) upon delivery the 
user's ParDo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #340

[Apache Jenkins Server]

See 


Changes:

[schroederc] Use ElementEncoders for beam.Create

[schroederc] Add test case for proto oneof fields in create_test.go

--
[...truncated 17.98 MB...]
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 24, 2018 10:00:58 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 24, 2018 10:00:58 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524220052-d26ba83e/output/results/staging/
May 24, 2018 10:00:59 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash RD0lOwHI0W7S-BwZnLTysQ> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524220052-d26ba83e/output/results/staging/pipeline-RD0lOwHI0W7S-BwZnLTysQ.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 24, 2018 10:01:00 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-24_15_00_59-17256475416120802083?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-24_15_00_59-17256475416120802083

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 24, 2018 10:01:00 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:

[jira] [Work logged] (BEAM-4402) Run tests against shaded JAR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4402?focusedWorklogId=105749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105749
 ]

ASF GitHub Bot logged work on BEAM-4402:


Author: ASF GitHub Bot
Created on: 24/May/18 21:52
Start Date: 24/May/18 21:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5471: [BEAM-4402] Test 
shadowJar everywhere
URL: https://github.com/apache/beam/pull/5471#issuecomment-391877260
 
 
   I merged boyuanzz's change, can you rebase on top?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105749)
Time Spent: 10m
Remaining Estimate: 0h

> Run tests against shaded JAR
> 
>
> Key: BEAM-4402
> URL: https://issues.apache.org/jira/browse/BEAM-4402
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We should run our tests against shaded jars.
> See https://github.com/apache/beam/pull/5471



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4312) Enforce ErrorProne analysis in Gearpump runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4312?focusedWorklogId=105747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105747
 ]

ASF GitHub Bot logged work on BEAM-4312:


Author: ASF GitHub Bot
Created on: 24/May/18 21:51
Start Date: 24/May/18 21:51
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5474: [BEAM-4312] Enforce 
ErrorProne analysis in gearpump runner
URL: https://github.com/apache/beam/pull/5474#issuecomment-391876750
 
 
   Run Gearpump ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105747)
Time Spent: 20m  (was: 10m)

> Enforce ErrorProne analysis in Gearpump runner project
> --
>
> Key: BEAM-4312
> URL: https://issues.apache.org/jira/browse/BEAM-4312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-gearpump}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-gearpump:assemble}}
> # Fix each ErrorProne warning from the {{runners/gearpump}} project.
> # In {{runners/gearpump/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4306) Enforce ErrorProne analysis in apex project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4306?focusedWorklogId=105748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105748
 ]

ASF GitHub Bot logged work on BEAM-4306:


Author: ASF GitHub Bot
Created on: 24/May/18 21:51
Start Date: 24/May/18 21:51
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5475: [BEAM-4306] Enforce 
ErrorProne analysis in apex runner
URL: https://github.com/apache/beam/pull/5475#issuecomment-391876880
 
 
   Run Apex ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105748)
Time Spent: 20m  (was: 10m)

> Enforce ErrorProne analysis in apex project
> ---
>
> Key: BEAM-4306
> URL: https://issues.apache.org/jira/browse/BEAM-4306
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-apex}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-apex:assemble}}
> # Fix each ErrorProne warning from the {{runners/apex}} project.
> # In {{runners/apex/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4306) Enforce ErrorProne analysis in apex project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4306?focusedWorklogId=105746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105746
 ]

ASF GitHub Bot logged work on BEAM-4306:


Author: ASF GitHub Bot
Created on: 24/May/18 21:50
Start Date: 24/May/18 21:50
Worklog Time Spent: 10m 
  Work Description: iemejia opened a new pull request #5475: [BEAM-4306] 
Enforce ErrorProne analysis in apex runner
URL: https://github.com/apache/beam/pull/5475
 
 
   A small chunk of static analysis fixes to pass errorprone validations.
   R: @tweise 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105746)
Time Spent: 10m
Remaining Estimate: 0h

> Enforce ErrorProne analysis in apex project
> ---
>
> Key: BEAM-4306
> URL: https://issues.apache.org/jira/browse/BEAM-4306
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-apex}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-apex:assemble}}
> # Fix each ErrorProne warning from the {{runners/apex}} project.
> # In {{runners/apex/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105745
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:49
Start Date: 24/May/18 21:49
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391876253
 
 
   Merging in the interest of moving things slightly better. _If_ our current 
release process is mostly right, _then_ this PR fixes it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105745)
Time Spent: 2h 10m  (was: 2h)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105744
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:49
Start Date: 24/May/18 21:49
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5472: [BEAM-4393]: 
Copy root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build_rules.gradle b/build_rules.gradle
index ff070d3aaa8..c3cf20d8ee2 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -547,6 +547,10 @@ ext.applyJavaNature = {
   shadowJar ({
 classifier = "shaded"
 mergeServiceFiles()
+into("META-INF/") {
+  from "${rootProject.projectDir}/LICENSE"
+  from "${rootProject.projectDir}/NOTICE"
+}
   } << configuration.shadowClosure)
 
   // Always configure the shadowTestJar classifier and merge service files.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105744)
Time Spent: 2h  (was: 1h 50m)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (542d882 -> 1e2218a)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 542d882  Merge pull request #5460: Update 
GroupAlsoByWindowEvaluatorFactory
 add d8c1a3d  [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar
 new 1e2218a  Merge pull request #5472: [BEAM-4393]: Copy root repo's 
LICENSE & NOTICE into shadowJar

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build_rules.gradle | 4 
 1 file changed, 4 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #5472: [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8
Merge: 542d882 d8c1a3d
Author: Kenn Knowles 
AuthorDate: Thu May 24 14:49:18 2018 -0700

Merge pull request #5472: [BEAM-4393]: Copy root repo's LICENSE & NOTICE 
into shadowJar

 build_rules.gradle | 4 
 1 file changed, 4 insertions(+)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105742
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:48
Start Date: 24/May/18 21:48
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391875802
 
 
   Yea, OK. I didn't mean this PR is wrong. It seems this PR does put the 
LICENSE in the actual thing we want to ship to users. I guess the rest has to 
do with the release process.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105742)
Time Spent: 1h 50m  (was: 1h 40m)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4312) Enforce ErrorProne analysis in Gearpump runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4312?focusedWorklogId=105743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105743
 ]

ASF GitHub Bot logged work on BEAM-4312:


Author: ASF GitHub Bot
Created on: 24/May/18 21:48
Start Date: 24/May/18 21:48
Worklog Time Spent: 10m 
  Work Description: iemejia opened a new pull request #5474: [BEAM-4312] 
Enforce ErrorProne analysis in gearpump runner
URL: https://github.com/apache/beam/pull/5474
 
 
   R: @kennknowles 
   CC: @manuzhang 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105743)
Time Spent: 10m
Remaining Estimate: 0h

> Enforce ErrorProne analysis in Gearpump runner project
> --
>
> Key: BEAM-4312
> URL: https://issues.apache.org/jira/browse/BEAM-4312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-gearpump}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-gearpump:assemble}}
> # Fix each ErrorProne warning from the {{runners/gearpump}} project.
> # In {{runners/gearpump/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4404) Fix docker run arguments

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4404:


 Summary: Fix docker run arguments
 Key: BEAM-4404
 URL: https://issues.apache.org/jira/browse/BEAM-4404
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Ben Sidhom
Assignee: Thomas Groh


Docker options need to be separate from container entrypoint arguments. See 
[https://github.com/bsidhom/beam/commit/65c152d14d54d1505bfc2637aeb32189e00d160d.]

DockerCommand in master does not have this fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4306) Enforce ErrorProne analysis in apex project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4306:
--

Assignee: Ismaël Mejía

> Enforce ErrorProne analysis in apex project
> ---
>
> Key: BEAM-4306
> URL: https://issues.apache.org/jira/browse/BEAM-4306
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-apex}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-apex:assemble}}
> # Fix each ErrorProne warning from the {{runners/apex}} project.
> # In {{runners/apex/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4312) Enforce ErrorProne analysis in Gearpump runner project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4312:
--

Assignee: Ismaël Mejía

> Enforce ErrorProne analysis in Gearpump runner project
> --
>
> Key: BEAM-4312
> URL: https://issues.apache.org/jira/browse/BEAM-4312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-gearpump}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-gearpump:assemble}}
> # Fix each ErrorProne warning from the {{runners/gearpump}} project.
> # In {{runners/gearpump/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105741
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:46
Start Date: 24/May/18 21:46
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391875106
 
 
   I didn't find any Jar{} closure so far in 
https://github.com/apache/beam/blob/master/build_rules.gradle, which means, we 
possibly use the default configuration. I'm not sure whether we need LICENSE in 
Jar.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105741)
Time Spent: 1h 40m  (was: 1.5h)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5460: Update GroupAlsoByWindowEvaluatorFactory

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 542d8828138ee6d97556181228b64ad2b1a364ff
Merge: 3ec7b09 a25fc8d
Author: Thomas Groh 
AuthorDate: Thu May 24 14:38:13 2018 -0700

Merge pull request #5460: Update GroupAlsoByWindowEvaluatorFactory

 .../GroupAlsoByWindowEvaluatorFactory.java | 149 -
 .../direct/portable/StateAndTimerProvider.java |  32 +
 2 files changed, 120 insertions(+), 61 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105737
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 24/May/18 21:37
Start Date: 24/May/18 21:37
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build_rules.gradle b/build_rules.gradle
index 0132c062c56..7e876035e94 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -319,9 +319,12 @@ ext.getJavaRelocatedPath = { String suffix ->
   + suffix)
 }
 
+// By default if there is at least one include rule then all included 
dependencies must be specified.
+// This overrides the default behavior of include all if no includes are 
specified.
+// See details here:
+// 
https://github.com/johnrengelman/shadow/blob/98191096a94674245c7b3e63975df9e14f67074e/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/internal/DefaultDependencyFilter.groovy#L123
 ext.DEFAULT_SHADOW_CLOSURE = {
   dependencies {
-exclude(".*")
 include(dependency(library.java.guava))
   }
   relocate("com.google.common", getJavaRelocatedPath("com.google.common")) {
diff --git a/sdks/java/harness/build.gradle b/sdks/java/harness/build.gradle
index 254f89a9221..b3206101cf2 100644
--- a/sdks/java/harness/build.gradle
+++ b/sdks/java/harness/build.gradle
@@ -16,6 +16,15 @@
  * limitations under the License.
  */
 
+
+// We specifically enumerate all the projects that we depend on since
+// the list is used in both defining the included set for the uber jar
+// and also the set of project level dependencies.
+def dependOnProjects = [":beam-model-pipeline", ":beam-model-fn-execution", 
":beam-sdks-java-core",
+":beam-sdks-java-fn-execution",
+
":beam-sdks-java-extensions-google-cloud-platform-core",
+":beam-runners-core-java", 
":beam-runners-core-construction-java"]
+
 apply from: project(":").file("build_rules.gradle")
 applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE <<
   // Create an uber jar without repackaging for the SDK harness
@@ -24,7 +33,12 @@ applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE <<
   // jars.
   {
 dependencies {
-  include('**/*.class')
+  // Directly include all projects depended on
+  dependOnProjects.each {
+include(project(path: it, configuration: "shadow"))
+  }
+  // Include all dependencies and transitive dependencies
+  include(dependency(".*:.*"))
 }
   })
 
@@ -32,13 +46,9 @@ description = "Apache Beam :: SDKs :: Java :: Harness"
 ext.summary = "This contains the SDK Fn Harness for Beam Java"
 
 dependencies {
-  compile project(path: ":beam-model-pipeline", configuration: "shadow")
-  compile project(path: ":beam-model-fn-execution", configuration: "shadow")
-  compile project(path: ":beam-sdks-java-core", configuration: "shadow")
-  compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
-  compile project(path: 
":beam-sdks-java-extensions-google-cloud-platform-core", configuration: 
"shadow")
-  compile project(path: ":beam-runners-core-java", configuration: "shadow")
-  compile project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
+  dependOnProjects.each {
+compile project(path: it, configuration: "shadow")
+  }
   compile library.java.jackson_databind
   compile library.java.findbugs_jsr305
   compile library.java.guava
diff --git a/sdks/java/io/google-cloud-platform/build.gradle 
b/sdks/java/io/google-cloud-platform/build.gradle
index c0cf6ed4d5e..788123d411d 100644
--- a/sdks/java/io/google-cloud-platform/build.gradle
+++ b/sdks/java/io/google-cloud-platform/build.gradle
@@ -24,9 +24,7 @@ applyJavaNature(
   // Bigtable needs to expose Guava types.
   shadowClosure: {
 dependencies {
-  exclude(".*")
-  // Exclude netty due to conflict with spark runner, see [BEAM-3519].
-  exclude("io/netty/**")
+  exclude(dependency(".*:.*"))
 }
   })
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105737)
Time Spent: 3h 10m  (was: 3h)

> Task 

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105736
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:37
Start Date: 24/May/18 21:37
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391870925
 
 
   That seems like a problem. The shadowJar is the real thing and the "normal" 
jar is just an intermediate calculation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105736)
Time Spent: 1.5h  (was: 1h 20m)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (3ec7b09 -> 542d882)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3ec7b09  [BEAM-4392] Fix :beam-runners-java-fn-execution failure
 add a25fc8d  Update GroupAlsoByWindowEvaluatorFactory
 new 542d882  Merge pull request #5460: Update 
GroupAlsoByWindowEvaluatorFactory

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../GroupAlsoByWindowEvaluatorFactory.java | 149 -
 ...utorFactory.java => StateAndTimerProvider.java} |  18 +--
 2 files changed, 97 insertions(+), 70 deletions(-)
 copy 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/{TransformExecutorFactory.java
 => StateAndTimerProvider.java} (66%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: [BEAM-4392] Fix :beam-runners-java-fn-execution failure

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3ec7b09dae1cfea0c0039384fceae6f6f9196a04
Merge: 0ea97a5 f46cfa0
Author: Lukasz Cwik 
AuthorDate: Thu May 24 14:37:41 2018 -0700

[BEAM-4392] Fix :beam-runners-java-fn-execution failure

 build_rules.gradle  |  5 -
 sdks/java/harness/build.gradle  | 26 +
 sdks/java/io/google-cloud-platform/build.gradle |  4 +---
 3 files changed, 23 insertions(+), 12 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (0ea97a5 -> 3ec7b09)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0ea97a5  Merge pull request #5465: [BEAM-4401] Use ElementEncoders for 
beam.Create
 add f46cfa0  Change bulid rules of sdks/java/harness to make harness as a 
uber-jar
 new 3ec7b09  [BEAM-4392] Fix :beam-runners-java-fn-execution failure

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build_rules.gradle  |  5 -
 sdks/java/harness/build.gradle  | 26 +
 sdks/java/io/google-cloud-platform/build.gradle |  4 +---
 3 files changed, 23 insertions(+), 12 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105735
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:36
Start Date: 24/May/18 21:36
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391870925
 
 
   That seems wrong. The shadowJar is the real thing and the "normal" jar is 
just an intermediate calculation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105735)
Time Spent: 1h 20m  (was: 1h 10m)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105734
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:35
Start Date: 24/May/18 21:35
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391870562
 
 
   I believe we run the jar rule to build a jar with only our code, then we run 
the shadowJar rule to bundle that jar with third party code. This only needs to 
be in shadowJar, but that leaves our code in jar without a LICENSE file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105734)
Time Spent: 1h 10m  (was: 1h)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105732
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:32
Start Date: 24/May/18 21:32
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391869880
 
 
   Hey @apilloud , my understanding so far is, we use shadowJar to rewrite Jar. 
How do you feel like? @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105732)
Time Spent: 1h  (was: 50m)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=105730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105730
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 24/May/18 21:28
Start Date: 24/May/18 21:28
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5443: [BEAM-4357]  Fix SQL 
shadow jar
URL: https://github.com/apache/beam/pull/5443#issuecomment-391868814
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105730)
Time Spent: 3.5h  (was: 3h 20m)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=105729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105729
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 24/May/18 21:27
Start Date: 24/May/18 21:27
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5443: [BEAM-4357]  Fix SQL 
shadow jar
URL: https://github.com/apache/beam/pull/5443#issuecomment-391868772
 
 
   Now we get 
`org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithIndexFn`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105729)
Time Spent: 3h 20m  (was: 3h 10m)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105726
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 21:22
Start Date: 24/May/18 21:22
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5472: [BEAM-4393]: Copy 
root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472#issuecomment-391867608
 
 
   LGTM! I verified that this fixes BEAM-4393.
   
   Only one question: should this go in jar instead of shadowJar?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105726)
Time Spent: 50m  (was: 40m)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-386) Dataflow runner to support Read.Bounded in streaming mode.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-386?focusedWorklogId=105722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105722
 ]

ASF GitHub Bot logged work on BEAM-386:
---

Author: ASF GitHub Bot
Created on: 24/May/18 21:18
Start Date: 24/May/18 21:18
Worklog Time Spent: 10m 
  Work Description: herohde opened a new pull request #5473: [BEAM-386] 
Tweak Go integration test driver
URL: https://github.com/apache/beam/pull/5473
 
 
* Rename parameter as per design doc feedback


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105722)
Time Spent: 10m
Remaining Estimate: 0h

> Dataflow runner to support Read.Bounded in streaming mode.
> --
>
> Key: BEAM-386
> URL: https://issues.apache.org/jira/browse/BEAM-386
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Pei He
>Assignee: Pei He
>Priority: Major
> Fix For: 0.2.0-incubating
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> UnboundedReadFromBoundedSource is done.
> Make Dataflow runner use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-386) Dataflow runner to support Read.Bounded in streaming mode.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-386?focusedWorklogId=105723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105723
 ]

ASF GitHub Bot logged work on BEAM-386:
---

Author: ASF GitHub Bot
Created on: 24/May/18 21:18
Start Date: 24/May/18 21:18
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #5473: [BEAM-386] Tweak Go 
integration test driver
URL: https://github.com/apache/beam/pull/5473#issuecomment-391866171
 
 
   Run Go PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105723)
Time Spent: 20m  (was: 10m)

> Dataflow runner to support Read.Bounded in streaming mode.
> --
>
> Key: BEAM-386
> URL: https://issues.apache.org/jira/browse/BEAM-386
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Pei He
>Assignee: Pei He
>Priority: Major
> Fix For: 0.2.0-incubating
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> UnboundedReadFromBoundedSource is done.
> Make Dataflow runner use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4393) Missing copyright notices for shaded packages

[Boyuan Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489804#comment-16489804
 ] 

Boyuan Zhang commented on BEAM-4393:


[https://github.com/apache/beam/pull/5472] trying to solve issues as follow:

1. There are some shaded jars(eg. java core) containing NOTICE & LICENSE, but 
some shaded jars(eg. java harness) not. 
2. The LICENSE included in shaded jar is the default one, not the LICENSE from 
root repo.

 

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105719
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 20:59
Start Date: 24/May/18 20:59
Worklog Time Spent: 10m 
  Work Description: boyuanzz opened a new pull request #5472: [BEAM-4393]: 
Copy root repo's LICENSE & NOTICE into shadowJar
URL: https://github.com/apache/beam/pull/5472
 
 
   r: @kennknowles 
   cc: @apilloud 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105719)
Time Spent: 40m  (was: 0.5h)

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3883) Python SDK stages artifacts when talking to job server

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka resolved BEAM-3883.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4402) Run tests against shaded JAR

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489766#comment-16489766
 ] 

Kenneth Knowles commented on BEAM-4402:
---

It also looks like we are building them with classifier "shaded" but they 
should be the default build. That might "just work" if the tests run against 
the default jar. But they may run just against the accumulated classes, which 
is actually the most likely to succeed even when Beam is broken.

> Run tests against shaded JAR
> 
>
> Key: BEAM-4402
> URL: https://issues.apache.org/jira/browse/BEAM-4402
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>
> We should run our tests against shaded jars.
> See https://github.com/apache/beam/pull/5471



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=105717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105717
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 24/May/18 20:34
Start Date: 24/May/18 20:34
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5467: [BEAM-4328]: 
beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#issuecomment-391850532
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105717)
Time Spent: 1.5h  (was: 1h 20m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)