Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3019

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1035) Support for new State API in SparkRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994294#comment-15994294
 ] 

ASF GitHub Bot commented on BEAM-1035:
--

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2853

[BEAM-1035][BEAM-1115] Support for new State and Timer API in Spark batch 
runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1035

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2853


commit 66a472519122e8b34b21fcb1bfe93834351cd07e
Author: JingsongLi 
Date:   2017-05-03T05:15:26Z

[BEAM-1035][BEAM-1115] Support for new State and Timer API in Spark batch 
runner




> Support for new State API in SparkRunner
> 
>
> Key: BEAM-1035
> URL: https://issues.apache.org/jira/browse/BEAM-1035
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Aviem Zur
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2853: [BEAM-1035][BEAM-1115] Support for new State and Ti...

[JingsongLi]

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2853

[BEAM-1035][BEAM-1115] Support for new State and Timer API in Spark batch 
runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1035

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2853


commit 66a472519122e8b34b21fcb1bfe93834351cd07e
Author: JingsongLi 
Date:   2017-05-03T05:15:26Z

[BEAM-1035][BEAM-1115] Support for new State and Timer API in Spark batch 
runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2079) Support TextIO as SQL source/sink

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2079:
--

Assignee: James Xu

> Support TextIO as SQL source/sink
> -
>
> Key: BEAM-2079
> URL: https://issues.apache.org/jira/browse/BEAM-2079
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>
> Use TextIO as source/sink to access text-format data in Beam SQL.
> It's an implementation of 
> {{https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BaseBeamTable.java}},
>  and can refer to 
> {{https://github.com/apache/beam/blob/DSL_SQL/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/kafka/BeamKafkaTable.java}}
>  as an example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2111) ClassCastException in streaming Spark pipeline.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-2111:

Summary: ClassCastException in streaming Spark pipeline.  (was: 
java.lang.ClassCastException in spark in streaming mode)

> ClassCastException in streaming Spark pipeline.
> ---
>
> Key: BEAM-2111
> URL: https://issues.apache.org/jira/browse/BEAM-2111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Aviem Zur
>
> Bug can be reproduced with :
> run Nexmark query5 (https://github.com/iemejia/beam/tree/BEAM-160-nexmark) in 
> streaming mode using Spark.
> Run main in 
> {code}org.apache.beam.integration.nexmark.drivers.NexmarkSparkDriver{code}
> with VMOptions:
> {code} -Dspark.ui.enabled=false -DSPARK_LOCAL_IP=localhost 
> -Dsun.io.serialization.extendedDebugInfo=true {code}
> with Program arguments:
> {code}--query=5  --streaming=true --numEventGenerators=4 
> --manageResources=false --monitorJobs=true --enforceEncodability=false 
> --enforceImmutability=false{code}
> StackTrace is 
> {code}
>  com.esotericsoftware.kryo.KryoException: java.lang.RuntimeException: 
> java.lang.ClassCastException: [J cannot be cast to [Ljava.lang.Object;
> Serialization trace:
> key (org.apache.beam.sdk.values.KV)
> value (org.apache.beam.sdk.values.KV)
> value (org.apache.beam.sdk.util.WindowedValue$TimestampedValueInSingleWindow)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:42)
>   at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:33)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>   at 
> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:228)
>   at 
> org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:181)
>   at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
>   at 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>   at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>   at 
> scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:29)
>   at 
> org.apache.beam.runners.spark.translation.SparkProcessContext.processPartition(SparkProcessContext.java:64)
>   at 
> org.apache.beam.runners.spark.translation.MultiDoFnFunction.call(MultiDoFnFunction.java:114)
>   at 
> org.apache.beam.runners.spark.translation.MultiDoFnFunction.call(MultiDoFnFunction.java:52)
>   at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
>   at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)

[jira] [Commented] (BEAM-818) ValueProvider for tempLocation, runner, etc, that is unavailable to transforms during construction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994233#comment-15994233
 ] 

ASF GitHub Bot commented on BEAM-818:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/2847


> ValueProvider for tempLocation, runner, etc, that is unavailable to 
> transforms during construction
> --
>
> Key: BEAM-818
> URL: https://issues.apache.org/jira/browse/BEAM-818
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This stops transforms from changing their operation based on 
> construction-time options, and instead requires that configuration to be 
> explicit, or to obtain the configuration at runtime.
> https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk--XmGMGgDwwNwWZaFbxLKvPqEQ/edit#



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2145) Do not publish javadoc for Java SDK's transforms.reflect directory

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-2145:
-

Assignee: (was: Davor Bonaci)

> Do not publish javadoc for Java SDK's transforms.reflect directory
> --
>
> Key: BEAM-2145
> URL: https://issues.apache.org/jira/browse/BEAM-2145
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, website
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> This is where the utilities for analyzing a {{DoFn}} live. They need to be in 
> the core SDK for {{DoFnTester}}, otherwise they would be in 
> runner-core-construction.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2847: [BEAM-818,BEAM-828] Make tempLocation an inaccessib...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/2847


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2143) (Mis)Running Dataflow Wordcount gives non-helpful errors

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-2143:
-

Assignee: (was: Daniel Halperin)

> (Mis)Running Dataflow Wordcount gives non-helpful errors
> 
>
> Key: BEAM-2143
> URL: https://issues.apache.org/jira/browse/BEAM-2143
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ben Chambers
>
> If you run a pipeline and forget to specify `tempLocation` (but specify 
> something else, such as `stagingLocation`) you get two messages indicating 
> you forgot to specify `stagingLocation`. 
> One says "no stagingLocation specified, choosing ..." the other says "error, 
> the staging location isn't readable" (if you give it just a bucket and not an 
> object within a bucket).
> This is surprising to me as a user, since (1) I specified a staging location 
> and (2) the flag I actually need to modify is `--tempLocation`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994217#comment-15994217
 ] 

Kenneth Knowles edited comment on BEAM-828 at 5/3/17 3:21 AM:
--

Actually [~vikasrk] is already in progress just using 
{{ProcessContext.getPipelineOptions()}} directly, which was an alternative I 
simply missed.

The move to {{ValueProvider}} has only to do with making new transforms unable 
to do the wrong thing, which is also important.

But we could conceivably still close on BEAM-885 (making them unavailable off 
the pipeline) without succeeding at BEAM-818 (setting up the value provider 
thing) by just porting transforms to use the one in the process context, which 
is nice.


was (Author: kenn):
Actually [~vikasrk] is already in progress just using 
{{ProcessContext.getPipelineOptions()}} directly, which was an alternative I 
simply missed.

The move to {{ValueProvider}} has only to do with making new transforms unable 
to do the wrong thing, which is also important.

But we could conceivably still close on BEAM-885 without succeeding at 
BEAM-818, which is nice.

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (BEAM-885) Move PipelineOptions from Pipeline.create() to Pipeline.run()

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-885:
--

> Move PipelineOptions from Pipeline.create() to Pipeline.run()
> -
>
> Key: BEAM-885
> URL: https://issues.apache.org/jira/browse/BEAM-885
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The specification of a Pipeline should be independent of its PipelineOptions. 
> This delays specification of the options, including choices like Pipeline 
> Runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994217#comment-15994217
 ] 

Kenneth Knowles edited comment on BEAM-828 at 5/3/17 3:20 AM:
--

Actually [~vikasrk] is already in progress just using 
{{ProcessContext.getPipelineOptions()}} directly, which was an alternative I 
simply missed.

The move to {{ValueProvider}} has only to do with making new transforms unable 
to do the wrong thing, which is also important.

But we could conceivably still close on BEAM-885 without succeeding at 
BEAM-818, which is nice.


was (Author: kenn):
Actually [~vikasrk] is already in progress just using 
{{ProcessContext.getPipelineOptions()}} directly, which was an alternative I 
simply missed.

The move to {{ValueProvider}} has only to do with making new transforms unable 
to do the wrong thing, which is also important.

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-828:
-
Issue Type: Bug  (was: Sub-task)
Parent: (was: BEAM-818)

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994217#comment-15994217
 ] 

Kenneth Knowles commented on BEAM-828:
--

Actually [~vikasrk] is already in progress just using 
{{ProcessContext.getPipelineOptions()}} directly, which was an alternative I 
simply missed.

The move to {{ValueProvider}} has only to do with making new transforms unable 
to do the wrong thing, which is also important.

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-828:
--
  Assignee: Vikas Kedigehalli  (was: Kenneth Knowles)

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994212#comment-15994212
 ] 

ASF GitHub Bot commented on BEAM-1340:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2852

[BEAM-1340,BEAM-1345] Tight up access and use internal annotations a bit in 
the Java SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This is a pass over `sdk.runners`, `sdk.transforms`, 
`sdk.transforms.reflect` (which is entirely internal), 
`sdk.transforms.windowing` trigger parts, and `sdk.values`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam Internal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2852.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2852


commit 664eca296a6310399e71ff350e611c06519db9cf
Author: Kenneth Knowles 
Date:   2017-05-03T02:31:22Z

Annotate internal methods of PCollection

commit 4f61e779e4575a96083bb6ef3d80e29499c544ab
Author: Kenneth Knowles 
Date:   2017-05-03T02:48:38Z

Annotate internal pieces of sdks.transforms

commit c003770a2fc2549e123788cdea97019d7896c9ff
Author: Kenneth Knowles 
Date:   2017-05-03T02:59:32Z

Tighten access control and internal annotations for triggers

commit 9d365b14d5fb6c0992424a28e08b40263a6c6536
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:34Z

The transforms.reflect package is not for users

commit 2e62dcd1075210d1ea0411484b0184b7c8431f4c
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:45Z

Annotate internal-only bits of Java sdk.runners




> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2852: [BEAM-1340,BEAM-1345] Tight up access and use inter...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2852

[BEAM-1340,BEAM-1345] Tight up access and use internal annotations a bit in 
the Java SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This is a pass over `sdk.runners`, `sdk.transforms`, 
`sdk.transforms.reflect` (which is entirely internal), 
`sdk.transforms.windowing` trigger parts, and `sdk.values`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam Internal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2852.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2852


commit 664eca296a6310399e71ff350e611c06519db9cf
Author: Kenneth Knowles 
Date:   2017-05-03T02:31:22Z

Annotate internal methods of PCollection

commit 4f61e779e4575a96083bb6ef3d80e29499c544ab
Author: Kenneth Knowles 
Date:   2017-05-03T02:48:38Z

Annotate internal pieces of sdks.transforms

commit c003770a2fc2549e123788cdea97019d7896c9ff
Author: Kenneth Knowles 
Date:   2017-05-03T02:59:32Z

Tighten access control and internal annotations for triggers

commit 9d365b14d5fb6c0992424a28e08b40263a6c6536
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:34Z

The transforms.reflect package is not for users

commit 2e62dcd1075210d1ea0411484b0184b7c8431f4c
Author: Kenneth Knowles 
Date:   2017-05-03T03:05:45Z

Annotate internal-only bits of Java sdk.runners




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2145) Do not publish javadoc for Java SDK's transforms.reflect directory

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2145:
-

 Summary: Do not publish javadoc for Java SDK's transforms.reflect 
directory
 Key: BEAM-2145
 URL: https://issues.apache.org/jira/browse/BEAM-2145
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core, website
Reporter: Kenneth Knowles
Assignee: Davor Bonaci
Priority: Blocker
 Fix For: First stable release


This is where the utilities for analyzing a {{DoFn}} live. They need to be in 
the core SDK for {{DoFnTester}}, otherwise they would be in 
runner-core-construction.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2144) Do not publish javadoc for Java SDK's util directory

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2144:
-

 Summary: Do not publish javadoc for Java SDK's util directory
 Key: BEAM-2144
 URL: https://issues.apache.org/jira/browse/BEAM-2144
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core, website
Reporter: Kenneth Knowles
Assignee: Davor Bonaci
Priority: Blocker
 Fix For: First stable release


This directory is a collection of non-user-facing code. No one should be using 
it unless they are already digging around our codebase and are comfortable 
being broken without a major version increment.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3017

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2020) Move CloudObject to Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994195#comment-15994195
 ] 

ASF GitHub Bot commented on BEAM-2020:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2805


> Move CloudObject to Dataflow runner
> ---
>
> Key: BEAM-2020
> URL: https://issues.apache.org/jira/browse/BEAM-2020
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This entails primarily eliminating Coder.asCloudObject() by adding the needed 
> accessors, and possibly a serialization registrar discipline, for coders in 
> the Runner API proto.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2805: [BEAM-2020] Add additional CloudObject Translators

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2805


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: [BEAM-2020] Add additional CloudObject Translators

[lcwik]

[BEAM-2020] Add additional CloudObject Translators

This closes #2805


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dc0fdcb7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dc0fdcb7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dc0fdcb7

Branch: refs/heads/master
Commit: dc0fdcb7e62067fdc629b8558659fce7e37e4856
Parents: 2d22485 73cdd99
Author: Luke Cwik 
Authored: Tue May 2 19:47:48 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 2 19:47:48 2017 -0700

--
 .../dataflow/util/CloudObjectTranslators.java   | 199 +++
 ...aultCoderCloudObjectTranslatorRegistrar.java |  21 +-
 .../runners/dataflow/util/CloudObjectsTest.java |  33 ++-
 .../beam/sdk/transforms/join/CoGbkResult.java   |   8 +
 4 files changed, 256 insertions(+), 5 deletions(-)
--

[1/2] beam git commit: Add Additional CloudObjectTranslators

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 2d22485c1 -> dc0fdcb7e


Add Additional CloudObjectTranslators

Add IterableLikeCoders, MapCoder

Add UnionCoder, CoGbkResultCoder, and NullableCoder translators.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/73cdd994
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/73cdd994
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/73cdd994

Branch: refs/heads/master
Commit: 73cdd99466bef0c35158d4dd89ac10e9cb056782
Parents: 2d22485
Author: Thomas Groh 
Authored: Mon May 1 22:29:34 2017 -0700
Committer: Thomas Groh 
Committed: Tue May 2 18:07:00 2017 -0700

--
 .../dataflow/util/CloudObjectTranslators.java   | 199 +++
 ...aultCoderCloudObjectTranslatorRegistrar.java |  21 +-
 .../runners/dataflow/util/CloudObjectsTest.java |  33 ++-
 .../beam/sdk/transforms/join/CoGbkResult.java   |   8 +
 4 files changed, 256 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/73cdd994/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
index c27bee7..f3e3312 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
@@ -18,6 +18,8 @@
 
 package org.apache.beam.runners.dataflow.util;
 
+import static com.google.common.base.Preconditions.checkArgument;
+
 import com.google.common.collect.ImmutableList;
 import java.util.ArrayList;
 import java.util.Collections;
@@ -27,9 +29,15 @@ import org.apache.beam.sdk.coders.ByteArrayCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.IterableLikeCoder;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.LengthPrefixCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
 import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.transforms.join.CoGbkResult.CoGbkResultCoder;
+import org.apache.beam.sdk.transforms.join.CoGbkResultSchema;
+import org.apache.beam.sdk.transforms.join.UnionCoder;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import 
org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
 import org.apache.beam.sdk.util.CloudObject;
@@ -39,6 +47,7 @@ import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.StringUtils;
 import org.apache.beam.sdk.util.Structs;
 import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
+import org.apache.beam.sdk.values.TupleTag;
 
 /** Utilities for creating {@link CloudObjectTranslator} instances for {@link 
Coder Coders}. */
 class CloudObjectTranslators {
@@ -373,4 +382,194 @@ class CloudObjectTranslators {
   }
 };
   }
+  public static CloudObjectTranslator iterableLike(
+  final Class clazz) {
+return new CloudObjectTranslator() {
+  @Override
+  public CloudObject toCloudObject(IterableLikeCoder target) {
+CloudObject base = CloudObject.forClass(clazz);
+return addComponents(base, 
Collections.singletonList(target.getElemCoder()));
+  }
+
+  @Override
+  public IterableLikeCoder fromCloudObject(CloudObject cloudObject) {
+List elemCoderList = getComponents(cloudObject);
+checkArgument(
+elemCoderList.size() == 1,
+"Expected 1 component for %s, got %s",
+cloudObject.getClassName(),
+elemCoderList.size());
+return InstanceBuilder.ofType(clazz)
+.fromFactoryMethod("of")
+.withArg(Coder.class, elemCoderList.get(0))
+.build();
+  }
+
+  @Override
+  public Class getSupportedClass() {
+return clazz;
+  }
+
+  @Override
+  public String cloudObjectClassName() {
+return CloudObject.forClass(clazz).getClassName();
+  }
+};
+  }
+
+  public static CloudObjectTranslator map() {
+return new CloudObjectTranslator() {
+  @Override
+  public CloudObject toCloudObject(MapCoder target) {
+CloudObject base = CloudObject.forClass(MapCoder.class);
+return addComponents(
+base, 

[jira] [Resolved] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-828.
--
Resolution: Won't Fix

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994180#comment-15994180
 ] 

Kenneth Knowles commented on BEAM-828:
--

Current plan is not to necessarily remove them, but to use them only as a 
conduit for inaccessible ValueProviders. So that makes this a dupe, too.

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994180#comment-15994180
 ] 

Kenneth Knowles edited comment on BEAM-828 at 5/3/17 2:28 AM:
--

Current plan is not to necessarily remove them, but to use them only as a 
conduit for inaccessible ValueProviders. So that makes this a near-dupe, too, 
though not quite as currently phrased.


was (Author: kenn):
Current plan is not to necessarily remove them, but to use them only as a 
conduit for inaccessible ValueProviders. So that makes this a dupe, too.

> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-170) Session windows should not be identified by their bounds

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994178#comment-15994178
 ] 

Kenneth Knowles commented on BEAM-170:
--

I don't think I'm going to have bandwidth for this, necessarily. We can always 
add a new {{WindowFn}} with the desired behavior. It is a rare occurrence for 
users.

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-170) Session windows should not be identified by their bounds

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-170:


Assignee: (was: Kenneth Knowles)

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>  Labels: backward-incompatible
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1641) Implement real synchronized processing time in Flink runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1641:
--
Issue Type: New Feature  (was: Bug)

> Implement real synchronized processing time in Flink runner
> ---
>
> Key: BEAM-1641
> URL: https://issues.apache.org/jira/browse/BEAM-1641
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Priority: Minor
>
> The "continuation trigger" for a processing time trigger is a synchronized 
> processing time trigger. Today, this throws an exception in the FlinkRunner.
> The supports the following:
>  - GBK1
>  - GBK2
> When GBK1 fires due to processing time past the first element in the pane and 
> that element arrives at GBK2, it will wait until all the other upstream keys 
> have also processed and emitted corresponding data.
> Sorry for the terseness of explanation - writing quickly so I don't forget.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1641) Implement real synchronized processing time in Flink runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1641:
-

Assignee: (was: Kenneth Knowles)

> Implement real synchronized processing time in Flink runner
> ---
>
> Key: BEAM-1641
> URL: https://issues.apache.org/jira/browse/BEAM-1641
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Priority: Minor
>
> The "continuation trigger" for a processing time trigger is a synchronized 
> processing time trigger. Today, this throws an exception in the FlinkRunner.
> The supports the following:
>  - GBK1
>  - GBK2
> When GBK1 fires due to processing time past the first element in the pane and 
> that element arrives at GBK2, it will wait until all the other upstream keys 
> have also processed and emitted corresponding data.
> Sorry for the terseness of explanation - writing quickly so I don't forget.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2128) PostCommit Java_MavenInstall broken

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2128.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> PostCommit Java_MavenInstall broken
> ---
>
> Key: BEAM-2128
> URL: https://issues.apache.org/jira/browse/BEAM-2128
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, testing
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
> Fix For: Not applicable
>
>
> The test 
> {{org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoes}}
> is broken since PR #2666 was merged to master.
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/3533/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3604

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2091

[Apache Jenkins Server]

See 

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b76eb56b
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b76eb56b
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b76eb56b

Branch: refs/heads/asf-site
Commit: b76eb56b6986f99aa73ce38f8e719803ab18fb4e
Parents: bf62dc9
Author: Ahmet Altay 
Authored: Tue May 2 18:26:45 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 18:26:45 2017 -0700

--
 .../mobile-gaming-example/index.html| 161 ++-
 1 file changed, 157 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/b76eb56b/content/get-started/mobile-gaming-example/index.html
--
diff --git a/content/get-started/mobile-gaming-example/index.html 
b/content/get-started/mobile-gaming-example/index.html
index 054024c..7a3fbb4 100644
--- a/content/get-started/mobile-gaming-example/index.html
+++ b/content/get-started/mobile-gaming-example/index.html
@@ -188,12 +188,24 @@
   
 
 
+
+  Adapt for: 
+  
+Java SDK
+Python SDK
+  
+
+
 This section provides a walkthrough of a series of example Apache Beam 
pipelines that demonstrate more complex functionality than the basic WordCount examples. The pipelines in 
this section process data from a hypothetical game that users play on their 
mobile phones. The pipelines demonstrate processing at increasing levels of 
complexity; the first pipeline, for example, shows how to run a batch analysis 
job to obtain relatively simple score data, while the later pipelines use 
Beam’s windowing and triggers features to provide low-latency data analysis 
and more complex intelligence about user’s play patterns.
 
-
+
   Note: These examples assume some familiarity with the 
Beam programming model. If you haven’t already, we recommend familiarizing 
yourself with the programming model documentation and running a basic example 
pipeline before continuing. Note also that these examples use the Java 8 lambda 
syntax, and thus require Java 8. However, you can create pipelines with 
equivalent functionality using Java 7.
 
 
+
+  Note: These examples assume some familiarity with the 
Beam programming model. If you haven’t already, we recommend familiarizing 
yourself with the programming model documentation and running a basic example 
pipeline before continuing.
+
+
 Every time a user plays an instance of our hypothetical mobile game, they 
generate a data event. Each data event consists of the following 
information:
 
 
@@ -224,10 +236,14 @@
 
 The UserScore pipeline is the 
simplest example for processing mobile game data. UserScore determines the total score per user 
over a finite data set (for example, one day’s worth of scores stored on the 
game server). Pipelines like UserScore 
are best run periodically after all relevant data has been gathered. For 
example, UserScore could run as a 
nightly job over data gathered during that day.
 
-
+
   Note: See https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java;>UserScore
 on GitHub for the complete example pipeline program.
 
 
+
+  Note: See https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/user_score.py;>UserScore
 on GitHub for the complete example pipeline program.
+
+
 What Does UserScore Do?
 
 In a day’s worth of scoring data, each user ID may have multiple records 
(if the user plays more than one instance of the game during the analysis 
window), each with their own score value and timestamp. If we want to determine 
the total score over all the instances a user plays during the day, our 
pipeline will need to group all the records together per individual user.
@@ -283,6 +299,28 @@
 
 
 
+class ExtractAndSumScore(beam.PTransform):
+  """A transform to extract key/score information and sum the 
scores.
+  The constructor argument `field` determines whether 'team' or 'user' info is
+  extracted.
+  """
+  def __init__(self, field):
+super(ExtractAndSumScore, self).__init__()
+self.field = field
+
+  def expand(self, pcoll):
+return (pcoll
+| beam.Map(lambda info: 
(info[self.field], info['score']))
+| beam.CombinePerKey(sum_ints))
+
+def configure_bigquery_write():
+  return [
+  ('user', 'STRING', 
lambda e: 
e[0]),
+  ('total_score', 'INTEGER', 
lambda e: 
e[1]),
+  ]
+
+
+
 ExtractAndSumScore is written to be 
more general, in that you can pass in the field by which you want to group the 
data (in the case of our game, by unique user or unique team). This means we 
can re-use ExtractAndSumScore in other 
pipelines that group score data 

[jira] [Created] (BEAM-2143) (Mis)Running Dataflow Wordcount gives non-helpful errors

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-2143:
--

 Summary: (Mis)Running Dataflow Wordcount gives non-helpful errors
 Key: BEAM-2143
 URL: https://issues.apache.org/jira/browse/BEAM-2143
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Ben Chambers
Assignee: Daniel Halperin


If you run a pipeline and forget to specify `tempLocation` (but specify 
something else, such as `stagingLocation`) you get two messages indicating you 
forgot to specify `stagingLocation`. 

One says "no stagingLocation specified, choosing ..." the other says "error, 
the staging location isn't readable" (if you give it just a bucket and not an 
object within a bucket).

This is surprising to me as a user, since (1) I specified a staging location 
and (2) the flag I actually need to modify is `--tempLocation`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam-site git commit: This closes #220

[altay]

This closes #220


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/3cafa86a
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/3cafa86a
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/3cafa86a

Branch: refs/heads/asf-site
Commit: 3cafa86a038320ad56c7ed04c4b1be2d04f606f7
Parents: 496f240 b76eb56
Author: Ahmet Altay 
Authored: Tue May 2 18:26:46 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 18:26:46 2017 -0700

--
 .../mobile-gaming-example/index.html| 161 ++-
 src/get-started/mobile-gaming-example.md| 155 +-
 2 files changed, 310 insertions(+), 6 deletions(-)
--

[1/3] beam-site git commit: Add Python snippets to Mobile Gaming.

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 496f24014 -> 3cafa86a0


Add Python snippets to Mobile Gaming.


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/bf62dc99
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/bf62dc99
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/bf62dc99

Branch: refs/heads/asf-site
Commit: bf62dc99069178c440b867a69042b05dda0cc439
Parents: 496f240
Author: Hadar Hod 
Authored: Fri Apr 21 09:36:57 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 18:23:32 2017 -0700

--
 src/get-started/mobile-gaming-example.md | 155 +-
 1 file changed, 153 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/bf62dc99/src/get-started/mobile-gaming-example.md
--
diff --git a/src/get-started/mobile-gaming-example.md 
b/src/get-started/mobile-gaming-example.md
index 7519d3b..5c97274 100644
--- a/src/get-started/mobile-gaming-example.md
+++ b/src/get-started/mobile-gaming-example.md
@@ -10,9 +10,21 @@ redirect_from: /use/mobile-gaming-example/
 * TOC
 {:toc}
 
+
+  Adapt for: 
+  
+Java SDK
+Python SDK
+  
+
+
 This section provides a walkthrough of a series of example Apache Beam 
pipelines that demonstrate more complex functionality than the basic 
[WordCount]({{ site.baseurl }}/get-started/wordcount-example) examples. The 
pipelines in this section process data from a hypothetical game that users play 
on their mobile phones. The pipelines demonstrate processing at increasing 
levels of complexity; the first pipeline, for example, shows how to run a batch 
analysis job to obtain relatively simple score data, while the later pipelines 
use Beam's windowing and triggers features to provide low-latency data analysis 
and more complex intelligence about user's play patterns.
 
-> **Note**: These examples assume some familiarity with the Beam programming 
model. If you haven't already, we recommend familiarizing yourself with the 
programming model documentation and running a basic example pipeline before 
continuing. Note also that these examples use the Java 8 lambda syntax, and 
thus require Java 8. However, you can create pipelines with equivalent 
functionality using Java 7.
+{:.language-java}
+> **Note**: These examples assume some familiarity with the Beam programming 
model. If you haven't already, we recommend familiarizing yourself with the 
programming model documentation and running a basic example pipeline before 
continuing. Note also that these examples use the Java 8 lambda syntax, and 
thus require Java 8. However, you can create pipelines with equivalent 
functionality using Java 7. 
+
+{:.language-py}
+> **Note**: These examples assume some familiarity with the Beam programming 
model. If you haven't already, we recommend familiarizing yourself with the 
programming model documentation and running a basic example pipeline before 
continuing.
 
 Every time a user plays an instance of our hypothetical mobile game, they 
generate a data event. Each data event consists of the following information:
 
@@ -44,8 +56,12 @@ The Mobile Game example pipelines vary in complexity, from 
simple batch analysis
 
 The `UserScore` pipeline is the simplest example for processing mobile game 
data. `UserScore` determines the total score per user over a finite data set 
(for example, one day's worth of scores stored on the game server). Pipelines 
like `UserScore` are best run periodically after all relevant data has been 
gathered. For example, `UserScore` could run as a nightly job over data 
gathered during that day.
 
+{:.language-java}
 > **Note:** See [UserScore on 
 > GitHub](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java)
 >  for the complete example pipeline program.
 
+{:.language-py}
+> **Note:** See [UserScore on 
GitHub](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/user_score.py)
 for the complete example pipeline program.
+
 ### What Does UserScore Do?
 
 In a day's worth of scoring data, each user ID may have multiple records (if 
the user plays more than one instance of the game during the analysis window), 
each with their own score value and timestamp. If we want to determine the 
total score over all the instances a user plays during the day, our pipeline 
will need to group all the records together per individual user.
@@ -99,6 +115,28 @@ public static class ExtractAndSumScore
 }
 ```
 
+```py
+class ExtractAndSumScore(beam.PTransform):
+  """A transform to extract key/score information and sum the scores.
+  The constructor argument 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3016

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #3602

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1806) a new option `asLeftOuterJoin` for CoGroupByKey

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-1806.
--
   Resolution: Later
Fix Version/s: Not applicable

move to BEAM-1987

> a new option `asLeftOuterJoin` for CoGroupByKey
> ---
>
> Key: BEAM-1806
> URL: https://issues.apache.org/jira/browse/BEAM-1806
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> Similar as BEAM-1805, restrict it as left-outer-join. 
> The first {{PCollection}} is used as the key, if it's empty, output is 
> ignored.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1805) a new option `asInnerJoin` for CoGroupByKey

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994134#comment-15994134
 ] 

ASF GitHub Bot commented on BEAM-1805:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/2339


> a new option `asInnerJoin` for CoGroupByKey
> ---
>
> Key: BEAM-1805
> URL: https://issues.apache.org/jira/browse/BEAM-1805
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> {{CoGroupByKey}} joins multiple PCollection>, act as full-outer join.
> Option {{asInnerJoin()}} restrict the output to convert to an inner-join 
> behavior.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2339: [BEAM-1805] a new option `asInnerJoin` for CoGroupB...

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/2339


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1805) a new option `asInnerJoin` for CoGroupByKey

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-1805.
--
   Resolution: Later
Fix Version/s: Not applicable

move to BEAM-1987

> a new option `asInnerJoin` for CoGroupByKey
> ---
>
> Key: BEAM-1805
> URL: https://issues.apache.org/jira/browse/BEAM-1805
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> {{CoGroupByKey}} joins multiple PCollection>, act as full-outer join.
> Option {{asInnerJoin()}} restrict the output to convert to an inner-join 
> behavior.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2845

[altay]

This closes #2845


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2d22485c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2d22485c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2d22485c

Branch: refs/heads/master
Commit: 2d22485c18cf6adcb8b4f7dc25c13e575d1cd57a
Parents: d7151fb 7a48ddd
Author: Ahmet Altay 
Authored: Tue May 2 18:01:12 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 18:01:12 2017 -0700

--
 .../clients/dataflow/dataflow_v1b3_messages.py  | 564 +--
 1 file changed, 528 insertions(+), 36 deletions(-)
--

[GitHub] beam pull request #2845: Updating Dataflow API protos and client

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2845


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2020) Move CloudObject to Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994126#comment-15994126
 ] 

ASF GitHub Bot commented on BEAM-2020:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2806


> Move CloudObject to Dataflow runner
> ---
>
> Key: BEAM-2020
> URL: https://issues.apache.org/jira/browse/BEAM-2020
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This entails primarily eliminating Coder.asCloudObject() by adding the needed 
> accessors, and possibly a serialization registrar discipline, for coders in 
> the Runner API proto.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Updating Dataflow API protos and client

[altay]

Repository: beam
Updated Branches:
  refs/heads/master d7151fb2b -> 2d22485c1


Updating Dataflow API protos and client


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7a48ddda
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7a48ddda
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7a48ddda

Branch: refs/heads/master
Commit: 7a48ddda873eb5c2b3d616a3d8bc225a576cd4cb
Parents: d7151fb
Author: Pablo 
Authored: Tue May 2 14:23:44 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 18:01:08 2017 -0700

--
 .../clients/dataflow/dataflow_v1b3_messages.py  | 564 +--
 1 file changed, 528 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7a48ddda/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
--
diff --git 
a/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 
b/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
index 4f1ccdb..eb88bce 100644
--- 
a/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
+++ 
b/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
@@ -101,6 +101,60 @@ class ApproximateSplitRequest(_messages.Message):
   position = _messages.MessageField('Position', 2)
 
 
+class AutoscalingEvent(_messages.Message):
+  """A structured message reporting an autoscaling decision made by the
+  Dataflow service.
+
+  Enums:
+EventTypeValueValuesEnum: The type of autoscaling event to report.
+
+  Fields:
+currentNumWorkers: The current number of workers the job has.
+description: A message describing why the system decided to adjust the
+  current number of workers, why it failed, or why the system decided to
+  not make any changes to the number of workers.
+eventType: The type of autoscaling event to report.
+targetNumWorkers: The target number of workers the worker pool wants to
+  resize to use.
+time: The time this event was emitted to indicate a new target or current
+  num_workers value.
+  """
+
+  class EventTypeValueValuesEnum(_messages.Enum):
+"""The type of autoscaling event to report.
+
+Values:
+  TYPE_UNKNOWN: Default type for the enum.  Value should never be
+returned.
+  TARGET_NUM_WORKERS_CHANGED: The TARGET_NUM_WORKERS_CHANGED type should
+be used when the target worker pool size has changed at the start of
+an actuation. An event should always be specified as
+TARGET_NUM_WORKERS_CHANGED if it reflects a change in the
+target_num_workers.
+  CURRENT_NUM_WORKERS_CHANGED: The CURRENT_NUM_WORKERS_CHANGED type should
+be used when actual worker pool size has been changed, but the
+target_num_workers has not changed.
+  ACTUATION_FAILURE: The ACTUATION_FAILURE type should be used when we
+want to report an error to the user indicating why the current number
+of workers in the pool could not be changed. Displayed in the current
+status and history widgets.
+  NO_CHANGE: Used when we want to report to the user a reason why we are
+not currently adjusting the number of workers. Should specify both
+target_num_workers, current_num_workers and a decision_message.
+"""
+TYPE_UNKNOWN = 0
+TARGET_NUM_WORKERS_CHANGED = 1
+CURRENT_NUM_WORKERS_CHANGED = 2
+ACTUATION_FAILURE = 3
+NO_CHANGE = 4
+
+  currentNumWorkers = _messages.IntegerField(1)
+  description = _messages.MessageField('StructuredMessage', 2)
+  eventType = _messages.EnumField('EventTypeValueValuesEnum', 3)
+  targetNumWorkers = _messages.IntegerField(4)
+  time = _messages.StringField(5)
+
+
 class AutoscalingSettings(_messages.Message):
   """Settings for WorkerPool autoscaling.
 
@@ -406,6 +460,7 @@ class CreateJobFromTemplateRequest(_messages.Message):
   create the job. Must be a valid Cloud Storage URL, beginning with
   `gs://`.
 jobName: Required. The job name to use for the created job.
+location: The location to which to direct the request.
 parameters: The runtime parameters to pass to the job.
   """
 
@@ -436,7 +491,8 @@ class CreateJobFromTemplateRequest(_messages.Message):
   environment = _messages.MessageField('RuntimeEnvironment', 1)
   gcsPath = _messages.StringField(2)
   jobName = _messages.StringField(3)
-  parameters = _messages.MessageField('ParametersValue', 4)
+  location = _messages.StringField(4)
+  parameters = _messages.MessageField('ParametersValue', 5)
 
 
 class 

[GitHub] beam pull request #2806: [BEAM-2020] Remove Coder.asCloudObject and related ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2806


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2142) add missing composite transforms Python snippets to programming guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994125#comment-15994125
 ] 

ASF GitHub Bot commented on BEAM-2142:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/227


> add missing composite transforms Python snippets to programming guide
> -
>
> Key: BEAM-2142
> URL: https://issues.apache.org/jira/browse/BEAM-2142
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: [BEAM-2020] Remove Coder.asCloudObject and related methods

[lcwik]

[BEAM-2020] Remove Coder.asCloudObject and related methods

This closes #2806


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d7151fb2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d7151fb2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d7151fb2

Branch: refs/heads/master
Commit: d7151fb2b80b6641071cb56383c2076ab1ac2da1
Parents: 30e6116 9495048
Author: Luke Cwik 
Authored: Tue May 2 17:59:59 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 2 17:59:59 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  |  2 +-
 .../dataflow/DataflowPipelineTranslator.java|  6 +--
 .../java/org/apache/beam/sdk/coders/Coder.java  |  7 ---
 .../org/apache/beam/sdk/coders/CustomCoder.java | 48 
 .../apache/beam/sdk/coders/IterableCoder.java   |  9 
 .../org/apache/beam/sdk/coders/KvCoder.java |  9 
 .../beam/sdk/coders/LengthPrefixCoder.java  |  6 ---
 .../apache/beam/sdk/coders/StructuredCoder.java | 43 --
 .../beam/sdk/testing/CoderProperties.java   |  5 +-
 .../sdk/transforms/windowing/GlobalWindow.java  |  6 ---
 .../transforms/windowing/IntervalWindow.java|  6 ---
 .../apache/beam/sdk/util/SerializableUtils.java | 26 ---
 .../org/apache/beam/sdk/util/WindowedValue.java | 15 --
 .../beam/sdk/coders/IterableCoderTest.java  | 10 
 .../org/apache/beam/sdk/coders/KvCoderTest.java | 11 -
 .../transforms/windowing/GlobalWindowTest.java  |  7 ---
 .../apache/beam/sdk/util/WindowedValueTest.java | 10 
 sdks/java/extensions/protobuf/pom.xml   | 10 
 .../sdk/extensions/protobuf/ProtoCoder.java | 29 
 19 files changed, 14 insertions(+), 251 deletions(-)
--

[1/2] beam git commit: Remove Coder.asCloudObject and related methods

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 30e611646 -> d7151fb2b


Remove Coder.asCloudObject and related methods

These methods belong in the Dataflow module.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/94950484
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/94950484
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/94950484

Branch: refs/heads/master
Commit: 94950484a416afb2d250c4b4b45e20a16a3bbb9d
Parents: 30e6116
Author: Thomas Groh <tg...@google.com>
Authored: Mon May 1 13:03:18 2017 -0700
Committer: Luke Cwik <lc...@google.com>
Committed: Tue May 2 17:59:25 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  |  2 +-
 .../dataflow/DataflowPipelineTranslator.java|  6 +--
 .../java/org/apache/beam/sdk/coders/Coder.java  |  7 ---
 .../org/apache/beam/sdk/coders/CustomCoder.java | 48 
 .../apache/beam/sdk/coders/IterableCoder.java   |  9 
 .../org/apache/beam/sdk/coders/KvCoder.java |  9 
 .../beam/sdk/coders/LengthPrefixCoder.java  |  6 ---
 .../apache/beam/sdk/coders/StructuredCoder.java | 43 --
 .../beam/sdk/testing/CoderProperties.java   |  5 +-
 .../sdk/transforms/windowing/GlobalWindow.java  |  6 ---
 .../transforms/windowing/IntervalWindow.java|  6 ---
 .../apache/beam/sdk/util/SerializableUtils.java | 26 ---
 .../org/apache/beam/sdk/util/WindowedValue.java | 15 --
 .../beam/sdk/coders/IterableCoderTest.java  | 10 
 .../org/apache/beam/sdk/coders/KvCoderTest.java | 11 -
 .../transforms/windowing/GlobalWindowTest.java  |  7 ---
 .../apache/beam/sdk/util/WindowedValueTest.java | 10 
 sdks/java/extensions/protobuf/pom.xml   | 10 
 .../sdk/extensions/protobuf/ProtoCoder.java | 29 
 19 files changed, 14 insertions(+), 251 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/94950484/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index c0b6328..bbad156 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170501-pr2718
+    
beam-master-20170502
 
1
 
6
   

http://git-wip-us.apache.org/repos/asf/beam/blob/94950484/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 0c0a2ef..28a9c1c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -62,6 +62,7 @@ import 
org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory.ParDoSingle;
 import 
org.apache.beam.runners.dataflow.TransformTranslator.StepTranslationContext;
 import org.apache.beam.runners.dataflow.TransformTranslator.TranslationContext;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import org.apache.beam.runners.dataflow.util.CloudObjects;
 import org.apache.beam.runners.dataflow.util.DoFnInfo;
 import org.apache.beam.runners.dataflow.util.OutputReference;
 import org.apache.beam.sdk.Pipeline;
@@ -87,7 +88,6 @@ import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.AppliedCombineFn;
 import org.apache.beam.sdk.util.CloudObject;
 import org.apache.beam.sdk.util.PropertyNames;
-import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.KV;
@@ -557,7 +557,7 @@ public class DataflowPipelineTranslator {
 
 @Override
 public void addEncodingInput(Coder coder) {
-  CloudObject encoding = SerializableUtils.ensureSerializable(coder);
+  CloudObject encoding = CloudObjects.asCloudObject(coder);
   addObject(getProperties(), PropertyNames.ENCODING, encoding);
 }
 
@@ -669,7 +669,7 @@ public class DataflowPipelineTranslator {
   if (valueCoder != null) {
 // Verify that encoding can be decoded, in order to catch serialization
 // failures as early as possible.
-CloudObject encoding = 
SerializableUtils.ensureSerializable(valueCoder);
+CloudObject encoding = CloudObject

[GitHub] beam-site pull request #227: [BEAM-2142] Add composite transforms Python sni...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/227


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/5a7282fe
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/5a7282fe
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/5a7282fe

Branch: refs/heads/asf-site
Commit: 5a7282fe42e9a396673aacfac8f466284dffa1ea
Parents: 6289505
Author: Ahmet Altay 
Authored: Tue May 2 17:59:11 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 17:59:11 2017 -0700

--
 content/documentation/io/built-in/index.html|  2 +-
 .../documentation/programming-guide/index.html  | 46 +---
 .../sdks/python-type-safety/index.html  | 10 ++---
 3 files changed, 45 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/io/built-in/index.html
--
diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index 639a877..cf262d3 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -170,7 +170,7 @@
   Java
   
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
-https://github.com/apache/beam/tree/master/sdks/java/io/hdfs;>Apache 
Hadoop HDFS
+https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system;>Apache
 Hadoop File System
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java;>TextIO
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/;>XML
   

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index b57f89c..5ad4bea 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -1200,18 +1200,18 @@ guest, [[], [order4]]
   yield element
 else:
   # Emit this word's long length to the 
'above_cutoff_lengths' output.
-  yield pvalue.OutputValue(
+  yield pvalue.TaggedOutput(
   'above_cutoff_lengths', len(element))
 if element.startswith(marker):
   # Emit this word to a different output with the 'marked 
strings' tag.
-  yield pvalue.OutputValue('marked strings', element)
+  yield pvalue.TaggedOutput('marked strings', element)
 
 
 # Producing multiple outputs is also available in Map and 
FlatMap.
 # Here is an example that uses FlatMap and shows that the tags 
do not need to be specified ahead of time.
 
 def even_odd(x):
-  yield pvalue.OutputValue('odd' if x % 2 else 'even', 
x)
+  yield pvalue.TaggedOutput('odd' if x % 2 else 'even', 
x)
   if x % 
10 == 0:
 yield x
 
@@ -1267,7 +1267,17 @@ guest, [[], [order4]]
 
 
 
-  
Python code snippet coming soon (BEAM-1926)
+# The CountWords Composite Transform inside the WordCount 
pipeline.
+class CountWords(beam.PTransform):
+
+  def expand(self, pcoll):
+return (pcoll
+# Convert lines of text into individual 
words.
+| 'ExtractWords' 
 beam.ParDo(ExtractWordsFn())
+# Count the number of times each word 
occurs.
+| beam.combiners.Count.PerElement()
+# Format each word and count into a printable 
string.
+| 'FormatCounts' 
 beam.ParDo(FormatCountsFn()))
 
 
 
@@ -1286,7 +1296,10 @@ guest, [[], [order4]]
 
 
 
-  
Python code snippet coming soon (BEAM-1926)
+class ComputeWordLengths(beam.PTransform):
+  def expand(self, pcoll):
+# transform logic goes here
+return pcoll | beam.Map(lambda x: len(x))
 
 
 
@@ -1307,7 +1320,10 @@ guest, [[], [order4]]
 
 
 
-  
Python code snippet coming soon (BEAM-1926)
+class ComputeWordLengths(beam.PTransform):
+  def expand(self, pcoll):
+# transform logic goes here
+return pcoll | beam.Map(lambda x: len(x))
 
 
 
@@ -1920,7 +1936,7 @@ Subsequent transforms, however, are applied to the result 
of the   
The Beam SDK for Python 
does not support triggers.
+  
# The Beam SDK for Python does not support triggers.
 
 
 
@@ -1956,6 +1972,10 @@ Subsequent transforms, however, are applied to the 
result of the   
# The Beam SDK for Python does not support triggers.
+
+
+
 This code sample sets a time-based trigger for a PCollection, which emits results one minute 
after the first element in that window has been processed. The last line in the 
code sample, 

[3/3] beam-site git commit: This closes #227

[altay]

This closes #227


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/496f2401
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/496f2401
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/496f2401

Branch: refs/heads/asf-site
Commit: 496f24014f544ed60f211eaf47722fd0e3be02de
Parents: a0e95ba 5a7282f
Author: Ahmet Altay 
Authored: Tue May 2 17:59:12 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 17:59:12 2017 -0700

--
 content/documentation/io/built-in/index.html|  2 +-
 .../documentation/programming-guide/index.html  | 46 +---
 .../sdks/python-type-safety/index.html  | 10 ++---
 src/documentation/programming-guide.md  | 30 ++---
 4 files changed, 68 insertions(+), 20 deletions(-)
--

[1/3] beam-site git commit: [BEAM-2142] Add composite transforms Python snippets to programming guide

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site a0e95bae3 -> 496f24014


[BEAM-2142] Add composite transforms Python snippets to programming guide


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/62895054
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/62895054
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/62895054

Branch: refs/heads/asf-site
Commit: 62895054e5915cdde9192fc6a866a58790e93c4f
Parents: a0e95ba
Author: melissa 
Authored: Tue May 2 15:49:41 2017 -0700
Committer: Ahmet Altay 
Committed: Tue May 2 17:55:55 2017 -0700

--
 src/documentation/programming-guide.md | 30 ++---
 1 file changed, 23 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/62895054/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 8e8b917..e3cd2d9 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -983,8 +983,8 @@ Your composite transform's parameters and return value must 
match the initial in
 ```
 
 ```py
-  Python code snippet coming soon (BEAM-1926)
-```
+{% github_sample 
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py 
tag:pipeline_monitoring_composite
+%}```
 
 ### Creating a composite transform
 
@@ -1003,8 +1003,8 @@ The following code sample shows how to declare a 
`PTransform` that accepts a `PC
 ```
 
 ```py
-  Python code snippet coming soon (BEAM-1926)
-```
+{% github_sample 
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py
 tag:model_composite_transform
+%}```
 
  Overriding the expand method
 
@@ -1024,8 +1024,8 @@ The following code sample shows how to override `expand` 
for the `ComputeWordLen
 ```
 
 ```py
-  Python code snippet coming soon (BEAM-1926)
-```
+{% github_sample 
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py
 tag:model_composite_transform
+%}```
 
 As long as you override the `expand` method in your `PTransform` subclass to 
accept the appropriate input `PCollection`(s) and return the corresponding 
output `PCollection`(s), you can include as many transforms as you want. These 
transforms can include core transforms, composite transforms, or the transforms 
included in the Beam SDK libraries.
 
@@ -1620,7 +1620,7 @@ The following example shows a billing scenario, and uses 
both early and late fir
 ```
 
 ```py
-  The Beam SDK for Python does not support triggers.
+  # The Beam SDK for Python does not support triggers.
 ```
 
 # **Default Trigger**
@@ -1655,6 +1655,10 @@ You set the trigger(s) for a `PCollection` by invoking 
the method `.triggering()
.discardingFiredPanes());
 ```
 
+```py
+  # The Beam SDK for Python does not support triggers.
+```
+
 This code sample sets a time-based trigger for a `PCollection`, which emits 
results one minute after the first element in that window has been processed. 
The last line in the code sample, `.discardingFiredPanes()`, is the window's 
**accumulation mode**.
 
  Window Accumulation Modes
@@ -1707,6 +1711,10 @@ You set the allowed lateness by using 
`.withAllowedLateness()` when you set your
   
.withAllowedLateness(Duration.standardMinutes(30));
 ```
 
+```py
+  # The Beam SDK for Python does not support triggers.
+```
+
 This allowed lateness propagates to all `PCollection`s derived as a result of 
applying transforms to the original `PCollection`. If you want to change the 
allowed lateness later in your pipeline, you can apply 
`Window.withAllowedLateness()` again, explicitly.
 
 
@@ -1748,6 +1756,10 @@ You can express this pattern using 
`AfterWatermark.pastEndOfWindow`. For example
   .withAllowedLateness(Duration.standardDays(2)));
 ```
 
+```py
+  # The Beam SDK for Python does not support triggers.
+```
+
  Other Composite Triggers
 
 You can also build other sorts of composite triggers. The following example 
code shows a simple composite trigger that fires whenever the pane has at least 
100 elements, or after a minute.
@@ -1758,3 +1770,7 @@ You can also build other sorts of composite triggers. The 
following example code
   
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardMinutes(1
 ```
 
+```py
+  # The Beam SDK for Python does not support triggers.
+```
+

[GitHub] beam pull request #2851: [BEAM 775] Removal of Aggregator class. Also remova...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2851

[BEAM 775] Removal of Aggregator class. Also removal from comments.

To be merged after the worker is rebuilt.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam remove-agg-class

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2851.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2851


commit 16cb77d418ccd2c83e96b4ffb22b0713ce59721b
Author: Pablo 
Date:   2017-05-03T00:50:20Z

Removal of Aggregator class. Also removal from comments.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2124) Deprecate .options usage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994108#comment-15994108
 ] 

ASF GitHub Bot commented on BEAM-2124:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2801


> Deprecate .options usage
> --
>
> Key: BEAM-2124
> URL: https://issues.apache.org/jira/browse/BEAM-2124
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py
>Reporter: María GH
>Assignee: María GH
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Deprecate .options usage

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 3bb0f8e6a -> 30e611646


Deprecate .options usage


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/444da273
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/444da273
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/444da273

Branch: refs/heads/master
Commit: 444da273ac8d01f2343eef76d8d4de5b0b78b409
Parents: 3bb0f8e
Author: Maria Garcia Herrero 
Authored: Sat Apr 29 09:43:55 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue May 2 17:36:56 2017 -0700

--
 .../examples/snippets/snippets_test.py  |  11 +-
 sdks/python/apache_beam/pipeline.py |  25 +++--
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 .../runners/dataflow/dataflow_runner.py |   7 +-
 .../runners/dataflow/dataflow_runner_test.py|   4 +-
 .../runners/dataflow/template_runner_test.py|   2 +-
 .../runners/dataflow/test_dataflow_runner.py|   4 +-
 .../apache_beam/runners/direct/direct_runner.py |   4 +-
 sdks/python/apache_beam/transforms/core.py  |   2 +-
 .../apache_beam/transforms/ptransform_test.py   | 102 +--
 10 files changed, 84 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/444da273/sdks/python/apache_beam/examples/snippets/snippets_test.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py 
b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index afd7918..370b436 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -31,7 +31,7 @@ from apache_beam import pvalue
 from apache_beam import typehints
 from apache_beam.transforms.util import assert_that
 from apache_beam.transforms.util import equal_to
-from apache_beam.utils.pipeline_options import TypeOptions
+from apache_beam.utils.pipeline_options import PipelineOptions
 from apache_beam.examples.snippets import snippets
 
 # pylint: disable=expression-not-assigned
@@ -245,10 +245,9 @@ class ParDoTest(unittest.TestCase):
 class TypeHintsTest(unittest.TestCase):
 
   def test_bad_types(self):
-p = TestPipeline()
-evens = None  # pylint: disable=unused-variable
-
 # [START type_hints_missing_define_numbers]
+p = TestPipeline(options=PipelineOptions(pipeline_type_check=True))
+
 numbers = p | beam.Create(['1', '2', '3'])
 # [END type_hints_missing_define_numbers]
 
@@ -269,7 +268,6 @@ class TypeHintsTest(unittest.TestCase):
 # To catch this early, we can assert what types we expect.
 with self.assertRaises(typehints.TypeCheckError):
   # [START type_hints_takes]
-  p.options.view_as(TypeOptions).pipeline_type_check = True
   evens = numbers | beam.Filter(lambda x: x % 2 == 0).with_input_types(int)
   # [END type_hints_takes]
 
@@ -315,10 +313,9 @@ class TypeHintsTest(unittest.TestCase):
 
   def test_runtime_checks_on(self):
 # pylint: disable=expression-not-assigned
-p = TestPipeline()
+p = TestPipeline(options=PipelineOptions(runtime_type_check=True))
 with self.assertRaises(typehints.TypeCheckError):
   # [START type_hints_runtime_on]
-  p.options.view_as(TypeOptions).runtime_type_check = True
   p | beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str)
   p.run()
   # [END type_hints_runtime_on]

http://git-wip-us.apache.org/repos/asf/beam/blob/444da273/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 100c50a..9200363 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -64,6 +64,7 @@ from apache_beam.utils.pipeline_options import SetupOptions
 from apache_beam.utils.pipeline_options import StandardOptions
 from apache_beam.utils.pipeline_options import TypeOptions
 from apache_beam.utils.pipeline_options_validator import 
PipelineOptionsValidator
+from apache_beam.utils.annotations import deprecated
 
 
 class Pipeline(object):
@@ -94,25 +95,24 @@ class Pipeline(object):
   ValueError: if either the runner or options argument is not of the
   expected type.
 """
-
 if options is not None:
   if isinstance(options, PipelineOptions):
-self.options = options
+self._options = options
   else:
 raise ValueError(
 'Parameter options, if specified, must be of type PipelineOptions. 
'
 'Received : %r', options)
 elif argv is not None:
   if isinstance(argv, list):
-self.options = PipelineOptions(argv)
+self._options = 

[2/2] beam git commit: Closes #2801

[robertwb]

Closes #2801


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/30e61164
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/30e61164
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/30e61164

Branch: refs/heads/master
Commit: 30e611646c7278d2439b02471a388843ad87ebde
Parents: 3bb0f8e 444da27
Author: Robert Bradshaw 
Authored: Tue May 2 17:37:12 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue May 2 17:37:12 2017 -0700

--
 .../examples/snippets/snippets_test.py  |  11 +-
 sdks/python/apache_beam/pipeline.py |  25 +++--
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 .../runners/dataflow/dataflow_runner.py |   7 +-
 .../runners/dataflow/dataflow_runner_test.py|   4 +-
 .../runners/dataflow/template_runner_test.py|   2 +-
 .../runners/dataflow/test_dataflow_runner.py|   4 +-
 .../apache_beam/runners/direct/direct_runner.py |   4 +-
 sdks/python/apache_beam/transforms/core.py  |   2 +-
 .../apache_beam/transforms/ptransform_test.py   | 102 +--
 10 files changed, 84 insertions(+), 79 deletions(-)
--

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[Chaoran Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994104#comment-15994104
 ] 

Chaoran Yu commented on BEAM-302:
-

After including SparkRunner dependency, I got the following exception 
"java.lang.NoClassDefFoundError: 
org/apache/spark/streaming/api/java/JavaStreamingContextFactory", which is a 
class that has ceased to exist for Spark 2.0+. I'm sure by digging deeper, the 
word count example can be run with SparkRunner. But my idea is that currently 
it's still not as simple as it should be. Spark runner, or basically most other 
runners supported by Beam, should receive the same support from scio as is the 
case for GCD runner. Is this an aspect where Spotify would like to see 
contributions from the community?

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1746) Include LICENSE and NOTICE in python dist files

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1746.
---
Resolution: Fixed

> Include LICENSE and NOTICE in python dist files
> ---
>
> Key: BEAM-1746
> URL: https://issues.apache.org/jira/browse/BEAM-1746
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.8 (latest),beam #5

[Apache Jenkins Server]

See 


--
[...truncated 1.21 MB...]
2017-05-03\T\00:26:23.061 [INFO] 2017-05-03\T\00:26:23.061 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
2017-05-03\T\00:26:23.087 [INFO] 2017-05-03\T\00:26:23.087 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
 (3 KB at 77.0 KB/sec)
2017-05-03\T\00:26:23.107 [INFO] 2017-05-03\T\00:26:23.107 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
2017-05-03\T\00:26:23.149 [INFO] 2017-05-03\T\00:26:23.149 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
 (3 KB at 62.8 KB/sec)
2017-05-03\T\00:26:23.153 [INFO] 2017-05-03\T\00:26:23.153 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
2017-05-03\T\00:26:23.180 [INFO] 2017-05-03\T\00:26:23.179 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
 (2 KB at 68.8 KB/sec)
2017-05-03\T\00:26:23.184 [INFO] 2017-05-03\T\00:26:23.184 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
2017-05-03\T\00:26:23.211 [INFO] 2017-05-03\T\00:26:23.211 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
 (2 KB at 71.3 KB/sec)
2017-05-03\T\00:26:23.255 [INFO] 2017-05-03\T\00:26:23.255 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
2017-05-03\T\00:26:23.284 [INFO] 2017-05-03\T\00:26:23.283 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
 (8 KB at 262.2 KB/sec)
2017-05-03\T\00:26:23.289 [INFO] 2017-05-03\T\00:26:23.289 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
2017-05-03\T\00:26:23.316 [INFO] 2017-05-03\T\00:26:23.316 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
 (2 KB at 71.0 KB/sec)
2017-05-03\T\00:26:23.322 [INFO] 2017-05-03\T\00:26:23.322 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
2017-05-03\T\00:26:23.350 [INFO] 2017-05-03\T\00:26:23.349 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
 (2 KB at 65.2 KB/sec)
2017-05-03\T\00:26:23.353 [INFO] 2017-05-03\T\00:26:23.353 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
2017-05-03\T\00:26:23.380 [INFO] 2017-05-03\T\00:26:23.380 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
 (2 KB at 53.4 KB/sec)
2017-05-03\T\00:26:23.385 [INFO] 2017-05-03\T\00:26:23.385 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
2017-05-03\T\00:26:23.411 [INFO] 2017-05-03\T\00:26:23.411 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
 (2 KB at 65.5 KB/sec)
2017-05-03\T\00:26:23.420 [INFO] 2017-05-03\T\00:26:23.419 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
2017-05-03\T\00:26:23.448 [INFO] 2017-05-03\T\00:26:23.448 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
 (3 KB at 69.9 KB/sec)
2017-05-03\T\00:26:23.453 [INFO] 2017-05-03\T\00:26:23.452 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
2017-05-03\T\00:26:23.480 [INFO] 2017-05-03\T\00:26:23.480 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
 (2 KB at 44.8 KB/sec)
2017-05-03\T\00:26:23.484 [INFO] 2017-05-03\T\00:26:23.483 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.0.9/maven-toolchain-2.0.9.pom
2017-05-03\T\00:26:23.511 [INFO] 2017-05-03\T\00:26:23.511 [INFO] Downloaded: 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3599

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #5

[Apache Jenkins Server]

See 


--
[...truncated 725.88 KB...]
2017-05-03\T\00:20:08.751 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
2017-05-03\T\00:20:08.787 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 657.7 KB/sec)
2017-05-03\T\00:20:08.787 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-03\T\00:20:08.788 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 3067.3 KB/sec)
2017-05-03\T\00:20:08.788 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-03\T\00:20:08.791 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 4042.1 KB/sec)
2017-05-03\T\00:20:08.791 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-03\T\00:20:08.804 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 4522.4 KB/sec)
2017-05-03\T\00:20:08.804 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-03\T\00:20:08.828 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 942.4 KB/sec)
2017-05-03\T\00:20:08.829 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-03\T\00:20:08.860 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 513.9 KB/sec)
2017-05-03\T\00:20:08.861 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-03\T\00:20:08.875 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1708.4 KB/sec)
2017-05-03\T\00:20:08.875 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-03\T\00:20:08.928 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 2942.0 KB/sec)
2017-05-03\T\00:20:08.928 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-03\T\00:20:08.934 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 3795.0 KB/sec)
2017-05-03\T\00:20:08.934 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-03\T\00:20:08.947 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 6348.1 KB/sec)
2017-05-03\T\00:20:08.947 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-03\T\00:20:08.974 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 105.2 KB/sec)
2017-05-03\T\00:20:08.995 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 760.9 KB/sec)
2017-05-03\T\00:20:09.023 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 6853.3 KB/sec)
2017-05-03\T\00:20:09.053 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 5002.0 KB/sec)
2017-05-03\T\00:20:09.210 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 7122.1 KB/sec)
2017-05-03\T\00:20:09.218 [INFO] 
2017-05-03\T\00:20:09.218 [INFO] --- 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #5

[Apache Jenkins Server]

See 


--
[...truncated 726.23 KB...]
2017-05-03\T\00:18:59.163 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-03\T\00:18:59.192 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 806.2 KB/sec)
2017-05-03\T\00:18:59.192 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-03\T\00:18:59.210 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 2629.1 KB/sec)
2017-05-03\T\00:18:59.210 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-03\T\00:18:59.219 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at .5 KB/sec)
2017-05-03\T\00:18:59.219 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-03\T\00:18:59.224 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1200.8 KB/sec)
2017-05-03\T\00:18:59.224 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-03\T\00:18:59.225 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 2842.1 KB/sec)
2017-05-03\T\00:18:59.225 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-03\T\00:18:59.260 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 588.0 KB/sec)
2017-05-03\T\00:18:59.260 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-03\T\00:18:59.272 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1974.8 KB/sec)
2017-05-03\T\00:18:59.273 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-03\T\00:18:59.301 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 5087.5 KB/sec)
2017-05-03\T\00:18:59.301 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-03\T\00:18:59.330 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 3153.4 KB/sec)
2017-05-03\T\00:18:59.330 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-03\T\00:18:59.349 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1006.4 KB/sec)
2017-05-03\T\00:18:59.349 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-03\T\00:18:59.381 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 8535.5 KB/sec)
2017-05-03\T\00:18:59.381 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 5765.7 KB/sec)
2017-05-03\T\00:18:59.382 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 108.1 KB/sec)
2017-05-03\T\00:18:59.426 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 5781.7 KB/sec)
2017-05-03\T\00:18:59.547 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 8531.7 KB/sec)
2017-05-03\T\00:18:59.554 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #5

[Apache Jenkins Server]

See 


--
[...truncated 722.74 KB...]
2017-05-03\T\00:16:26.662 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
2017-05-03\T\00:16:26.693 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 714.1 KB/sec)
2017-05-03\T\00:16:26.693 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-03\T\00:16:26.703 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 2683.9 KB/sec)
2017-05-03\T\00:16:26.703 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-03\T\00:16:26.705 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 3637.9 KB/sec)
2017-05-03\T\00:16:26.705 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-03\T\00:16:26.719 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 4027.8 KB/sec)
2017-05-03\T\00:16:26.719 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-03\T\00:16:26.723 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1220.5 KB/sec)
2017-05-03\T\00:16:26.723 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-03\T\00:16:26.754 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 620.0 KB/sec)
2017-05-03\T\00:16:26.754 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-03\T\00:16:26.776 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1888.2 KB/sec)
2017-05-03\T\00:16:26.776 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-03\T\00:16:26.805 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 4909.6 KB/sec)
2017-05-03\T\00:16:26.805 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-03\T\00:16:26.830 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 3134.7 KB/sec)
2017-05-03\T\00:16:26.830 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-03\T\00:16:26.839 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 7101.2 KB/sec)
2017-05-03\T\00:16:26.839 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-03\T\00:16:26.845 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1028.5 KB/sec)
2017-05-03\T\00:16:26.861 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 9436.2 KB/sec)
2017-05-03\T\00:16:26.866 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 116.0 KB/sec)
2017-05-03\T\00:16:26.919 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 5916.7 KB/sec)
2017-05-03\T\00:16:27.073 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 7971.2 KB/sec)
2017-05-03\T\00:16:27.080 [INFO] 
2017-05-03\T\00:16:27.080 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.8 (latest),beam #4

[Apache Jenkins Server]

See 


Changes:

[davor] Include LICENSE and NOTICE in python dist files

--
[...truncated 1.21 MB...]
2017-05-03\T\00:15:18.182 [INFO] 2017-05-03\T\00:15:18.182 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
2017-05-03\T\00:15:18.209 [INFO] 2017-05-03\T\00:15:18.209 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
 (3 KB at 74.1 KB/sec)
2017-05-03\T\00:15:18.213 [INFO] 2017-05-03\T\00:15:18.213 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
2017-05-03\T\00:15:18.240 [INFO] 2017-05-03\T\00:15:18.240 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
 (3 KB at 97.7 KB/sec)
2017-05-03\T\00:15:18.244 [INFO] 2017-05-03\T\00:15:18.244 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
2017-05-03\T\00:15:18.271 [INFO] 2017-05-03\T\00:15:18.270 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
 (2 KB at 71.4 KB/sec)
2017-05-03\T\00:15:18.274 [INFO] 2017-05-03\T\00:15:18.274 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
2017-05-03\T\00:15:18.302 [INFO] 2017-05-03\T\00:15:18.302 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
 (2 KB at 68.7 KB/sec)
2017-05-03\T\00:15:18.306 [INFO] 2017-05-03\T\00:15:18.306 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
2017-05-03\T\00:15:18.335 [INFO] 2017-05-03\T\00:15:18.334 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
 (8 KB at 271.5 KB/sec)
2017-05-03\T\00:15:18.340 [INFO] 2017-05-03\T\00:15:18.340 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
2017-05-03\T\00:15:18.368 [INFO] 2017-05-03\T\00:15:18.368 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
 (2 KB at 71.0 KB/sec)
2017-05-03\T\00:15:18.371 [INFO] 2017-05-03\T\00:15:18.371 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
2017-05-03\T\00:15:18.398 [INFO] 2017-05-03\T\00:15:18.398 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
 (2 KB at 65.2 KB/sec)
2017-05-03\T\00:15:18.401 [INFO] 2017-05-03\T\00:15:18.400 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
2017-05-03\T\00:15:18.428 [INFO] 2017-05-03\T\00:15:18.428 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
 (2 KB at 51.5 KB/sec)
2017-05-03\T\00:15:18.432 [INFO] 2017-05-03\T\00:15:18.432 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
2017-05-03\T\00:15:18.458 [INFO] 2017-05-03\T\00:15:18.458 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
 (2 KB at 65.5 KB/sec)
2017-05-03\T\00:15:18.462 [INFO] 2017-05-03\T\00:15:18.462 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
2017-05-03\T\00:15:18.490 [INFO] 2017-05-03\T\00:15:18.490 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
 (3 KB at 72.4 KB/sec)
2017-05-03\T\00:15:18.494 [INFO] 2017-05-03\T\00:15:18.494 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
2017-05-03\T\00:15:18.520 [INFO] 2017-05-03\T\00:15:18.520 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
 (2 KB at 48.3 KB/sec)
2017-05-03\T\00:15:18.523 [INFO] 2017-05-03\T\00:15:18.523 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.0.9/maven-toolchain-2.0.9.pom
2017-05-03\T\00:15:18.549 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3601

[Apache Jenkins Server]

See 


Changes:

[davor] Include LICENSE and NOTICE in python dist files

--
[...truncated 598.00 KB...]
2017-05-03T00:08:40.621 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-05-03T00:08:40.621 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2017-05-03T00:08:40.621 [INFO] 
:
 Recompile with -Xlint:unchecked for details.
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-03T00:08:41.007 [INFO] 
2017-05-03T00:08:41.007 [INFO] --- maven-resources-plugin:3.0.2:testResources 
(default-testResources) @ beam-runners-direct-java ---
2017-05-03T00:08:41.009 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-05-03T00:08:41.009 [INFO] skip non existing resourceDirectory 

2017-05-03T00:08:41.010 [INFO] Copying 3 resources
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-03T00:08:41.098 [INFO] 
2017-05-03T00:08:41.098 [INFO] --- maven-compiler-plugin:3.6.1:testCompile 
(default-testCompile) @ beam-runners-direct-java ---
2017-05-03T00:08:41.122 [INFO] Changes detected - recompiling the module!
2017-05-03T00:08:41.122 [INFO] Compiling 39 source files to 

2017-05-03T00:08:42.457 [INFO] 
:
 Some input files use or override a deprecated API.
2017-05-03T00:08:42.457 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-05-03T00:08:42.457 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2017-05-03T00:08:42.458 [INFO] 
:
 Recompile with -Xlint:unchecked for details.
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-03T00:08:42.550 [INFO] 
2017-05-03T00:08:42.550 [INFO] --- maven-checkstyle-plugin:2.17:check (default) 
@ beam-runners-direct-java ---
2017-05-03T00:08:47.928 [INFO] Starting audit...
Audit done.
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-03T00:08:48.022 [INFO] 
2017-05-03T00:08:48.023 [INFO] >>> findbugs-maven-plugin:3.0.4:check (default) 
> :findbugs @ beam-runners-direct-java >>>
2017-05-03T00:08:48.038 [INFO] 
2017-05-03T00:08:48.038 [INFO] --- findbugs-maven-plugin:3.0.4:findbugs 
(findbugs) @ beam-runners-direct-java ---
2017-05-03T00:08:48.046 [INFO] Fork Value is true
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #4

[Apache Jenkins Server]

See 


Changes:

[davor] Include LICENSE and NOTICE in python dist files

--
[...truncated 726.31 KB...]
2017-05-03\T\00:09:03.659 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
2017-05-03\T\00:09:03.696 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 609.6 KB/sec)
2017-05-03\T\00:09:03.696 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-03\T\00:09:03.697 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 2996.0 KB/sec)
2017-05-03\T\00:09:03.697 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-03\T\00:09:03.702 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 3712.1 KB/sec)
2017-05-03\T\00:09:03.702 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-03\T\00:09:03.716 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 4157.7 KB/sec)
2017-05-03\T\00:09:03.716 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-03\T\00:09:03.730 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1019.8 KB/sec)
2017-05-03\T\00:09:03.730 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-03\T\00:09:03.768 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 513.9 KB/sec)
2017-05-03\T\00:09:03.768 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-03\T\00:09:03.778 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1779.0 KB/sec)
2017-05-03\T\00:09:03.778 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-03\T\00:09:03.794 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 5124.6 KB/sec)
2017-05-03\T\00:09:03.794 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-03\T\00:09:03.821 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 7664.1 KB/sec)
2017-05-03\T\00:09:03.821 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-03\T\00:09:03.833 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1063.6 KB/sec)
2017-05-03\T\00:09:03.833 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-03\T\00:09:03.841 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 10205.5 KB/sec)
2017-05-03\T\00:09:03.842 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 2846.6 KB/sec)
2017-05-03\T\00:09:03.863 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 114.9 KB/sec)
2017-05-03\T\00:09:03.927 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 5652.8 KB/sec)
2017-05-03\T\00:09:04.043 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #4

[Apache Jenkins Server]

See 


Changes:

[davor] Include LICENSE and NOTICE in python dist files

--
[...truncated 723.21 KB...]
2017-05-03\T\00:07:29.257 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-03\T\00:07:29.286 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 833.1 KB/sec)
2017-05-03\T\00:07:29.286 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-03\T\00:07:29.296 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 3712.1 KB/sec)
2017-05-03\T\00:07:29.296 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-03\T\00:07:29.301 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 4957.3 KB/sec)
2017-05-03\T\00:07:29.301 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-03\T\00:07:29.303 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 2300.5 KB/sec)
2017-05-03\T\00:07:29.303 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-03\T\00:07:29.316 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1261.8 KB/sec)
2017-05-03\T\00:07:29.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-03\T\00:07:29.345 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 648.2 KB/sec)
2017-05-03\T\00:07:29.345 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-03\T\00:07:29.365 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1993.1 KB/sec)
2017-05-03\T\00:07:29.365 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-03\T\00:07:29.394 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 5124.6 KB/sec)
2017-05-03\T\00:07:29.394 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-03\T\00:07:29.413 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 11960.6 KB/sec)
2017-05-03\T\00:07:29.413 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-03\T\00:07:29.433 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 2992.2 KB/sec)
2017-05-03\T\00:07:29.433 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-03\T\00:07:29.454 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 950.2 KB/sec)
2017-05-03\T\00:07:29.454 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 6380.3 KB/sec)
2017-05-03\T\00:07:29.461 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 116.0 KB/sec)
2017-05-03\T\00:07:29.510 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 6010.3 KB/sec)
2017-05-03\T\00:07:29.643 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 

Build failed in Jenkins: beam_PerformanceTests_Dataflow #365

[Apache Jenkins Server]

See 


Changes:

[klk] Use processing time for synchronized processing time in Flink runner

[klk] Simplify type parameters of StateSpec and related

[klk] Make DoFnSignatures robust to StateSpec subclasses

[kirpichov] Removes leftover no-op withoutValidation methods

[kirpichov] Removes TextIO.Read.Bound

[kirpichov] Converts TextIO.Read to AutoValue

[kirpichov] Removes TextIO.Write.Bound

[kirpichov] Converts TextIO.Write to AutoValue

[kirpichov] Moves TextSource and TextSink to top level

[lcwik] [BEAM-2135] Move hdfs to hadoop-file-system

[dhalperi] [BEAM-2141] Disable JDBC tests

[dhalperi] BigQuery: swap from asSingleton to asIterable for Cleanup

[lcwik] [BEAM-2020] Convert all unknown Coders into CustomCoder CloudObjects

[chamikara] [BEAM-539] Fixes several issues of FileSink.

[altay] [BEAM-1316] Remove the usage of mock from ptransform tests

[lcwik] [BEAM-2135] Fix pointers to GCP-core

[klk] Skip null options when converting back to argv

[klk] Instantiate runner briefly in Pipeline

[lcwik] [BEAM-2135] Fix pointers to sdks/java/io/hadoop-file-system

[davor] [BEAM-2093] Use the jackson version from the maven property in maven

[dhalperi] [BEAM-59] Delete old restrictions on output file paths

[lcwik] [BEAM-2139] Disable SplittableDoFn ValidatesRunner tests for Streaming

[dhalperi] [BEAM-59] Move GcsFileSystem to gcp-core

[dhalperi] Full removal of Aggregators in Java SDK and Runners

[davor] Include LICENSE and NOTICE in python dist files

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 3bb0f8e6ad8a2b378a656731732a79c9c16e4a8d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3bb0f8e6ad8a2b378a656731732a79c9c16e4a8d
 > git rev-list eec903fb538eb9c3d41d737dba94a41f7c1c469e # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson4423006436169070601.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson5338004820536756484.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson1500182718279178360.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

[jira] [Commented] (BEAM-1746) Include LICENSE and NOTICE in python dist files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994055#comment-15994055
 ] 

ASF GitHub Bot commented on BEAM-1746:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2245


> Include LICENSE and NOTICE in python dist files
> ---
>
> Key: BEAM-1746
> URL: https://issues.apache.org/jira/browse/BEAM-1746
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2245: [BEAM-1746] Include LICENSE and NOTICE in python di...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2245


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Include LICENSE and NOTICE in python dist files

[davor]

Repository: beam
Updated Branches:
  refs/heads/master 4682238dc -> 3bb0f8e6a


Include LICENSE and NOTICE in python dist files


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6d5e1ffb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6d5e1ffb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6d5e1ffb

Branch: refs/heads/master
Commit: 6d5e1ffb4f9f392924bf04daedbc715a260f94cc
Parents: 4682238
Author: Ahmet Altay 
Authored: Tue Mar 14 13:43:06 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 2 16:56:35 2017 -0700

--
 .gitignore  |  2 ++
 pom.xml | 29 +
 sdks/python/MANIFEST.in |  2 ++
 sdks/python/pom.xml |  2 +-
 4 files changed, 34 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6d5e1ffb/.gitignore
--
diff --git a/.gitignore b/.gitignore
index 69946a9..9cfae09 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,6 +22,8 @@ env/
 sdks/python/**/*.c
 sdks/python/**/*.so
 sdks/python/**/*.egg
+sdks/python/LICENSE
+sdks/python/NOTICE
 
 # Ignore IntelliJ files.
 .idea/

http://git-wip-us.apache.org/repos/asf/beam/blob/6d5e1ffb/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 59f6963..127babd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1508,6 +1508,8 @@
 
   **/*.pyc
   **/*.egg-info/
+  **/sdks/python/LICENSE
+  **/sdks/python/NOTICE
 
 false
   
@@ -1613,6 +1615,33 @@
 org.apache.maven.plugins
 maven-shade-plugin
   
+  
+org.apache.maven.plugins
+maven-resources-plugin
+3.0.2
+false
+
+  
+python-copy-resources
+prepare-package
+
+  copy-resources
+
+
+  ${basedir}/sdks/python
+
+
+  ${basedir}
+  
+LICENSE
+NOTICE
+  
+
+
+
+  
+
+  
 
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/6d5e1ffb/sdks/python/MANIFEST.in
--
diff --git a/sdks/python/MANIFEST.in b/sdks/python/MANIFEST.in
index ff572ba..41d80ef 100644
--- a/sdks/python/MANIFEST.in
+++ b/sdks/python/MANIFEST.in
@@ -16,3 +16,5 @@
 #
 
 include README.md
+include NOTICE
+include LICENSE

http://git-wip-us.apache.org/repos/asf/beam/blob/6d5e1ffb/sdks/python/pom.xml
--
diff --git a/sdks/python/pom.xml b/sdks/python/pom.xml
index aa19770..f9134a5 100644
--- a/sdks/python/pom.xml
+++ b/sdks/python/pom.xml
@@ -47,7 +47,7 @@
   exec-maven-plugin
   
 
-  
+  
   none
 
   

[2/2] beam git commit: This closes #2245

[davor]

This closes #2245


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3bb0f8e6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3bb0f8e6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3bb0f8e6

Branch: refs/heads/master
Commit: 3bb0f8e6ad8a2b378a656731732a79c9c16e4a8d
Parents: 4682238 6d5e1ff
Author: Davor Bonaci 
Authored: Tue May 2 16:57:48 2017 -0700
Committer: Davor Bonaci 
Committed: Tue May 2 16:57:48 2017 -0700

--
 .gitignore  |  2 ++
 pom.xml | 29 +
 sdks/python/MANIFEST.in |  2 ++
 sdks/python/pom.xml |  2 +-
 4 files changed, 34 insertions(+), 1 deletion(-)
--

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #3

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Full removal of Aggregators in Java SDK and Runners

--
[...truncated 728.74 KB...]
2017-05-02\T\23:52:14.932 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-02\T\23:52:14.976 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 2430.7 KB/sec)
2017-05-02\T\23:52:14.976 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:52:14.977 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 471.6 KB/sec)
2017-05-02\T\23:52:14.977 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:52:15.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 2245.6 KB/sec)
2017-05-02\T\23:52:15.004 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:52:15.024 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 744.5 KB/sec)
2017-05-02\T\23:52:15.024 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:52:15.064 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 5014.8 KB/sec)
2017-05-02\T\23:52:15.064 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:52:15.075 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 1695.9 KB/sec)
2017-05-02\T\23:52:15.076 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:52:15.082 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1362.4 KB/sec)
2017-05-02\T\23:52:15.083 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:52:15.120 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 291.0 KB/sec)
2017-05-02\T\23:52:15.120 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:52:15.196 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 690.7 KB/sec)
2017-05-02\T\23:52:15.196 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:52:15.240 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 1671.8 KB/sec)
2017-05-02\T\23:52:15.240 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:52:15.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 5795.7 KB/sec)
2017-05-02\T\23:52:15.261 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 3740.8 KB/sec)
2017-05-02\T\23:52:15.273 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 68.0 KB/sec)
2017-05-02\T\23:52:15.275 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 4344.6 KB/sec)
2017-05-02\T\23:52:15.526 [INFO] Downloaded: 

[jira] [Resolved] (BEAM-2063) type DATE/TIME/TIMESTAMP support

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-2063.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

merge as part of BEAM-2006

> type DATE/TIME/TIMESTAMP support
> 
>
> Key: BEAM-2063
> URL: https://issues.apache.org/jira/browse/BEAM-2063
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>Priority: Blocker
> Fix For: Not applicable
>
>
> Support data type DATE/TIME/TIMESTAMP, which could be used in aggregation.
> New data type support include changes:
> 1. type mapping in {{BeamSQLRow}}, and coder support;
> 2. {{BeamSqlPrimitive}} for static values;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #3

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Full removal of Aggregators in Java SDK and Runners

--
[...truncated 721.95 KB...]
2017-05-02\T\23:48:06.972 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-02\T\23:48:07.010 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 640.8 KB/sec)
2017-05-02\T\23:48:07.010 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:48:07.012 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 3067.3 KB/sec)
2017-05-02\T\23:48:07.012 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:48:07.013 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 4230.1 KB/sec)
2017-05-02\T\23:48:07.013 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:48:07.017 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 5603.9 KB/sec)
2017-05-02\T\23:48:07.017 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:48:07.043 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1048.6 KB/sec)
2017-05-02\T\23:48:07.043 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:48:07.060 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 2446.1 KB/sec)
2017-05-02\T\23:48:07.060 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:48:07.074 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 559.2 KB/sec)
2017-05-02\T\23:48:07.074 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:48:07.100 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 5484.9 KB/sec)
2017-05-02\T\23:48:07.100 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:48:07.140 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 7481.7 KB/sec)
2017-05-02\T\23:48:07.140 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:48:07.142 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1101.1 KB/sec)
2017-05-02\T\23:48:07.143 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:48:07.152 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 2925.7 KB/sec)
2017-05-02\T\23:48:07.164 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 9729.6 KB/sec)
2017-05-02\T\23:48:07.168 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 120.8 KB/sec)
2017-05-02\T\23:48:07.212 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 6335.8 KB/sec)
2017-05-02\T\23:48:07.319 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3015

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.8 (latest),beam #2

[Apache Jenkins Server]

See 


--
[...truncated 1.20 MB...]
2017-05-02\T\23:47:14.277 [INFO] 2017-05-02\T\23:47:14.277 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
2017-05-02\T\23:47:14.307 [INFO] 2017-05-02\T\23:47:14.307 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
 (3 KB at 66.7 KB/sec)
2017-05-02\T\23:47:14.311 [INFO] 2017-05-02\T\23:47:14.310 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
2017-05-02\T\23:47:14.339 [INFO] 2017-05-02\T\23:47:14.339 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
 (3 KB at 94.2 KB/sec)
2017-05-02\T\23:47:14.343 [INFO] 2017-05-02\T\23:47:14.343 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
2017-05-02\T\23:47:14.372 [INFO] 2017-05-02\T\23:47:14.372 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
 (2 KB at 64.0 KB/sec)
2017-05-02\T\23:47:14.376 [INFO] 2017-05-02\T\23:47:14.376 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
2017-05-02\T\23:47:14.405 [INFO] 2017-05-02\T\23:47:14.404 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
 (2 KB at 68.7 KB/sec)
2017-05-02\T\23:47:14.410 [INFO] 2017-05-02\T\23:47:14.409 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
2017-05-02\T\23:47:14.441 [INFO] 2017-05-02\T\23:47:14.441 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
 (8 KB at 237.6 KB/sec)
2017-05-02\T\23:47:14.446 [INFO] 2017-05-02\T\23:47:14.446 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
2017-05-02\T\23:47:14.474 [INFO] 2017-05-02\T\23:47:14.474 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
 (2 KB at 68.5 KB/sec)
2017-05-02\T\23:47:14.479 [INFO] 2017-05-02\T\23:47:14.479 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
2017-05-02\T\23:47:14.509 [INFO] 2017-05-02\T\23:47:14.509 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
 (2 KB at 60.7 KB/sec)
2017-05-02\T\23:47:14.512 [INFO] 2017-05-02\T\23:47:14.511 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
2017-05-02\T\23:47:14.541 [INFO] 2017-05-02\T\23:47:14.540 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
 (2 KB at 49.7 KB/sec)
2017-05-02\T\23:47:14.546 [INFO] 2017-05-02\T\23:47:14.546 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
2017-05-02\T\23:47:14.576 [INFO] 2017-05-02\T\23:47:14.576 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
 (2 KB at 56.8 KB/sec)
2017-05-02\T\23:47:14.581 [INFO] 2017-05-02\T\23:47:14.581 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
2017-05-02\T\23:47:14.612 [INFO] 2017-05-02\T\23:47:14.612 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
 (3 KB at 65.4 KB/sec)
2017-05-02\T\23:47:14.617 [INFO] 2017-05-02\T\23:47:14.617 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
2017-05-02\T\23:47:14.645 [INFO] 2017-05-02\T\23:47:14.645 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
 (2 KB at 43.3 KB/sec)
2017-05-02\T\23:47:14.649 [INFO] 2017-05-02\T\23:47:14.649 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.0.9/maven-toolchain-2.0.9.pom
2017-05-02\T\23:47:14.679 [INFO] 2017-05-02\T\23:47:14.679 [INFO] Downloaded: 

[jira] [Commented] (BEAM-775) Remove Aggregators from the Java SDK

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994041#comment-15994041
 ] 

Daniel Halperin commented on BEAM-775:
--

Remaining work:

* Rebuild DataflowWorker to stop using the three deprecated functions.
* Update the SDK with the new worker and remove the three deprecated functions.
* run a recursive grep on the code to check uses of the word Aggregator. E.g., 

{code}
[16:38:40 dhalperi@dhalperi:beam 6bf4ab7b0 ] grep -i aggregator -R . | grep -v 
"py:"  | grep -v spark   []
./runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java:
  Object aggregatorFactory,
./examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java: 
* and using a selected runner; defining DoFns; creating a custom aggregator;
./sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnTesterTest.java:
   * A {@link DoFn} that adds values to an aggregator and converts input to 
String in
./sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LatestFnTest.java:
  public void testExtractOutputDefaultAggregator() {
./sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateUniqueTest.java:
 * Tests for the ApproximateUnique aggregator transform.
./sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java: 
* An {@code Aggregator} enables monitoring of values of type {@code 
InputT},
./sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java:public
 interface Aggregator {
./sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Latest.java: * 
Example 2: track a latest computed value in an aggregator:
./sdks/java/core/src/main/java/org/apache/beam/sdk/util/SystemDoFnInternal.java:
 * Currently, the only effect of this is to mark any aggregators reported by 
an annotated
Binary file 
./sdks/java/core/target/classes/org/apache/beam/sdk/transforms/Aggregator.class 
matches
./sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/WindowedWordCount.java:
 * and using a selected runner; defining DoFns; creating a custom aggregator;
./sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/src/main/java/WindowedWordCount.java:
 * and using a selected runner; defining DoFns; creating a custom aggregator;
{code}

> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Pablo Estrada
>  Labels: backward-incompatible
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam-site git commit: Fix broken HDFS link on built-in IO page

[lcwik]

Fix broken HDFS link on built-in IO page

This closes #228


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/a0e95bae
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/a0e95bae
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/a0e95bae

Branch: refs/heads/asf-site
Commit: a0e95bae34e667efa752476f7b9c810041d6ccaa
Parents: aac3894 9685636
Author: Luke Cwik 
Authored: Tue May 2 16:41:12 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 2 16:41:12 2017 -0700

--
 src/documentation/io/built-in.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] beam-site git commit: Fix broken HDFS link on built-in IO page

[lcwik]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site aac389466 -> a0e95bae3


Fix broken HDFS link on built-in IO page


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/96856361
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/96856361
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/96856361

Branch: refs/heads/asf-site
Commit: 96856361602b358078bab37f512a7041f13128dc
Parents: aac3894
Author: melissa 
Authored: Tue May 2 16:07:20 2017 -0700
Committer: Luke Cwik 
Committed: Tue May 2 16:29:03 2017 -0700

--
 src/documentation/io/built-in.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/96856361/src/documentation/io/built-in.md
--
diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 580f3ed..6a73f6b 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -24,7 +24,7 @@ Consult the [Programming Guide I/O section]({{site.baseurl 
}}/documentation/prog
   Java
   
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
-https://github.com/apache/beam/tree/master/sdks/java/io/hdfs;>Apache 
Hadoop HDFS
+https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system;>Apache
 Hadoop File System
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java;>TextIO
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/;>XML
   

[GitHub] beam-site pull request #228: Fix broken HDFS link on built-in IO page

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/228


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #2

[Apache Jenkins Server]

See 


--
[...truncated 727.24 KB...]
2017-05-02\T\23:39:43.113 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-02\T\23:39:43.143 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 781.0 KB/sec)
2017-05-02\T\23:39:43.143 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:39:43.151 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 3142.1 KB/sec)
2017-05-02\T\23:39:43.151 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:39:43.151 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 4436.5 KB/sec)
2017-05-02\T\23:39:43.151 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:39:43.160 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 5260.8 KB/sec)
2017-05-02\T\23:39:43.160 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:39:43.180 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1094.8 KB/sec)
2017-05-02\T\23:39:43.180 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:39:43.209 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 2219.2 KB/sec)
2017-05-02\T\23:39:43.209 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:39:43.214 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 559.2 KB/sec)
2017-05-02\T\23:39:43.214 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:39:43.229 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 6000.6 KB/sec)
2017-05-02\T\23:39:43.229 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:39:43.243 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 14225.9 KB/sec)
2017-05-02\T\23:39:43.243 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:39:43.256 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 8728.6 KB/sec)
2017-05-02\T\23:39:43.256 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:39:43.267 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1215.5 KB/sec)
2017-05-02\T\23:39:43.277 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 3191.7 KB/sec)
2017-05-02\T\23:39:43.283 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 139.2 KB/sec)
2017-05-02\T\23:39:43.322 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 7275.6 KB/sec)
2017-05-02\T\23:39:43.412 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 10957.1 KB/sec)
2017-05-02\T\23:39:43.418 [INFO] 

[GitHub] beam pull request #2838: [BEAM-775] Full removal of Aggregators in Java SDK ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2838


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-775) Remove Aggregators from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994020#comment-15994020
 ] 

ASF GitHub Bot commented on BEAM-775:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2838


> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Pablo Estrada
>  Labels: backward-incompatible
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #2

[Apache Jenkins Server]

See 


--
[...truncated 726.61 KB...]
2017-05-02\T\23:38:19.515 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-02\T\23:38:19.552 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 3303.2 KB/sec)
2017-05-02\T\23:38:19.552 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:38:19.555 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 595.1 KB/sec)
2017-05-02\T\23:38:19.555 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:38:19.565 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 3432.0 KB/sec)
2017-05-02\T\23:38:19.565 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:38:19.585 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1048.6 KB/sec)
2017-05-02\T\23:38:19.585 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:38:19.595 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 3143.6 KB/sec)
2017-05-02\T\23:38:19.595 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:38:19.627 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 504.8 KB/sec)
2017-05-02\T\23:38:19.627 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:38:19.645 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1643.2 KB/sec)
2017-05-02\T\23:38:19.645 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:38:19.675 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 11591.4 KB/sec)
2017-05-02\T\23:38:19.675 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:38:19.687 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 4058.2 KB/sec)
2017-05-02\T\23:38:19.687 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:38:19.713 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 2659.7 KB/sec)
2017-05-02\T\23:38:19.713 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:38:19.737 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 5636.4 KB/sec)
2017-05-02\T\23:38:19.739 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 835.7 KB/sec)
2017-05-02\T\23:38:19.740 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 105.2 KB/sec)
2017-05-02\T\23:38:19.792 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 5489.5 KB/sec)
2017-05-02\T\23:38:19.944 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 7636.7 KB/sec)
2017-05-02\T\23:38:19.952 [INFO] 

[1/3] beam git commit: Full removal of Aggregators in Java SDK and Runners

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 5bfd3e049 -> 4682238dc


http://git-wip-us.apache.org/repos/asf/beam/blob/615761a7/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DelegatingAggregator.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DelegatingAggregator.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DelegatingAggregator.java
deleted file mode 100644
index cfaf0a6..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DelegatingAggregator.java
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms;
-
-import static com.google.common.base.Preconditions.checkNotNull;
-
-import com.google.common.base.MoreObjects;
-import java.io.Serializable;
-import java.util.Objects;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-
-/**
- * An {@link Aggregator} that delegates calls to {@link #addValue} to another 
aggregator.
- *
- * This {@link Aggregator} is designed to be constructed without a 
delegate, at pipeline
- * construction time, and serialized within a {@link DoFn}. The delegate 
aggregator to which it
- * submits values must be provided by the runner at execution time.
- *
- * @param  the type of input element
- * @param  the type of output element
- */
-public class DelegatingAggregator
-implements Aggregator, Serializable {
-  private static final AtomicInteger ID_GEN = new AtomicInteger();
-  private final int id;
-
-  private final String name;
-
-  private final CombineFn combineFn;
-
-  private Aggregator delegate;
-
-  public DelegatingAggregator(String name,
-  CombineFn combiner) {
-this.id = ID_GEN.getAndIncrement();
-this.name = checkNotNull(name, "name cannot be null");
-// Safe contravariant cast
-@SuppressWarnings("unchecked")
-CombineFn specificCombiner =
-(CombineFn) checkNotNull(combiner, 
"combineFn cannot be null");
-this.combineFn = specificCombiner;
-  }
-
-  @Override
-  public void addValue(AggInputT value) {
-if (delegate == null) {
-  throw new IllegalStateException(
-  String.format(
-  "addValue cannot be called on Aggregator outside of the 
execution of a %s.",
-  DoFn.class.getSimpleName()));
-} else {
-  delegate.addValue(value);
-}
-  }
-
-  @Override
-  public String getName() {
-return name;
-  }
-
-  @Override
-  public CombineFn getCombineFn() {
-return combineFn;
-  }
-
-  /**
-   * Sets the current delegate of the Aggregator.
-   *
-   * @param delegate the delegate to set in this aggregator
-   */
-  public void setDelegate(Aggregator delegate) {
-this.delegate = delegate;
-  }
-
-  @Override
-  public String toString() {
-return MoreObjects.toStringHelper(getClass())
-.add("name", name)
-.add("combineFn", combineFn)
-.toString();
-  }
-
-  @Override
-  public int hashCode() {
-return Objects.hash(id, name, combineFn.getClass());
-  }
-
-  /**
-   * Indicates whether some other object is "equal to" this one.
-   *
-   * {@code DelegatingAggregator} instances are equal if they have the same 
name, their
-   * CombineFns are the same class, and they have identical IDs.
-   */
-  @Override
-  public boolean equals(Object o) {
-if (o == this) {
-  return true;
-}
-if (o == null) {
-  return false;
-}
-if (o instanceof DelegatingAggregator) {
-  DelegatingAggregator that = (DelegatingAggregator) o;
-  return Objects.equals(this.id, that.id)
-  && Objects.equals(this.name, that.name)
-  && Objects.equals(this.combineFn.getClass(), 
that.combineFn.getClass());
-}
-return false;
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/615761a7/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Latest.java

[2/3] beam git commit: Full removal of Aggregators in Java SDK and Runners

[dhalperi]

Full removal of Aggregators in Java SDK and Runners


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/615761a7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/615761a7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/615761a7

Branch: refs/heads/master
Commit: 615761a77d2da6229dfa2cad5376d265afea8a62
Parents: 5bfd3e0
Author: Pablo 
Authored: Tue May 2 14:49:39 2017 -0700
Committer: Dan Halperin 
Committed: Tue May 2 16:37:27 2017 -0700

--
 .../cookbook/CombinePerKeyExamples.java |   4 -
 .../beam/runners/core/AggregatorFactory.java|  38 
 .../apache/beam/runners/core/DoFnRunners.java   |  27 ++-
 .../apache/beam/runners/core/LateDataUtils.java |   6 +-
 .../beam/runners/core/SimpleDoFnRunner.java |   1 -
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   3 -
 .../core/LateDataDroppingDoFnRunnerTest.java|  27 ---
 .../beam/runners/core/SimpleDoFnRunnerTest.java |   9 -
 .../runners/core/SimpleOldDoFnRunnerTest.java   |   2 +-
 .../runners/core/StatefulDoFnRunnerTest.java|   1 -
 .../runners/direct/AggregatorContainer.java | 200 ---
 .../beam/runners/direct/EvaluationContext.java  |  23 +--
 .../GroupAlsoByWindowEvaluatorFactory.java  |  23 +--
 .../beam/runners/direct/ParDoEvaluator.java |  14 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 -
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 -
 .../runners/direct/StepTransformResult.java |   8 -
 .../beam/runners/direct/TransformResult.java|   6 -
 .../beam/runners/direct/ParDoEvaluatorTest.java |   5 -
 .../beam/runners/spark/SparkPipelineResult.java |   5 -
 .../spark/aggregators/SparkAggregators.java | 110 --
 .../SparkGroupAlsoByWindowViaWindowSet.java |  57 ++
 .../spark/translation/MultiDoFnFunction.java|   2 -
 ...SparkGroupAlsoByWindowViaOutputBufferFn.java |  10 -
 .../spark/translation/SparkRuntimeContext.java  |  81 
 .../spark/aggregators/ClearAggregatorsRule.java |  38 
 .../metrics/sink/NamedAggregatorsTest.java  | 101 --
 .../beam/sdk/AggregatorPipelineExtractor.java   |  84 
 .../beam/sdk/AggregatorRetrievalException.java  |  33 ---
 .../org/apache/beam/sdk/AggregatorValues.java   |  51 -
 .../main/java/org/apache/beam/sdk/Pipeline.java |  10 -
 .../beam/sdk/annotations/Experimental.java  |   3 -
 .../apache/beam/sdk/transforms/Aggregator.java  |  14 +-
 .../sdk/transforms/DelegatingAggregator.java| 126 
 .../org/apache/beam/sdk/transforms/Latest.java  |   2 -
 .../harness/control/ProcessBundleHandler.java   |   2 -
 .../fn/harness/fake/FakeAggregatorFactory.java  |  52 -
 37 files changed, 52 insertions(+), 1129 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/615761a7/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java
 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java
index 39553a5..693f0c4 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java
@@ -45,10 +45,6 @@ import org.apache.beam.sdk.values.PCollection;
  * list of play names in which that word appears, and saves this information
  * to a bigquery table.
  *
- * Concepts: the Combine.perKey transform, which lets you combine the 
values in a
- * key-grouped Collection, and how to use an Aggregator to track information 
in the
- * Monitoring UI.
- *
  * Note: Before running this example, you must create a BigQuery dataset to 
contain your output
  * table.
  *

http://git-wip-us.apache.org/repos/asf/beam/blob/615761a7/runners/core-java/src/main/java/org/apache/beam/runners/core/AggregatorFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/AggregatorFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/AggregatorFactory.java
deleted file mode 100644
index 24a605f..000
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/AggregatorFactory.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this 

[3/3] beam git commit: This closes #2838

[dhalperi]

This closes #2838


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4682238d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4682238d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4682238d

Branch: refs/heads/master
Commit: 4682238dc596a6d3d04b408049d7070731043cf2
Parents: 5bfd3e0 615761a
Author: Dan Halperin 
Authored: Tue May 2 16:38:30 2017 -0700
Committer: Dan Halperin 
Committed: Tue May 2 16:38:30 2017 -0700

--
 .../cookbook/CombinePerKeyExamples.java |   4 -
 .../beam/runners/core/AggregatorFactory.java|  38 
 .../apache/beam/runners/core/DoFnRunners.java   |  27 ++-
 .../apache/beam/runners/core/LateDataUtils.java |   6 +-
 .../beam/runners/core/SimpleDoFnRunner.java |   1 -
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   3 -
 .../core/LateDataDroppingDoFnRunnerTest.java|  27 ---
 .../beam/runners/core/SimpleDoFnRunnerTest.java |   9 -
 .../runners/core/SimpleOldDoFnRunnerTest.java   |   2 +-
 .../runners/core/StatefulDoFnRunnerTest.java|   1 -
 .../runners/direct/AggregatorContainer.java | 200 ---
 .../beam/runners/direct/EvaluationContext.java  |  23 +--
 .../GroupAlsoByWindowEvaluatorFactory.java  |  23 +--
 .../beam/runners/direct/ParDoEvaluator.java |  14 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 -
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 -
 .../runners/direct/StepTransformResult.java |   8 -
 .../beam/runners/direct/TransformResult.java|   6 -
 .../beam/runners/direct/ParDoEvaluatorTest.java |   5 -
 .../beam/runners/spark/SparkPipelineResult.java |   5 -
 .../spark/aggregators/SparkAggregators.java | 110 --
 .../SparkGroupAlsoByWindowViaWindowSet.java |  57 ++
 .../spark/translation/MultiDoFnFunction.java|   2 -
 ...SparkGroupAlsoByWindowViaOutputBufferFn.java |  10 -
 .../spark/translation/SparkRuntimeContext.java  |  81 
 .../spark/aggregators/ClearAggregatorsRule.java |  38 
 .../metrics/sink/NamedAggregatorsTest.java  | 101 --
 .../beam/sdk/AggregatorPipelineExtractor.java   |  84 
 .../beam/sdk/AggregatorRetrievalException.java  |  33 ---
 .../org/apache/beam/sdk/AggregatorValues.java   |  51 -
 .../main/java/org/apache/beam/sdk/Pipeline.java |  10 -
 .../beam/sdk/annotations/Experimental.java  |   3 -
 .../apache/beam/sdk/transforms/Aggregator.java  |  14 +-
 .../sdk/transforms/DelegatingAggregator.java| 126 
 .../org/apache/beam/sdk/transforms/Latest.java  |   2 -
 .../harness/control/ProcessBundleHandler.java   |   2 -
 .../fn/harness/fake/FakeAggregatorFactory.java  |  52 -
 37 files changed, 52 insertions(+), 1129 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3598

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #2

[Apache Jenkins Server]

See 


--
[...truncated 724.19 KB...]
2017-05-02\T\23:36:49.164 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-02\T\23:36:49.192 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 833.1 KB/sec)
2017-05-02\T\23:36:49.192 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:36:49.222 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 2111.9 KB/sec)
2017-05-02\T\23:36:49.223 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:36:49.236 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1019.8 KB/sec)
2017-05-02\T\23:36:49.236 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:36:49.247 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 2115.1 KB/sec)
2017-05-02\T\23:36:49.247 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:36:49.259 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 2630.4 KB/sec)
2017-05-02\T\23:36:49.259 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:36:49.288 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 456.3 KB/sec)
2017-05-02\T\23:36:49.288 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:36:49.307 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1494.8 KB/sec)
2017-05-02\T\23:36:49.307 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:36:49.391 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 3079.3 KB/sec)
2017-05-02\T\23:36:49.391 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:36:49.399 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 2231.5 KB/sec)
2017-05-02\T\23:36:49.399 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:36:49.431 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 4690.0 KB/sec)
2017-05-02\T\23:36:49.431 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:36:49.431 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 698.5 KB/sec)
2017-05-02\T\23:36:49.451 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 6497.6 KB/sec)
2017-05-02\T\23:36:49.458 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 80.2 KB/sec)
2017-05-02\T\23:36:49.528 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 4166.0 KB/sec)
2017-05-02\T\23:36:49.675 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 6398.8 KB/sec)
2017-05-02\T\23:36:49.683 [INFO] 
2017-05-02\T\23:36:49.684 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.8 (latest),beam #1

[Apache Jenkins Server]

See 


--
[...truncated 1.20 MB...]
2017-05-02\T\23:33:55.238 [INFO] 2017-05-02\T\23:33:55.237 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
2017-05-02\T\23:33:55.271 [INFO] 2017-05-02\T\23:33:55.271 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.9/maven-profile-2.0.9.pom
 (3 KB at 58.9 KB/sec)
2017-05-02\T\23:33:55.275 [INFO] 2017-05-02\T\23:33:55.275 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
2017-05-02\T\23:33:55.303 [INFO] 2017-05-02\T\23:33:55.303 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.9/maven-artifact-manager-2.0.9.pom
 (3 KB at 94.2 KB/sec)
2017-05-02\T\23:33:55.307 [INFO] 2017-05-02\T\23:33:55.307 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
2017-05-02\T\23:33:55.334 [INFO] 2017-05-02\T\23:33:55.334 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.9/maven-repository-metadata-2.0.9.pom
 (2 KB at 68.8 KB/sec)
2017-05-02\T\23:33:55.337 [INFO] 2017-05-02\T\23:33:55.337 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
2017-05-02\T\23:33:55.364 [INFO] 2017-05-02\T\23:33:55.363 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.9/maven-plugin-registry-2.0.9.pom
 (2 KB at 74.0 KB/sec)
2017-05-02\T\23:33:55.368 [INFO] 2017-05-02\T\23:33:55.368 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
2017-05-02\T\23:33:55.399 [INFO] 2017-05-02\T\23:33:55.398 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.pom
 (8 KB at 253.4 KB/sec)
2017-05-02\T\23:33:55.404 [INFO] 2017-05-02\T\23:33:55.402 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
2017-05-02\T\23:33:55.430 [INFO] 2017-05-02\T\23:33:55.429 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.9/maven-plugin-parameter-documenter-2.0.9.pom
 (2 KB at 71.0 KB/sec)
2017-05-02\T\23:33:55.435 [INFO] 2017-05-02\T\23:33:55.435 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
2017-05-02\T\23:33:55.462 [INFO] 2017-05-02\T\23:33:55.462 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.pom
 (2 KB at 65.2 KB/sec)
2017-05-02\T\23:33:55.465 [INFO] 2017-05-02\T\23:33:55.465 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
2017-05-02\T\23:33:55.492 [INFO] 2017-05-02\T\23:33:55.492 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.0.9/maven-reporting-2.0.9.pom
 (2 KB at 53.4 KB/sec)
2017-05-02\T\23:33:55.497 [INFO] 2017-05-02\T\23:33:55.497 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
2017-05-02\T\23:33:55.525 [INFO] 2017-05-02\T\23:33:55.525 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.9/maven-error-diagnostics-2.0.9.pom
 (2 KB at 60.8 KB/sec)
2017-05-02\T\23:33:55.530 [INFO] 2017-05-02\T\23:33:55.530 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
2017-05-02\T\23:33:55.558 [INFO] 2017-05-02\T\23:33:55.558 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.9/maven-plugin-descriptor-2.0.9.pom
 (3 KB at 72.4 KB/sec)
2017-05-02\T\23:33:55.563 [INFO] 2017-05-02\T\23:33:55.563 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
2017-05-02\T\23:33:55.594 [INFO] 2017-05-02\T\23:33:55.591 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.9/maven-monitor-2.0.9.pom
 (2 KB at 44.8 KB/sec)
2017-05-02\T\23:33:55.596 [INFO] 2017-05-02\T\23:33:55.596 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.0.9/maven-toolchain-2.0.9.pom
2017-05-02\T\23:33:55.627 [INFO] 2017-05-02\T\23:33:55.627 [INFO] Downloaded: 

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[Neville Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994005#comment-15994005
 ] 

Neville Li commented on BEAM-302:
-

You need the spark runner dependency which is not included by default.

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #1

[Apache Jenkins Server]

See 


--
[...truncated 725.73 KB...]
2017-05-02\T\23:28:20.732 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
2017-05-02\T\23:28:20.764 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 735.1 KB/sec)
2017-05-02\T\23:28:20.764 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:28:20.768 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 3390.2 KB/sec)
2017-05-02\T\23:28:20.768 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:28:20.771 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 4436.5 KB/sec)
2017-05-02\T\23:28:20.771 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:28:20.792 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1240.8 KB/sec)
2017-05-02\T\23:28:20.792 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:28:20.816 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 2997.4 KB/sec)
2017-05-02\T\23:28:20.817 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:28:20.842 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 1956.9 KB/sec)
2017-05-02\T\23:28:20.843 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:28:20.848 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 491.7 KB/sec)
2017-05-02\T\23:28:20.848 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:28:20.854 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 5754.7 KB/sec)
2017-05-02\T\23:28:20.854 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:28:20.860 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 14556.7 KB/sec)
2017-05-02\T\23:28:20.860 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:28:20.881 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 8435.7 KB/sec)
2017-05-02\T\23:28:20.881 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:28:20.899 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1120.9 KB/sec)
2017-05-02\T\23:28:20.901 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 3116.1 KB/sec)
2017-05-02\T\23:28:20.908 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 134.5 KB/sec)
2017-05-02\T\23:28:20.957 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 6758.2 KB/sec)
2017-05-02\T\23:28:21.037 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 10741.5 KB/sec)
2017-05-02\T\23:28:21.043 [INFO] 
2017-05-02\T\23:28:21.043 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #1

[Apache Jenkins Server]

See 


--
[...truncated 729.63 KB...]
2017-05-02\T\23:25:53.589 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
2017-05-02\T\23:25:53.619 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-minicluster/2.7.3/hadoop-minicluster-2.7.3.jar
 (25 KB at 806.2 KB/sec)
2017-05-02\T\23:25:53.619 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
2017-05-02\T\23:25:53.628 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-ipc/1.8.1/avro-ipc-1.8.1.jar
 (129 KB at 3220.7 KB/sec)
2017-05-02\T\23:25:53.628 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
2017-05-02\T\23:25:53.628 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-mapred/1.8.1/avro-mapred-1.8.1-hadoop2.jar
 (182 KB at 4436.5 KB/sec)
2017-05-02\T\23:25:53.628 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
2017-05-02\T\23:25:53.632 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 5858.6 KB/sec)
2017-05-02\T\23:25:53.632 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
2017-05-02\T\23:25:53.661 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.7.3/hadoop-yarn-server-tests-2.7.3-tests.jar
 (75 KB at 1019.8 KB/sec)
2017-05-02\T\23:25:53.661 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
2017-05-02\T\23:25:53.674 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.3/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar
 (216 KB at 2503.0 KB/sec)
2017-05-02\T\23:25:53.674 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
2017-05-02\T\23:25:53.697 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.7.3/hadoop-yarn-server-nodemanager-2.7.3.jar
 (703 KB at 6441.0 KB/sec)
2017-05-02\T\23:25:53.697 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
2017-05-02\T\23:25:53.697 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.3/hadoop-yarn-server-web-proxy-2.7.3.jar
 (58 KB at 523.3 KB/sec)
2017-05-02\T\23:25:53.697 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
2017-05-02\T\23:25:53.733 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.3/hadoop-yarn-server-resourcemanager-2.7.3.jar
 (1257 KB at 8668.4 KB/sec)
2017-05-02\T\23:25:53.733 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
2017-05-02\T\23:25:53.734 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar
 (527 KB at 3607.0 KB/sec)
2017-05-02\T\23:25:53.734 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
2017-05-02\T\23:25:53.736 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-hs/2.7.3/hadoop-mapreduce-client-hs-2.7.3.jar
 (188 KB at 1264.8 KB/sec)
2017-05-02\T\23:25:53.755 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3-tests.jar
 (1878 KB at 11244.4 KB/sec)
2017-05-02\T\23:25:53.765 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 133.7 KB/sec)
2017-05-02\T\23:25:53.799 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
 (1521 KB at 7206.6 KB/sec)
2017-05-02\T\23:25:53.888 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar
 (3277 KB at 10920.6 KB/sec)
2017-05-02\T\23:25:53.892 [INFO] 

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[Chaoran Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993999#comment-15993999
 ] 

Chaoran Yu commented on BEAM-302:
-

[~neville]
I tried the latest SNAPSHOT version of scio but it's still not working with 
Spark runner out of the box. For example, running sbt "scio-examples/run-main 
com.spotify.scio.examples.WordCount --runner=SparkRunner --input=README.md 
--output=wc" gave me the following exception:
java.lang.IllegalArgumentException: Unknown 'runner' specified 'SparkRunner', 
supported pipeline runners [DataflowRunner, DirectRunner]
Caused by: java.lang.ClassNotFoundException: SparkRunner
The same thing happened in the scio repl. Looking at the code, more things need 
to be done to integrate with Spark/Flink etc.

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1954) "test" extra need nose in the requirements list

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj updated BEAM-1954:

Labels: newbie  (was: )

> "test" extra need nose in the requirements list
> ---
>
> Key: BEAM-1954
> URL: https://issues.apache.org/jira/browse/BEAM-1954
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 0.6.0
>Reporter: Marc Boissonneault
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: newbie
>
> When installing beam with pip:
> {code}
> pip install apache-beam[gcp,test]==0.6.0
> {code}
> And trying to use the TestPipeline:
> {code}
> from apache_beam.test_pipeline import TestPipeline
> {code}
> We receive the following error:
> {panel}
> from nose.plugins.skip import SkipTest
> ImportError: No module named nose.plugins.skip
> {panel}
> Once nose is installed everything works as expected
> See: https://github.com/apache/beam/blob/master/sdks/python/setup.py#L100-L102



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)