[jira] [Updated] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-2140:
---
Priority: Major  (was: Blocker)

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-2140:
---
Fix Version/s: (was: First stable release)

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997815#comment-15997815
 ] 

Aljoscha Krettek commented on BEAM-2140:


Yes, that seems ok.

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1213) WordCount example failure on Apex Runner

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997802#comment-15997802
 ] 

Thomas Weise commented on BEAM-1213:


I have not tested this with the RC but I expect it to be fixed, it looks 
related to what we saw due to APEXCORE-703.

> WordCount example failure on Apex Runner
> 
>
> Key: BEAM-1213
> URL: https://issues.apache.org/jira/browse/BEAM-1213
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Sandeep Deshmukh
>Assignee: Thomas Weise
> Fix For: First stable release
>
>
> Tried running WordCount example using apex runner (on YARN) with 
> https://github.com/apache/incubator-beam/pull/1693 , fix for httpclient 
> version issue fixed.
> Create /tmp/input-beam/ dir on all nodes and pushed a test data in that 
> directory. Ran the following command in examples/java/
> ~/bin/apache-maven-3.3.9/bin/mvn  compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.WordCount 
> -Dexec.args="--inputFile=/tmp/input-beam/java.java --output=/tmp/output-beam/ 
> --runner=ApexRunner --embeddedExecution=false" -Papex-runner
> INFO: using 
> /disk2/devhdfs/nn_20161013/usercache/sandeep/appcache/application_1482319446115_4336/container_1482319446115_4336_01_14/tmp/chkp5198889890309038810
>  as the basepath for checkpointing.
> Dec 23, 2016 7:17:44 AM com.datatorrent.stram.engine.StreamingContainer main
> SEVERE: Fatal exception in container!
> java.lang.IllegalArgumentException: Invalid slice: offset=1426, length=0 
> array.length=67108864
> at com.datatorrent.netlet.util.Slice.(Slice.java:43)
> at 
> com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
> at 
> com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
> at 
> com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
> at com.datatorrent.bufferserver.server.Server.purge(Server.java:204)
> at 
> com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:776)
> at 
> com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at 
> com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Dec 23, 2016 7:17:44 AM com.datatorrent.bufferserver.server.Server 
> unregistered
> INFO: Server stopped listening at /0:0:0:0:0:0:0:0:40904
> Dec 23, 2016 7:17:44 AM com.datatorrent.stram.engine.StreamingContainer main
> INFO: Exit status for container: 1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1213) WordCount example failure on Apex Runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997796#comment-15997796
 ] 

Kenneth Knowles commented on BEAM-1213:
---

[~thw] I noticed in the Apex release candidate vote that you had tested on 
Beam. Does that address this issue? Did you test on YARN or embedded? I've 
gotten lost in the weeds for a little bit provisioning an appropriate YARN 
cluster and properly bundling the right versions of dependencies and whatnot.

> WordCount example failure on Apex Runner
> 
>
> Key: BEAM-1213
> URL: https://issues.apache.org/jira/browse/BEAM-1213
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Sandeep Deshmukh
>Assignee: Thomas Weise
> Fix For: First stable release
>
>
> Tried running WordCount example using apex runner (on YARN) with 
> https://github.com/apache/incubator-beam/pull/1693 , fix for httpclient 
> version issue fixed.
> Create /tmp/input-beam/ dir on all nodes and pushed a test data in that 
> directory. Ran the following command in examples/java/
> ~/bin/apache-maven-3.3.9/bin/mvn  compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.WordCount 
> -Dexec.args="--inputFile=/tmp/input-beam/java.java --output=/tmp/output-beam/ 
> --runner=ApexRunner --embeddedExecution=false" -Papex-runner
> INFO: using 
> /disk2/devhdfs/nn_20161013/usercache/sandeep/appcache/application_1482319446115_4336/container_1482319446115_4336_01_14/tmp/chkp5198889890309038810
>  as the basepath for checkpointing.
> Dec 23, 2016 7:17:44 AM com.datatorrent.stram.engine.StreamingContainer main
> SEVERE: Fatal exception in container!
> java.lang.IllegalArgumentException: Invalid slice: offset=1426, length=0 
> array.length=67108864
> at com.datatorrent.netlet.util.Slice.(Slice.java:43)
> at 
> com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
> at 
> com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
> at 
> com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
> at com.datatorrent.bufferserver.server.Server.purge(Server.java:204)
> at 
> com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:776)
> at 
> com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at 
> com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Dec 23, 2016 7:17:44 AM com.datatorrent.bufferserver.server.Server 
> unregistered
> INFO: Server stopped listening at /0:0:0:0:0:0:0:0:40904
> Dec 23, 2016 7:17:44 AM com.datatorrent.stram.engine.StreamingContainer main
> INFO: Exit status for container: 1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1213) WordCount example failure on Apex Runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1213:
-

Assignee: Thomas Weise  (was: Kenneth Knowles)

> WordCount example failure on Apex Runner
> 
>
> Key: BEAM-1213
> URL: https://issues.apache.org/jira/browse/BEAM-1213
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Sandeep Deshmukh
>Assignee: Thomas Weise
> Fix For: First stable release
>
>
> Tried running WordCount example using apex runner (on YARN) with 
> https://github.com/apache/incubator-beam/pull/1693 , fix for httpclient 
> version issue fixed.
> Create /tmp/input-beam/ dir on all nodes and pushed a test data in that 
> directory. Ran the following command in examples/java/
> ~/bin/apache-maven-3.3.9/bin/mvn  compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.WordCount 
> -Dexec.args="--inputFile=/tmp/input-beam/java.java --output=/tmp/output-beam/ 
> --runner=ApexRunner --embeddedExecution=false" -Papex-runner
> INFO: using 
> /disk2/devhdfs/nn_20161013/usercache/sandeep/appcache/application_1482319446115_4336/container_1482319446115_4336_01_14/tmp/chkp5198889890309038810
>  as the basepath for checkpointing.
> Dec 23, 2016 7:17:44 AM com.datatorrent.stram.engine.StreamingContainer main
> SEVERE: Fatal exception in container!
> java.lang.IllegalArgumentException: Invalid slice: offset=1426, length=0 
> array.length=67108864
> at com.datatorrent.netlet.util.Slice.(Slice.java:43)
> at 
> com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
> at 
> com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
> at 
> com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
> at com.datatorrent.bufferserver.server.Server.purge(Server.java:204)
> at 
> com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:776)
> at 
> com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:705)
> at 
> com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:310)
> Dec 23, 2016 7:17:44 AM com.datatorrent.bufferserver.server.Server 
> unregistered
> INFO: Server stopped listening at /0:0:0:0:0:0:0:0:40904
> Dec 23, 2016 7:17:44 AM com.datatorrent.stram.engine.StreamingContainer main
> INFO: Exit status for container: 1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1717) Maven release/deploy tries to uploads some artifacts more than once

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997789#comment-15997789
 ] 

Jean-Baptiste Onofré commented on BEAM-1717:


Thanks Davor !

> Maven release/deploy tries to uploads some artifacts more than once 
> 
>
> Key: BEAM-1717
> URL: https://issues.apache.org/jira/browse/BEAM-1717
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Amit Sela
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Running maven {{release}} or {{deploy}} causes some artifacts to deploy more 
> than once which fails deployments to release Nexus.
> While this is not an issue for the Apache release process (because it uses a 
> staging Nexus), this affects users who wish to deploy their own fork. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3045

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2174) Allow coder factories to create Coders for a wider range of types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997756#comment-15997756
 ] 

ASF GitHub Bot commented on BEAM-2174:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2910

[BEAM-2174] Update CoderRegistry to allow creating coders through 
CoderFactory for a wider range of types

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This removes getInstanceComponents and hard codes the inference for 
commonly "known" types into Create removing the additionally complexity that 
was brought along with it.

Also:
* @DefaultCoder to become a CoderFactory
* Created a ProtoCoderFactory which delegates to the ProtoCoder for all 
Message types
* Created a SerializableCoderFactory which delegates to the 
SerializableCoder for all Serializable types
* Created a WritableCoderFactory which delegates to the Hadoop 
WritableCoder for all Hadoop Writable types
* Renamed getDefaultCoder to getCoder on CoderRegistry

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam coder_registrar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2910


commit 917cc45411c602143450e109390b6a7750fc2f06
Author: Luke Cwik 
Date:   2017-05-05T02:55:47Z

[BEAM-2174] Update CoderRegistry to allow creating coders through 
CoderFactory for a wider range of types




> Allow coder factories to create Coders for a wider range of types
> -
>
> Key: BEAM-2174
> URL: https://issues.apache.org/jira/browse/BEAM-2174
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> By allowing the CoderFactory to see they type it can look at annotations on 
> the type allowing:
> * @DefaultCoder to become a CoderFactory
> * Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
> Message types
> * Creating a SerializableCoderFactory which delegates to the 
> SerializableCoder for all Serializable types
> * Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
> for all Hadoop Writable types
> This requires plumbing through TypeDescriptor as the primary method of 
> looking up coders within the CoderRegistry
> This also removes the concept of fallback coder providers since every coder 
> factory is treated as a fallback.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2910: [BEAM-2174] Update CoderRegistry to allow creating ...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2910

[BEAM-2174] Update CoderRegistry to allow creating coders through 
CoderFactory for a wider range of types

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This removes getInstanceComponents and hard codes the inference for 
commonly "known" types into Create removing the additionally complexity that 
was brought along with it.

Also:
* @DefaultCoder to become a CoderFactory
* Created a ProtoCoderFactory which delegates to the ProtoCoder for all 
Message types
* Created a SerializableCoderFactory which delegates to the 
SerializableCoder for all Serializable types
* Created a WritableCoderFactory which delegates to the Hadoop 
WritableCoder for all Hadoop Writable types
* Renamed getDefaultCoder to getCoder on CoderRegistry

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam coder_registrar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2910


commit 917cc45411c602143450e109390b6a7750fc2f06
Author: Luke Cwik 
Date:   2017-05-05T02:55:47Z

[BEAM-2174] Update CoderRegistry to allow creating coders through 
CoderFactory for a wider range of types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2174) Allow coder factories to create Coders for a wider range of types

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2174:

Description: 
By allowing the CoderFactory to see they type it can look at annotations on the 
type allowing:
* @DefaultCoder to become a CoderFactory
* Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
Message types
* Creating a SerializableCoderFactory which delegates to the SerializableCoder 
for all Serializable types
* Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
for all Hadoop Writable types

This requires plumbing through TypeDescriptor as the primary method of looking 
up coders within the CoderRegistry

This also removes the concept of fallback coder providers since every coder 
factory is treated as a fallback.

  was:
By allowing the CoderFactory to see they type it can look at annotations on the 
type allowing:
* @DefaultCoder to become a CoderFactory
* Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
Message types
* Creating a SerializableCoderFactory which delegates to the SerializableCoder 
for all Serializable types
* Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
for all Hadoop Writable types

This requires plumbing through TypeDescriptor as the primary method of looking 
up coders within the CoderRegistry


> Allow coder factories to create Coders for a wider range of types
> -
>
> Key: BEAM-2174
> URL: https://issues.apache.org/jira/browse/BEAM-2174
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> By allowing the CoderFactory to see they type it can look at annotations on 
> the type allowing:
> * @DefaultCoder to become a CoderFactory
> * Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
> Message types
> * Creating a SerializableCoderFactory which delegates to the 
> SerializableCoder for all Serializable types
> * Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
> for all Hadoop Writable types
> This requires plumbing through TypeDescriptor as the primary method of 
> looking up coders within the CoderRegistry
> This also removes the concept of fallback coder providers since every coder 
> factory is treated as a fallback.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3663

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2067) Add support for generic CoderProvider -> CoderFactory mapping with CoderRegistrar

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-2067.
-
Resolution: Duplicate

Duplicate of
https://issues.apache.org/jira/browse/BEAM-2174

> Add support for generic CoderProvider -> CoderFactory mapping with 
> CoderRegistrar
> -
>
> Key: BEAM-2067
> URL: https://issues.apache.org/jira/browse/BEAM-2067
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: First stable release
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Instead of relying on Class to CoderFactory in the CoderRegistrar, we 
> should really ask if CoderFactory can handle a specific type or allow for 
> CoderProvider be able answer on its behalf.
> This includes re-adding support for ProtoCoder as a "fallback" coder provider.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2174) Allow coder factories to create Coders for a wider range of types

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2174:

Fix Version/s: First stable release

> Allow coder factories to create Coders for a wider range of types
> -
>
> Key: BEAM-2174
> URL: https://issues.apache.org/jira/browse/BEAM-2174
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> By allowing the CoderFactory to see they type it can look at annotations on 
> the type allowing:
> * @DefaultCoder to become a CoderFactory
> * Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
> Message types
> * Creating a SerializableCoderFactory which delegates to the 
> SerializableCoder for all Serializable types
> * Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
> for all Hadoop Writable types
> This requires plumbing through TypeDescriptor as the primary method of 
> looking up coders within the CoderRegistry



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #1365

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2174) Allow coder factories to create Coders for a wider range of types

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2174:

Description: 
By allowing the CoderFactory to see they type it can look at annotations on the 
type allowing:
* @DefaultCoder to become a CoderFactory
* Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
Message types
* Creating a SerializableCoderFactory which delegates to the SerializableCoder 
for all Serializable types
* Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
for all Hadoop Writable types

This requires plumbing through TypeDescriptor as the primary method of looking 
up coders within the CoderRegistry

> Allow coder factories to create Coders for a wider range of types
> -
>
> Key: BEAM-2174
> URL: https://issues.apache.org/jira/browse/BEAM-2174
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> By allowing the CoderFactory to see they type it can look at annotations on 
> the type allowing:
> * @DefaultCoder to become a CoderFactory
> * Creating a ProtoCoderFactory which delegates to the ProtoCoder for all 
> Message types
> * Creating a SerializableCoderFactory which delegates to the 
> SerializableCoder for all Serializable types
> * Creating a WritableCoderFactory which delegates to the Hadoop WritableCoder 
> for all Hadoop Writable types
> This requires plumbing through TypeDescriptor as the primary method of 
> looking up coders within the CoderRegistry



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997742#comment-15997742
 ] 

Aviem Zur edited comment on BEAM-1582 at 5/5/17 3:03 AM:
-

I don't think we should disable this test as it is a very important one.

In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
 so I'm not sure why this is such an issue? 

Removing this from 'First Stable Release' for now, if anyone objects please let 
us know.

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})


was (Author: aviemzur):
I don't think we should disable this test as it is a very important one.

In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue? 

Removing this from 'First Stable Release' for now, if anyone objects please let 
us know.

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
>  Labels: flake
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1582:
---

Assignee: (was: Amit Sela)

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Priority: Minor
>  Labels: flake
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1582:

Fix Version/s: (was: First stable release)

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>  Labels: flake
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997742#comment-15997742
 ] 

Aviem Zur edited comment on BEAM-1582 at 5/5/17 3:02 AM:
-

I don't think we should disable this test as it is a very important one.

In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue? 

Removing this from 'First Stable Release' for now, if anyone objects please let 
us know.

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})


was (Author: aviemzur):
I don't think we should disable this test as it is a very important one.
In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue? 

Removing this from 'First Stable Release' for now, if anyone objects please let 
us know.

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
>  Labels: flake
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1582:

Priority: Minor  (was: Major)

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
>  Labels: flake
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997742#comment-15997742
 ] 

Aviem Zur edited comment on BEAM-1582 at 5/5/17 3:01 AM:
-

I don't think we should disable this test as it is a very important one.
In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue?

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})


was (Author: aviemzur):
I don't think we should disable this test as it is a very important one.
In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue?

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>  Labels: flake
> Fix For: First stable release
>
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997742#comment-15997742
 ] 

Aviem Zur edited comment on BEAM-1582 at 5/5/17 3:02 AM:
-

I don't think we should disable this test as it is a very important one.
In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue? 

Removing this from 'First Stable Release' for now, if anyone objects please let 
us know.

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})


was (Author: aviemzur):
I don't think we should disable this test as it is a very important one.
In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue?

(P.S. The {{@ValidatesRunner}} suggestion was so this would run in PostCommit 
rather than PreCommit so if it ever does flake it won't cause friction for 
contributors working on something unrelated, but I see this is already the 
case, this test already runs in PostCommit only using a different annotation 
{{@UsesCheckpointRecovery}})

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>  Labels: flake
> Fix For: First stable release
>
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2174) Allow coder factories to create Coders for a wider range of types

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-2174:
---

 Summary: Allow coder factories to create Coders for a wider range 
of types
 Key: BEAM-2174
 URL: https://issues.apache.org/jira/browse/BEAM-2174
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997742#comment-15997742
 ] 

Aviem Zur commented on BEAM-1582:
-

I don't think we should disable this test as it is a very important one.
In the last 30 builds that Jenkins saves the history for this test has not 
failed once: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/
So I'm not sure why this is such an issue?

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>  Labels: flake
> Fix For: First stable release
>
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1953

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2679

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2173) PostCommit failure in DoFnSignaturesTest

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2173.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> PostCommit failure in DoFnSignaturesTest
> 
>
> Key: BEAM-2173
> URL: https://issues.apache.org/jira/browse/BEAM-2173
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>
> The package {{state}} is now in the path to timers, so the expectation that 
> the error message "doesn't mention state" is not going to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3044

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1283) DoFn finishBundle should be required to specify the window for output

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997734#comment-15997734
 ] 

Kenneth Knowles commented on BEAM-1283:
---

I think we should get in the habit of using separate tickets (perhaps subtasks) 
for different SDKs. Otherwise ownership is unclear and parallelism is not 
obvious.

> DoFn finishBundle should be required to specify the window for output
> -
>
> Key: BEAM-1283
> URL: https://issues.apache.org/jira/browse/BEAM-1283
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Kenneth Knowles
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The spec is here in Javadoc: 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to 
> use the {{WindowFn}} of the input {{PCollection}} to determine what windows 
> the element should be in, throwing an exception if the {{WindowFn}} attempts 
> to access any information about the input element. The output element will 
> have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically 
> wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is 
> allowed to see the element type. The spec just avoids this by limiting which 
> {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The 
> spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged 
> then you'll just have a bogus proto window regardless of explicit timestamp 
> or not.
> The use cases for these methods are best addressed by state plus window 
> expiry callback, so we should revisit this spec and probably just wipe it.
> There are some rare case where you might need to output from {{FinishBundle}} 
> in a way that is not _actually_ sensitive to bundling (perhaps modulo some 
> downstream notion of equivalence) in which case you had better know what 
> window you are outputting to. Often it should be the global window.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #3662

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2173) PostCommit failure in DoFnSignaturesTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997733#comment-15997733
 ] 

ASF GitHub Bot commented on BEAM-2173:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2909


> PostCommit failure in DoFnSignaturesTest
> 
>
> Key: BEAM-2173
> URL: https://issues.apache.org/jira/browse/BEAM-2173
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> The package {{state}} is now in the path to timers, so the expectation that 
> the error message "doesn't mention state" is not going to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2909: [BEAM-2173] Adjust error message expectation in DoF...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2909


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Adjust error message expectation in DoFnSignaturesTest

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master b130d7aac -> 7903e59c4


Adjust error message expectation in DoFnSignaturesTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6be71c9a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6be71c9a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6be71c9a

Branch: refs/heads/master
Commit: 6be71c9a1c2ec43478e2ddaedaba67608de129ae
Parents: b130d7a
Author: Kenneth Knowles 
Authored: Thu May 4 19:03:50 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 19:03:50 2017 -0700

--
 .../apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6be71c9a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
index 5b97a55..cffb0ad 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
@@ -385,7 +385,7 @@ public class DoFnSignaturesTest {
 thrown.expectMessage("my-id");
 thrown.expectMessage("myfield1");
 thrown.expectMessage("myfield2");
-thrown.expectMessage(not(mentionsState()));
+thrown.expectMessage(not(containsString("State"))); // lowercase "state" 
is in the package name
 thrown.expectMessage(mentionsTimers());
 DoFnSignature sig =
 DoFnSignatures.getSignature(
@@ -407,7 +407,7 @@ public class DoFnSignaturesTest {
 thrown.expectMessage("Timer declarations must be final");
 thrown.expectMessage("Non-final field");
 thrown.expectMessage("myfield");
-thrown.expectMessage(not(mentionsState()));
+thrown.expectMessage(not(containsString("State"))); // lowercase "state" 
is in the package name
 thrown.expectMessage(mentionsTimers());
 DoFnSignature sig =
 DoFnSignatures.getSignature(

[2/2] beam git commit: This closes #2909: Adjust error message expectation in DoFnSignaturesTest

[kenn]

This closes #2909: Adjust error message expectation in DoFnSignaturesTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7903e59c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7903e59c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7903e59c

Branch: refs/heads/master
Commit: 7903e59c4e07822f8def85ee6a7f8ef3ccb1ca7a
Parents: b130d7a 6be71c9
Author: Kenneth Knowles 
Authored: Thu May 4 19:21:59 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 19:21:59 2017 -0700

--
 .../apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Resolved] (BEAM-1283) DoFn finishBundle should be required to specify the window for output

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1283.
---
Resolution: Fixed

> DoFn finishBundle should be required to specify the window for output
> -
>
> Key: BEAM-1283
> URL: https://issues.apache.org/jira/browse/BEAM-1283
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The spec is here in Javadoc: 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to 
> use the {{WindowFn}} of the input {{PCollection}} to determine what windows 
> the element should be in, throwing an exception if the {{WindowFn}} attempts 
> to access any information about the input element. The output element will 
> have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically 
> wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is 
> allowed to see the element type. The spec just avoids this by limiting which 
> {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The 
> spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged 
> then you'll just have a bogus proto window regardless of explicit timestamp 
> or not.
> The use cases for these methods are best addressed by state plus window 
> expiry callback, so we should revisit this spec and probably just wipe it.
> There are some rare case where you might need to output from {{FinishBundle}} 
> in a way that is not _actually_ sensitive to bundling (perhaps modulo some 
> downstream notion of equivalence) in which case you had better know what 
> window you are outputting to. Often it should be the global window.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (BEAM-1283) DoFn finishBundle should be required to specify the window for output

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reopened BEAM-1283:
---
  Assignee: Sourabh Bajaj  (was: Thomas Groh)

Assigning to Sourabh for the python change. This is complete within the Java 
SDK.

> DoFn finishBundle should be required to specify the window for output
> -
>
> Key: BEAM-1283
> URL: https://issues.apache.org/jira/browse/BEAM-1283
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Kenneth Knowles
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The spec is here in Javadoc: 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to 
> use the {{WindowFn}} of the input {{PCollection}} to determine what windows 
> the element should be in, throwing an exception if the {{WindowFn}} attempts 
> to access any information about the input element. The output element will 
> have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically 
> wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is 
> allowed to see the element type. The spec just avoids this by limiting which 
> {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The 
> spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged 
> then you'll just have a bogus proto window regardless of explicit timestamp 
> or not.
> The use cases for these methods are best addressed by state plus window 
> expiry callback, so we should revisit this spec and probably just wipe it.
> There are some rare case where you might need to output from {{FinishBundle}} 
> in a way that is not _actually_ sensitive to bundling (perhaps modulo some 
> downstream notion of equivalence) in which case you had better know what 
> window you are outputting to. Often it should be the global window.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1316) DoFn#startBundle should not be able to output

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1316.
---
Resolution: Fixed

> DoFn#startBundle should not be able to output
> -
>
> Key: BEAM-1316
> URL: https://issues.apache.org/jira/browse/BEAM-1316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> While within startBundle and finishBundle, the window in which elements are 
> output is not generally defined. Elements must always be output from within a 
> windowed context, or the {{WindowFn}} used by the {{PCollection}} may not 
> operate appropriately.
> startBundle and finishBundle are suitable for operational duties, similarly 
> to {{setup}} and {{teardown}}, but within the scope of some collection of 
> input elements. This includes actions such as clearing field state within a 
> DoFn and ensuring all live RPCs complete successfully before committing 
> inputs.
> Sometimes it might be reasonable to output from {{@FinishBundle}} but it is 
> hard to imagine a situation where output from {{@StartBundle}} is useful in a 
> way that doesn't seriously abuse things.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-1412) Maven install postcommit broken (to be investigated)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-1412.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Maven install postcommit broken (to be investigated)
> 
>
> Key: BEAM-1412
> URL: https://issues.apache.org/jira/browse/BEAM-1412
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/ 
> went red / yellow / red around here: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/2556/
> I've just looked in on two that were both a failure to transfer a dependency. 
> There may be other things going on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2173) PostCommit failure in DoFnSignaturesTest

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2173:
-

 Summary: PostCommit failure in DoFnSignaturesTest
 Key: BEAM-2173
 URL: https://issues.apache.org/jira/browse/BEAM-2173
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Blocker


The package {{state}} is now in the path to timers, so the expectation that the 
error message "doesn't mention state" is not going to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2173) PostCommit failure in DoFnSignaturesTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997722#comment-15997722
 ] 

ASF GitHub Bot commented on BEAM-2173:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2909

[BEAM-2173] Adjust error message expectation in DoFnSignaturesTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam state-message

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2909


commit 6be71c9a1c2ec43478e2ddaedaba67608de129ae
Author: Kenneth Knowles 
Date:   2017-05-05T02:03:50Z

Adjust error message expectation in DoFnSignaturesTest




> PostCommit failure in DoFnSignaturesTest
> 
>
> Key: BEAM-2173
> URL: https://issues.apache.org/jira/browse/BEAM-2173
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> The package {{state}} is now in the path to timers, so the expectation that 
> the error message "doesn't mention state" is not going to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2909: [BEAM-2173] Adjust error message expectation in DoF...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2909

[BEAM-2173] Adjust error message expectation in DoFnSignaturesTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam state-message

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2909


commit 6be71c9a1c2ec43478e2ddaedaba67608de129ae
Author: Kenneth Knowles 
Date:   2017-05-05T02:03:50Z

Adjust error message expectation in DoFnSignaturesTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Apex #1364

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1952

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2143) (Mis)Running Dataflow Wordcount gives non-helpful errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997704#comment-15997704
 ] 

ASF GitHub Bot commented on BEAM-2143:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2907

[BEAM-2143]: Fix default temp location for DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
- https://github.com/apache/beam/pull/2602 made `gs://some-bucket` path 
invalid and expects it to be of the form `gs://some-bucket/basename`. So adding 
`/temp` as the basename for the default `tempLocation`.
- Also fix log messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam df_temp_loc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2907.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2907


commit ea165bfba449cd5572f9342bf8107018dc584a30
Author: Vikas Kedigehalli 
Date:   2017-05-05T01:30:35Z

Fix default temp location for DataflowRunner




> (Mis)Running Dataflow Wordcount gives non-helpful errors
> 
>
> Key: BEAM-2143
> URL: https://issues.apache.org/jira/browse/BEAM-2143
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-gcp
>Reporter: Ben Chambers
>Assignee: Vikas Kedigehalli
>Priority: Blocker
> Fix For: First stable release
>
>
> If you run a pipeline and forget to specify `tempLocation` (but specify 
> something else, such as `stagingLocation`) you get two messages indicating 
> you forgot to specify `stagingLocation`. 
> One says "no stagingLocation specified, choosing ..." the other says "error, 
> the staging location isn't readable" (if you give it just a bucket and not an 
> object within a bucket).
> This is surprising to me as a user, since (1) I specified a staging location 
> and (2) the flag I actually need to modify is `--tempLocation`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2016) Delete HDFSFileSource/Sink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997705#comment-15997705
 ] 

ASF GitHub Bot commented on BEAM-2016:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2908

[BEAM-2016] Delete HdfsFileSource & Sink

R: @jbonofre 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam remove-hdfs-file-sourcesink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2908.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2908


commit a07d48913ae8ff34c5cab76187b09023e429
Author: Dan Halperin 
Date:   2017-05-05T01:33:16Z

[BEAM-2016] Delete HdfsFileSource & Sink




> Delete HDFSFileSource/Sink
> --
>
> Key: BEAM-2016
> URL: https://issues.apache.org/jira/browse/BEAM-2016
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Jean-Baptiste Onofré
> Fix For: First stable release
>
>
> After https://issues.apache.org/jira/browse/BEAM-2005, delete 
> https://github.com/apache/beam/tree/master/sdks/java/io/hdfs since it'll be 
> redundant with the ability to read HDFS via other file-based IOs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2907: [BEAM-2143]: Fix default temp location for Dataflow...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2907

[BEAM-2143]: Fix default temp location for DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
- https://github.com/apache/beam/pull/2602 made `gs://some-bucket` path 
invalid and expects it to be of the form `gs://some-bucket/basename`. So adding 
`/temp` as the basename for the default `tempLocation`.
- Also fix log messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam df_temp_loc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2907.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2907


commit ea165bfba449cd5572f9342bf8107018dc584a30
Author: Vikas Kedigehalli 
Date:   2017-05-05T01:30:35Z

Fix default temp location for DataflowRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2908: [BEAM-2016] Delete HdfsFileSource & Sink

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2908

[BEAM-2016] Delete HdfsFileSource & Sink

R: @jbonofre 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam remove-hdfs-file-sourcesink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2908.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2908


commit a07d48913ae8ff34c5cab76187b09023e429
Author: Dan Halperin 
Date:   2017-05-05T01:33:16Z

[BEAM-2016] Delete HdfsFileSource & Sink




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2678

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1951

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2053) Add HDFS file provider to wordcount examples

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-2053:
-

Assignee: (was: Daniel Halperin)

> Add HDFS file provider to wordcount examples
> 
>
> Key: BEAM-2053
> URL: https://issues.apache.org/jira/browse/BEAM-2053
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Priority: Minor
>
> Instructions for running the example on YARN refer to HDFS, but HDFS is 
> currently not supported in the example project:
> https://beam.apache.org/documentation/runners/apex/
> Using local files is sufficient in a sandbox, for multi-node cluster we need 
> a distributed FS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2155) Investigate problem creating Map PCollection view on Dataflow

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-2155:
-

Assignee: (was: Daniel Halperin)

> Investigate problem creating Map PCollection view on Dataflow
> -
>
> Key: BEAM-2155
> URL: https://issues.apache.org/jira/browse/BEAM-2155
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>
> A user reports a side input seemingly never becoming available: 
> http://stackoverflow.com/questions/43755254/dataflow-map-side-input-issue
> The code seems good at first glance, and it works in the direct runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #1363

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2067) Add support for generic CoderProvider -> CoderFactory mapping with CoderRegistrar

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-2067:
---
Fix Version/s: First stable release

> Add support for generic CoderProvider -> CoderFactory mapping with 
> CoderRegistrar
> -
>
> Key: BEAM-2067
> URL: https://issues.apache.org/jira/browse/BEAM-2067
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: First stable release
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Instead of relying on Class to CoderFactory in the CoderRegistrar, we 
> should really ask if CoderFactory can handle a specific type or allow for 
> CoderProvider be able answer on its behalf.
> This includes re-adding support for ProtoCoder as a "fallback" coder provider.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

beam git commit: Revert "This closes #2905"

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 9f27c33ec -> b130d7aac


Revert "This closes #2905"

I misread Jenkins and should not have merged. I am sorry.

This reverts commit 9f27c33ec7e7c61afbca0395f932275b354eb428, reversing
changes made to 5fc3d335919207c23bc6fd2047e9e38351754ff1.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b130d7aa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b130d7aa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b130d7aa

Branch: refs/heads/master
Commit: b130d7aac466860ed5a7abec0bef33d9e0dd3c6d
Parents: 9f27c33
Author: Dan Halperin 
Authored: Thu May 4 18:14:13 2017 -0700
Committer: Dan Halperin 
Committed: Thu May 4 18:14:13 2017 -0700

--
 examples/java/pom.xml   |  12 -
 .../beam/examples/WindowedWordCountIT.java  |   8 +-
 .../org/apache/beam/examples/WordCountIT.java   |   2 +-
 .../examples/testing/ExplicitShardedFile.java   | 126 ---
 .../examples/testing/FileChecksumMatcher.java   | 168 --
 .../testing/FileChecksumMatcherTest.java| 148 
 .../examples/testing/NumberedShardedFile.java   | 226 ---
 .../testing/NumberedShardedFileTest.java| 182 ---
 .../beam/examples/testing/ShardedFile.java  |  42 
 .../beam/sdk/coders/StructuralByteArray.java|   5 +-
 .../beam/sdk/testing/FileChecksumMatcher.java   | 169 ++
 .../beam/sdk/testing/MatcherDeserializer.java   |   4 +-
 .../beam/sdk/testing/MatcherSerializer.java |   4 +-
 .../org/apache/beam/sdk/util/CoderUtils.java|   8 +-
 .../beam/sdk/util/ExplicitShardedFile.java  | 125 ++
 .../beam/sdk/util/NumberedShardedFile.java  | 225 ++
 .../org/apache/beam/sdk/util/ShardedFile.java   |  42 
 .../sdk/testing/FileChecksumMatcherTest.java| 147 
 .../beam/sdk/util/NumberedShardedFileTest.java  | 182 +++
 19 files changed, 905 insertions(+), 920 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b130d7aa/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 09473cd..d673da2 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -492,11 +492,6 @@
 
 
 
-  com.google.code.findbugs
-  jsr305
-
-
-
   org.apache.avro
   avro
 
@@ -559,13 +554,6 @@
 -->
 
   org.apache.beam
-  beam-sdks-java-core
-  tests
-  test
-
-
-
-  org.apache.beam
   beam-runners-direct-java
   test
 

http://git-wip-us.apache.org/repos/asf/beam/blob/b130d7aa/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
index 01bc402..b5eddb5 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -33,21 +33,21 @@ import java.util.concurrent.ThreadLocalRandom;
 
 import org.apache.beam.examples.common.ExampleUtils;
 import org.apache.beam.examples.common.WriteOneFilePerWindow.PerWindowFiles;
-import org.apache.beam.examples.testing.ExplicitShardedFile;
-import org.apache.beam.examples.testing.FileChecksumMatcher;
-import org.apache.beam.examples.testing.NumberedShardedFile;
-import org.apache.beam.examples.testing.ShardedFile;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.io.FileSystems;
 import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.testing.FileChecksumMatcher;
 import org.apache.beam.sdk.testing.SerializableMatcher;
 import org.apache.beam.sdk.testing.StreamingIT;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.testing.TestPipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.util.ExplicitShardedFile;
 import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.NumberedShardedFile;
+import org.apache.beam.sdk.util.ShardedFile;
 import org.hamcrest.Description;
 import org.hamcrest.TypeSafeMatcher;
 import org.joda.time.Duration;

http://git-wip-us.apache.org/repos/asf/beam/blob/b130d7aa/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #3660

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3661

[Apache Jenkins Server]

See 


Changes:

[dhalperi] [BEAM-1871] Move out test only utilities used only in examples/java 
out

--
[...truncated 1.44 MB...]
2017-05-05T01:05:15.207 [INFO]   symbol:   class FastNanoClockAndSleeper
2017-05-05T01:05:15.207 [INFO]   location: class 
it.pkg.testing.FileChecksumMatcherTest
2017-05-05T01:05:15.207 [INFO] 2017-05-05\T\01:05:15.207 [ERROR] 
:[55,4]
 cannot find symbol
2017-05-05T01:05:15.207 [INFO]   symbol:   class Mock
2017-05-05T01:05:15.207 [INFO]   location: class 
it.pkg.testing.NumberedShardedFileTest
2017-05-05T01:05:15.207 [INFO] 2017-05-05\T\01:05:15.207 [ERROR] 
:[50,4]
 cannot find symbol
2017-05-05T01:05:15.207 [INFO]   symbol:   class Mock
2017-05-05T01:05:15.207 [INFO]   location: class 
it.pkg.testing.FileChecksumMatcherTest
2017-05-05T01:05:15.207 [INFO] 2017-05-05\T\01:05:15.207 [INFO] 16 errors 
2017-05-05T01:05:15.207 [INFO] 2017-05-05\T\01:05:15.207 [INFO] 
-
2017-05-05T01:05:15.208 [INFO] 2017-05-05\T\01:05:15.208 [INFO] 

2017-05-05T01:05:15.208 [INFO] 2017-05-05\T\01:05:15.208 [INFO] BUILD FAILURE
2017-05-05T01:05:15.208 [INFO] 2017-05-05\T\01:05:15.208 [INFO] 

2017-05-05T01:05:15.208 [INFO] 2017-05-05\T\01:05:15.208 [INFO] Total time: 
7.396 s
2017-05-05T01:05:15.208 [INFO] 2017-05-05\T\01:05:15.208 [INFO] Finished at: 
2017-05-05T01:05:15+00:00
2017-05-05T01:05:15.557 [INFO] 2017-05-05\T\01:05:15.557 [INFO] Final Memory: 
34M/334M
2017-05-05T01:05:15.557 [INFO] 2017-05-05\T\01:05:15.557 [INFO] 

2017-05-05T01:05:15.561 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] Failed to 
execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:testCompile 
(default-testCompile) on project basic: Compilation failure: Compilation 
failure:
2017-05-05T01:05:15.561 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[24,26]
 package org.mockito does not exist
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[24,1]
 static import only from classes and interfaces
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[25,26]
 package org.mockito does not exist
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[25,1]
 static import only from classes and interfaces
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[26,26]
 package org.mockito does not exist
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[26,1]
 static import only from classes and interfaces
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] 
:[37,35]
 cannot find symbol
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] symbol:   
class FastNanoClockAndSleeper
2017-05-05T01:05:15.562 [INFO] 2017-05-05\T\01:05:15.561 [ERROR] location: 
package 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1950

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3043

[Apache Jenkins Server]

See 


--
GitHub pull request #2891 of commit 02e80a7d5890ac2281d970cb1977399f79431d5b, 
no merge conflicts.
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse refs/remotes/origin/pr/2891/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/2891/merge^{commit} # timeout=10
 > git rev-parse origin/pr/2891/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse refs/remotes/origin/pr/2891/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/2891/merge^{commit} # timeout=10
 > git rev-parse origin/pr/2891/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse refs/remotes/origin/pr/2891/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/2891/merge^{commit} # timeout=10
 > git rev-parse origin/pr/2891/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3042

[Apache Jenkins Server]

See 


--
[...truncated 1.86 MB...]
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2677

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2119

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997662#comment-15997662
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2905


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2905: [BEAM-1871] Move out test only utilities used only ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1871] Move out test only utilities used only in examples/java out of sdks/java/core.

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 5fc3d3359 -> 9f27c33ec


[BEAM-1871] Move out test only utilities used only in examples/java out of 
sdks/java/core.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c158e05c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c158e05c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c158e05c

Branch: refs/heads/master
Commit: c158e05ccdd1d06bd4f49c972288bf28c4e5d703
Parents: 5fc3d33
Author: Luke Cwik 
Authored: Thu May 4 16:14:34 2017 -0700
Committer: Dan Halperin 
Committed: Thu May 4 17:40:42 2017 -0700

--
 examples/java/pom.xml   |  12 +
 .../beam/examples/WindowedWordCountIT.java  |   8 +-
 .../org/apache/beam/examples/WordCountIT.java   |   2 +-
 .../examples/testing/ExplicitShardedFile.java   | 126 +++
 .../examples/testing/FileChecksumMatcher.java   | 168 ++
 .../testing/FileChecksumMatcherTest.java| 148 
 .../examples/testing/NumberedShardedFile.java   | 226 +++
 .../testing/NumberedShardedFileTest.java| 182 +++
 .../beam/examples/testing/ShardedFile.java  |  42 
 .../beam/sdk/coders/StructuralByteArray.java|   5 +-
 .../beam/sdk/testing/FileChecksumMatcher.java   | 169 --
 .../beam/sdk/testing/MatcherDeserializer.java   |   4 +-
 .../beam/sdk/testing/MatcherSerializer.java |   4 +-
 .../org/apache/beam/sdk/util/CoderUtils.java|   8 +-
 .../beam/sdk/util/ExplicitShardedFile.java  | 125 --
 .../beam/sdk/util/NumberedShardedFile.java  | 225 --
 .../org/apache/beam/sdk/util/ShardedFile.java   |  42 
 .../sdk/testing/FileChecksumMatcherTest.java| 147 
 .../beam/sdk/util/NumberedShardedFileTest.java  | 182 ---
 19 files changed, 920 insertions(+), 905 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c158e05c/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index d673da2..09473cd 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -492,6 +492,11 @@
 
 
 
+  com.google.code.findbugs
+  jsr305
+
+
+
   org.apache.avro
   avro
 
@@ -554,6 +559,13 @@
 -->
 
   org.apache.beam
+  beam-sdks-java-core
+  tests
+  test
+
+
+
+  org.apache.beam
   beam-runners-direct-java
   test
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c158e05c/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
index b5eddb5..01bc402 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -33,21 +33,21 @@ import java.util.concurrent.ThreadLocalRandom;
 
 import org.apache.beam.examples.common.ExampleUtils;
 import org.apache.beam.examples.common.WriteOneFilePerWindow.PerWindowFiles;
+import org.apache.beam.examples.testing.ExplicitShardedFile;
+import org.apache.beam.examples.testing.FileChecksumMatcher;
+import org.apache.beam.examples.testing.NumberedShardedFile;
+import org.apache.beam.examples.testing.ShardedFile;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.io.FileSystems;
 import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.StreamingOptions;
-import org.apache.beam.sdk.testing.FileChecksumMatcher;
 import org.apache.beam.sdk.testing.SerializableMatcher;
 import org.apache.beam.sdk.testing.StreamingIT;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.testing.TestPipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.util.ExplicitShardedFile;
 import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.sdk.util.NumberedShardedFile;
-import org.apache.beam.sdk.util.ShardedFile;
 import org.hamcrest.Description;
 import org.hamcrest.TypeSafeMatcher;
 import org.joda.time.Duration;

http://git-wip-us.apache.org/repos/asf/beam/blob/c158e05c/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java 

[2/2] beam git commit: This closes #2905

[dhalperi]

This closes #2905


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9f27c33e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9f27c33e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9f27c33e

Branch: refs/heads/master
Commit: 9f27c33ec7e7c61afbca0395f932275b354eb428
Parents: 5fc3d33 c158e05
Author: Dan Halperin 
Authored: Thu May 4 17:40:45 2017 -0700
Committer: Dan Halperin 
Committed: Thu May 4 17:40:45 2017 -0700

--
 examples/java/pom.xml   |  12 +
 .../beam/examples/WindowedWordCountIT.java  |   8 +-
 .../org/apache/beam/examples/WordCountIT.java   |   2 +-
 .../examples/testing/ExplicitShardedFile.java   | 126 +++
 .../examples/testing/FileChecksumMatcher.java   | 168 ++
 .../testing/FileChecksumMatcherTest.java| 148 
 .../examples/testing/NumberedShardedFile.java   | 226 +++
 .../testing/NumberedShardedFileTest.java| 182 +++
 .../beam/examples/testing/ShardedFile.java  |  42 
 .../beam/sdk/coders/StructuralByteArray.java|   5 +-
 .../beam/sdk/testing/FileChecksumMatcher.java   | 169 --
 .../beam/sdk/testing/MatcherDeserializer.java   |   4 +-
 .../beam/sdk/testing/MatcherSerializer.java |   4 +-
 .../org/apache/beam/sdk/util/CoderUtils.java|   8 +-
 .../beam/sdk/util/ExplicitShardedFile.java  | 125 --
 .../beam/sdk/util/NumberedShardedFile.java  | 225 --
 .../org/apache/beam/sdk/util/ShardedFile.java   |  42 
 .../sdk/testing/FileChecksumMatcherTest.java| 147 
 .../beam/sdk/util/NumberedShardedFileTest.java  | 182 ---
 19 files changed, 920 insertions(+), 905 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3659

[Apache Jenkins Server]

See 
<https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/3659/display/redirect?page=changes>

Changes:

[klk] Move Java sdk.util.state to sdk.state

[klk] Add @Internal and @Experimental to state package

[klk] Move user-facing timer-related classes out of util

[klk] Mark TimeDomain experimental alongside Timers; improve javadoc

[klk] Move WindowingStrategy from util to values

[klk] Move PathValidator adjacent to other filesystem bits

[klk] Move SideInputReader to runners/core-java

[klk] Move Reshuffle and ReshuffleTrigger out of util

[klk] Move ValueWithRecordId to sdk.values, annotated

[klk] Move some PCollectionView bits out of util

[klk] Update Dataflow worker to 20170504-2

--
[...truncated 1.57 MB...]

Expected: iterable containing [<ValueInGlobalWindow{value=FinishBundle, 
pane=PaneInfo.NO_FIRING}>]
 but: item 0: was <TimestampedValueInGlobalWindow{value=FinishBundle, 
timestamp=294247-01-09T04:00:54.775Z, pane=PaneInfo.NO_FIRING}>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at 
org.apache.beam.fn.harness.control.ProcessBundleHandlerTest.testCreatingAndProcessingDoFn(ProcessBundleHandlerTest.java:436)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)

2017-05-05T00:37:07.352 [INFO] Running 
org.apache.beam.fn.harness.control.BeamFnControlClientTest
2017-05-05T00:37:07.389 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.036 s - in 
org.apache.beam.fn.harness.control.BeamFnControlClientTest
2017-05-05T00:37:07.390 [INFO] Running 
org.apache.beam.fn.harness.control.RegisterHandlerTest
2017-05-05T00:37:07.493 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.101 s - in 
org.apache.beam.fn.harness.control.Regist

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1949

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3658

[Apache Jenkins Server]

See 


--
[...truncated 1.55 MB...]
java.lang.AssertionError: 

Expected: iterable containing []
 but: item 0: was 
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at 
org.apache.beam.fn.harness.control.ProcessBundleHandlerTest.testCreatingAndProcessingDoFn(ProcessBundleHandlerTest.java:436)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)

2017-05-05T00:32:36.097 [INFO] Running 
org.apache.beam.fn.harness.control.BeamFnControlClientTest
2017-05-05T00:32:36.128 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.029 s - in 
org.apache.beam.fn.harness.control.BeamFnControlClientTest
2017-05-05T00:32:36.128 [INFO] Running 
org.apache.beam.fn.harness.control.RegisterHandlerTest
2017-05-05T00:32:36.231 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.101 s - in 
org.apache.beam.fn.harness.control.RegisterHandlerTest
2017-05-05T00:32:36.231 [INFO] Running org.apache.beam.fn.harness.FnHarnessTest
2017-05-05T00:32:36.893 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.66 s - in org.apache.beam.fn.harness.FnHarnessTest
2017-05-05T00:32:36.893 [INFO] Running 
org.apache.beam.fn.harness.stream.AdvancingPhaserTest
2017-05-05T00:32:36.900 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.006 s - in 
org.apache.beam.fn.harness.stream.AdvancingPhaserTest
2017-05-05T00:32:36.900 [INFO] Running 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2676

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2172) ProcessBundleHandlerTest.testCreatingAndProcessingDoFn fails at HEAD

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997642#comment-15997642
 ] 

ASF GitHub Bot commented on BEAM-2172:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2906


> ProcessBundleHandlerTest.testCreatingAndProcessingDoFn fails at HEAD
> 
>
> Key: BEAM-2172
> URL: https://issues.apache.org/jira/browse/BEAM-2172
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
>
> java.lang.AssertionError: 
> Expected: iterable containing [ pane=PaneInfo.NO_FIRING}>]
>  but: item 0: was  timestamp=294247-01-09T04:00:54.775Z, pane=PaneInfo.NO_FIRING}>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:956)
>   at org.junit.Assert.assertThat(Assert.java:923)
>   at 
> org.apache.beam.fn.harness.control.ProcessBundleHandlerTest.testCreatingAndProcessingDoFn(ProcessBundleHandlerTest.java:436)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2675

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2906: [BEAM-2172] Update expectation in ProcessBundleHand...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2906


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #2906

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 3042d761a -> 5fc3d3359


This closes #2906


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5fc3d335
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5fc3d335
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5fc3d335

Branch: refs/heads/master
Commit: 5fc3d335919207c23bc6fd2047e9e38351754ff1
Parents: 3042d76 040dbd4
Author: Thomas Groh 
Authored: Thu May 4 17:14:48 2017 -0700
Committer: Thomas Groh 
Committed: Thu May 4 17:14:48 2017 -0700

--
 .../beam/fn/harness/control/ProcessBundleHandlerTest.java  | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--

[2/2] beam git commit: Update expectation in ProcessBundleHandlerTest

[tgroh]

Update expectation in ProcessBundleHandlerTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/040dbd4f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/040dbd4f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/040dbd4f

Branch: refs/heads/master
Commit: 040dbd4fd21a31dd3eb0f34fb9610ce857111db8
Parents: 3042d76
Author: Thomas Groh 
Authored: Thu May 4 17:01:24 2017 -0700
Committer: Thomas Groh 
Committed: Thu May 4 17:14:48 2017 -0700

--
 .../beam/fn/harness/control/ProcessBundleHandlerTest.java  | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/040dbd4f/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
--
diff --git 
a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
 
b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
index 3e06369..748ffea 100644
--- 
a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
+++ 
b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
@@ -18,6 +18,7 @@
 
 package org.apache.beam.fn.harness.control;
 
+import static 
org.apache.beam.sdk.util.WindowedValue.timestampedValueInGlobalWindow;
 import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.empty;
@@ -433,7 +434,10 @@ public class ProcessBundleHandlerTest {
 additionalOutputValues.clear();
 
 Iterables.getOnlyElement(finishFunctions).run();
-assertThat(mainOutputValues, 
contains(valueInGlobalWindow("FinishBundle")));
+assertThat(
+mainOutputValues,
+contains(
+timestampedValueInGlobalWindow("FinishBundle", 
GlobalWindow.INSTANCE.maxTimestamp(;
 mainOutputValues.clear();
   }
 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3657

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-2170] PubsubMessageWithAttributesCoder should not NPE on 
messages

[kirpichov] Makes PubsubMessagePayloadOnlyCoder not require whole-stream context

[kirpichov] Moves coder choice into PubsubSource

[davor] [BEAM-2144] Remove transform.reflect and utils from Javadocs

--
[...truncated 1.55 MB...]
java.lang.AssertionError: 

Expected: iterable containing []
 but: item 0: was 
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at 
org.apache.beam.fn.harness.control.ProcessBundleHandlerTest.testCreatingAndProcessingDoFn(ProcessBundleHandlerTest.java:436)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)

2017-05-05T00:13:13.602 [INFO] Running 
org.apache.beam.fn.harness.control.BeamFnControlClientTest
2017-05-05T00:13:13.655 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.051 s - in 
org.apache.beam.fn.harness.control.BeamFnControlClientTest
2017-05-05T00:13:13.655 [INFO] Running 
org.apache.beam.fn.harness.control.RegisterHandlerTest
2017-05-05T00:13:13.774 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.118 s - in 
org.apache.beam.fn.harness.control.RegisterHandlerTest
2017-05-05T00:13:13.775 [INFO] Running org.apache.beam.fn.harness.FnHarnessTest
2017-05-05T00:13:14.607 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.831 s - in org.apache.beam.fn.harness.FnHarnessTest
2017-05-05T00:13:14.607 [INFO] Running 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2674

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997608#comment-15997608
 ] 

ASF GitHub Bot commented on BEAM-1340:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2891


> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[14/19] beam git commit: Move WindowingStrategy from util to values

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTest.java
index eae465c..71bfdb5 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTest.java
@@ -43,6 +43,7 @@ import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.WindowingStrategy;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.junit.Rule;

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
index b704d07..58e2bbd 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
@@ -40,7 +40,6 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SimpleFunction;
-import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollection.IsBounded;
 import org.hamcrest.Matchers;
 import org.junit.Rule;

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
--
diff --git 
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
 
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
index 3057050..d4e6f63 100644
--- 
a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
+++ 
b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
@@ -23,7 +23,6 @@ import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
 import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.Coder.Context;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.util.VarInt;

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
--
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
index f29aeb9..46bd6d4 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
@@ -61,8 +61,8 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.util.NullSideInputReader;
 import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
--
diff --git 
a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
 
b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
index 6d5e230..3e06369 100644
--- 
a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
+++ 
b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
@@ -75,9 +75,9 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.WindowedValue;
-import 

[09/19] beam git commit: Move user-facing timer-related classes out of util

[kenn]

Move user-facing timer-related classes out of util


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ca41af8f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ca41af8f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ca41af8f

Branch: refs/heads/master
Commit: ca41af8fe4711ab4a81c2a33746a64e64fb0ca37
Parents: ac01ec7
Author: Kenneth Knowles 
Authored: Tue May 2 12:46:46 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:55 2017 -0700

--
 .../operators/ApexGroupByKeyOperator.java   |  2 +-
 .../operators/ApexTimerInternals.java   |  2 +-
 .../operators/ApexTimerInternalsTest.java   |  2 +-
 .../runners/core/construction/Triggers.java |  2 +-
 .../construction/PTransformMatchersTest.java|  8 +-
 .../apache/beam/runners/core/DoFnAdapters.java  |  2 +-
 .../apache/beam/runners/core/DoFnRunner.java|  2 +-
 .../runners/core/InMemoryTimerInternals.java|  2 +-
 .../core/LateDataDroppingDoFnRunner.java|  2 +-
 ...eBoundedSplittableProcessElementInvoker.java |  2 +-
 .../beam/runners/core/ProcessFnRunner.java  |  2 +-
 .../core/PushbackSideInputDoFnRunner.java   |  2 +-
 .../org/apache/beam/runners/core/ReduceFn.java  |  2 +-
 .../runners/core/ReduceFnContextFactory.java|  4 +-
 .../beam/runners/core/ReduceFnRunner.java   |  2 +-
 .../beam/runners/core/SimpleDoFnRunner.java |  6 +-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |  2 +-
 .../core/SimplePushbackSideInputDoFnRunner.java |  2 +-
 .../beam/runners/core/SplittableParDo.java  |  2 +-
 .../beam/runners/core/StatefulDoFnRunner.java   |  2 +-
 .../beam/runners/core/TimerInternals.java   |  2 +-
 .../AfterDelayFromFirstElementStateMachine.java |  2 +-
 .../AfterProcessingTimeStateMachine.java|  2 +-
 ...rSynchronizedProcessingTimeStateMachine.java |  2 +-
 .../triggers/AfterWatermarkStateMachine.java|  2 +-
 .../triggers/DefaultTriggerStateMachine.java|  2 +-
 .../core/triggers/TriggerStateMachine.java  |  2 +-
 .../TriggerStateMachineContextFactory.java  |  4 +-
 .../triggers/TriggerStateMachineRunner.java |  2 +-
 .../core/InMemoryTimerInternalsTest.java|  2 +-
 .../runners/core/KeyedWorkItemCoderTest.java|  2 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   |  2 +-
 .../beam/runners/core/ReduceFnTester.java   |  2 +-
 .../beam/runners/core/SimpleDoFnRunnerTest.java |  8 +-
 .../SimplePushbackSideInputDoFnRunnerTest.java  |  2 +-
 .../beam/runners/core/TimerInternalsTest.java   |  2 +-
 .../triggers/TriggerStateMachineTester.java |  4 +-
 .../core/triggers/TriggerStateMachinesTest.java |  2 +-
 .../runners/direct/DirectTimerInternals.java|  2 +-
 .../beam/runners/direct/WatermarkManager.java   |  2 +-
 .../direct/DirectTimerInternalsTest.java|  2 +-
 ...leManagerRemovingTransformEvaluatorTest.java |  2 +-
 .../runners/direct/EvaluationContextTest.java   |  2 +-
 .../runners/direct/WatermarkManagerTest.java|  2 +-
 .../metrics/DoFnRunnerWithMetricsUpdate.java|  2 +-
 .../wrappers/streaming/DoFnOperator.java|  2 +-
 .../flink/streaming/DoFnOperatorTest.java   |  8 +-
 .../spark/stateful/SparkTimerInternals.java |  2 +-
 .../translation/DoFnRunnerWithMetrics.java  |  2 +-
 .../apache/beam/sdk/options/ValueProviders.java |  1 -
 .../org/apache/beam/sdk/state/TimeDomain.java   | 43 +++
 .../java/org/apache/beam/sdk/state/Timer.java   | 78 
 .../org/apache/beam/sdk/state/TimerSpec.java| 30 
 .../org/apache/beam/sdk/state/TimerSpecs.java   | 41 ++
 .../java/org/apache/beam/sdk/state/Timers.java  | 58 +++
 .../beam/sdk/testing/TestPipelineOptions.java   |  2 -
 .../org/apache/beam/sdk/transforms/DoFn.java|  6 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |  2 +-
 .../beam/sdk/transforms/GroupIntoBatches.java   |  8 +-
 .../org/apache/beam/sdk/transforms/Regex.java   |  1 -
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  2 +-
 .../sdk/transforms/reflect/DoFnInvoker.java |  2 +-
 .../sdk/transforms/reflect/DoFnSignature.java   |  4 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  |  4 +-
 .../transforms/windowing/AfterWatermark.java|  2 +-
 .../org/apache/beam/sdk/util/ClassPath.java |  1 +
 .../org/apache/beam/sdk/util/TimeDomain.java| 43 ---
 .../java/org/apache/beam/sdk/util/Timer.java| 78 
 .../org/apache/beam/sdk/util/TimerSpec.java | 30 
 .../org/apache/beam/sdk/util/TimerSpecs.java| 41 --
 .../java/org/apache/beam/sdk/util/Timers.java   | 58 ---
 .../sdk/io/DrunkWritableByteChannelFactory.java |  1 -
 .../org/apache/beam/sdk/io/TFRecordIOTest.java  |  2 -
 .../apache/beam/sdk/metrics/GaugeCellTest.java  |  1 +
 

[19/19] beam git commit: This closes #2891: Vital moves of user-facing stuff out of util

[kenn]

This closes #2891: Vital moves of user-facing stuff out of util

  Update Dataflow worker to 20170504-2
  Move some PCollectionView bits out of util
  Move ValueWithRecordId to sdk.values, annotated
  Move Reshuffle and ReshuffleTrigger out of util
  Move SideInputReader to runners/core-java
  Move PathValidator adjacent to other filesystem bits
  Move WindowingStrategy from util to values
  Mark TimeDomain experimental alongside Timers; improve javadoc
  Move user-facing timer-related classes out of util
  Add @Internal and @Experimental to state package
  Move Java sdk.util.state to sdk.state


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3042d761
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3042d761
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3042d761

Branch: refs/heads/master
Commit: 3042d761a223029a5be2b5ccab257396c71b07e6
Parents: 2f4dd8d 02e80a7
Author: Kenneth Knowles <k...@google.com>
Authored: Thu May 4 17:02:18 2017 -0700
Committer: Kenneth Knowles <k...@google.com>
Committed: Thu May 4 17:02:18 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|   2 +-
 .../beam/runners/apex/ApexRunnerRegistrar.java  |   1 -
 .../beam/runners/apex/ApexRunnerResult.java |   2 -
 .../beam/runners/apex/ApexYarnLauncher.java |   2 -
 .../translation/ReadUnboundedTranslator.java|   1 -
 .../apex/translation/TransformTranslator.java   |   1 -
 .../operators/ApexGroupByKeyOperator.java   |   6 +-
 .../operators/ApexParDoOperator.java|   6 +-
 .../ApexReadUnboundedInputOperator.java |   2 -
 .../operators/ApexTimerInternals.java   |   2 +-
 .../translation/utils/ApexStateInternals.java   |  20 +-
 .../utils/CoderAdapterStreamCodec.java  |   2 -
 .../translation/utils/StateInternalsProxy.java  |   4 +-
 .../utils/ValueAndCoderKryoSerializable.java|   2 -
 .../apex/translation/utils/ValuesSource.java|   2 -
 .../beam/runners/apex/ApexYarnLauncherTest.java |   2 -
 .../apex/examples/UnboundedTextSource.java  |   2 -
 .../runners/apex/examples/WordCountTest.java|   2 -
 .../translation/ApexGroupByKeyOperatorTest.java |   4 +-
 .../apex/translation/ParDoTranslatorTest.java   |   2 +-
 .../operators/ApexTimerInternalsTest.java   |   2 +-
 .../utils/ApexStateInternalsTest.java   |  12 +-
 .../translation/utils/CollectionSource.java |   2 -
 .../runners/core/construction/PCollections.java |   2 +-
 .../core/construction/PrimitiveCreate.java  |   2 +-
 .../core/construction/SdkComponents.java|   2 +-
 .../runners/core/construction/Triggers.java |   4 +-
 .../core/construction/WindowingStrategies.java  |   4 +-
 .../core/construction/PCollectionsTest.java |   2 +-
 .../construction/PTransformMatchersTest.java|  18 +-
 .../construction/ReplacementOutputsTest.java|   2 +-
 .../core/construction/SdkComponentsTest.java|   4 +-
 .../construction/WindowingStrategiesTest.java   |   4 +-
 .../apache/beam/runners/core/DoFnAdapters.java  |   4 +-
 .../apache/beam/runners/core/DoFnRunner.java|   2 +-
 .../apache/beam/runners/core/DoFnRunners.java   |   4 +-
 .../runners/core/GlobalCombineFnRunner.java |   1 -
 .../runners/core/GlobalCombineFnRunners.java|  41 +-
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |   2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   2 +-
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   3 +-
 .../core/GroupByKeyViaGroupByKeyOnly.java   |   2 +-
 .../runners/core/InMemoryStateInternals.java|  22 +-
 .../runners/core/InMemoryTimerInternals.java|   2 +-
 .../core/LateDataDroppingDoFnRunner.java|   4 +-
 .../apache/beam/runners/core/LateDataUtils.java |   2 +-
 .../runners/core/MergingActiveWindowSet.java|   2 +-
 .../beam/runners/core/MergingStateAccessor.java |   2 +-
 .../apache/beam/runners/core/NonEmptyPanes.java |   8 +-
 .../beam/runners/core/NullSideInputReader.java  |  61 ++
 ...eBoundedSplittableProcessElementInvoker.java |   5 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 +-
 .../beam/runners/core/ProcessFnRunner.java  |   3 +-
 .../core/PushbackSideInputDoFnRunner.java   |   2 +-
 .../core/ReadyCheckingSideInputReader.java  |  34 +
 .../org/apache/beam/runners/core/ReduceFn.java  |   6 +-
 .../runners/core/ReduceFnContextFactory.java|  13 +-
 .../beam/runners/core/ReduceFnRunner.java   |   7 +-
 .../beam/runners/core/SideInputHandler.java |   5 +-
 .../beam/runners/core/SideInputReader.java  |  47 ++
 .../beam/runners/core/SimpleDoFnRunner.java |  13 +-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   5 +-
 .../core/SimplePushbackSideInputDoFnRunner.java |   3 +-
 .../beam/runners/core/SplittableParDo.java  |   8 +-
 .../apache/beam/runners/core/StateAccessor.java |   2 +-
 .../beam/runners/core/StateIn

[08/19] beam git commit: Move user-facing timer-related classes out of util

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/ca41af8f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
index 2b6d5be..8ae2c65 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
@@ -63,6 +63,7 @@ import net.bytebuddy.matcher.ElementMatchers;
 import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.state.Timer;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignature.OnTimerMethod;
@@ -77,7 +78,6 @@ import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParam
 import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter;
 import org.apache.beam.sdk.transforms.splittabledofn.HasDefaultTracker;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
-import org.apache.beam.sdk.util.Timer;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.values.TypeDescriptor;
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ca41af8f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
index d5a1a94..6fd4052 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.transforms.reflect;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderRegistry;
 import org.apache.beam.sdk.state.State;
+import org.apache.beam.sdk.state.Timer;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFn.FinishBundle;
 import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
@@ -28,7 +29,6 @@ import org.apache.beam.sdk.transforms.DoFn.StateId;
 import org.apache.beam.sdk.transforms.DoFn.TimerId;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.Timer;
 
 /**
  * Interface for invoking the {@code DoFn} processing methods.

http://git-wip-us.apache.org/repos/asf/beam/blob/ca41af8f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 72ad4b0..0b4bf90 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -29,6 +29,8 @@ import javax.annotation.Nullable;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.state.State;
 import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.Timer;
+import org.apache.beam.sdk.state.TimerSpec;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFn.StateId;
 import org.apache.beam.sdk.transforms.DoFn.TimerId;
@@ -38,8 +40,6 @@ import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParam
 import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.Timer;
-import org.apache.beam.sdk.util.TimerSpec;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TypeDescriptor;
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ca41af8f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
index 

[16/19] beam git commit: Move WindowingStrategy from util to values

[kenn]

Move WindowingStrategy from util to values

WindowingStrategy is a property on PCollection that transform authors
regularly mess with. It is part of the public API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b61e5bb2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b61e5bb2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b61e5bb2

Branch: refs/heads/master
Commit: b61e5bb2b2e1951c62e7defb752561a0a1af5456
Parents: b4c808a
Author: Kenneth Knowles 
Authored: Tue May 2 12:53:26 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:55 2017 -0700

--
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../translation/ApexGroupByKeyOperatorTest.java |   2 +-
 .../apex/translation/ParDoTranslatorTest.java   |   2 +-
 .../runners/core/construction/PCollections.java |   2 +-
 .../core/construction/PrimitiveCreate.java  |   2 +-
 .../core/construction/SdkComponents.java|   2 +-
 .../core/construction/WindowingStrategies.java  |   4 +-
 .../core/construction/PCollectionsTest.java |   2 +-
 .../construction/PTransformMatchersTest.java|   2 +-
 .../construction/ReplacementOutputsTest.java|   2 +-
 .../core/construction/SdkComponentsTest.java|   4 +-
 .../construction/WindowingStrategiesTest.java   |   4 +-
 .../apache/beam/runners/core/DoFnRunners.java   |   2 +-
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |   2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   2 +-
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   2 +-
 .../core/GroupByKeyViaGroupByKeyOnly.java   |   2 +-
 .../core/LateDataDroppingDoFnRunner.java|   2 +-
 .../apache/beam/runners/core/LateDataUtils.java |   2 +-
 .../apache/beam/runners/core/NonEmptyPanes.java |   4 +-
 .../org/apache/beam/runners/core/ReduceFn.java  |   2 +-
 .../runners/core/ReduceFnContextFactory.java|   2 +-
 .../beam/runners/core/ReduceFnRunner.java   |   4 +-
 .../beam/runners/core/SimpleDoFnRunner.java |   2 +-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   2 +-
 .../beam/runners/core/SplittableParDo.java  |   2 +-
 .../beam/runners/core/StatefulDoFnRunner.java   |   2 +-
 .../apache/beam/runners/core/WatermarkHold.java |   2 +-
 ...roupAlsoByWindowViaOutputBufferDoFnTest.java |   2 +-
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../core/LateDataDroppingDoFnRunnerTest.java|   2 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   |   4 +-
 .../beam/runners/core/ReduceFnTester.java   |   4 +-
 .../beam/runners/core/SideInputHandlerTest.java |   2 +-
 .../beam/runners/core/SimpleDoFnRunnerTest.java |   2 +-
 .../beam/runners/core/SplittableParDoTest.java  |   2 +-
 .../runners/core/StatefulDoFnRunnerTest.java|   2 +-
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../beam/runners/direct/DirectGroupByKey.java   |   2 +-
 .../beam/runners/direct/EvaluationContext.java  |   2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 +-
 .../beam/runners/direct/ParDoEvaluator.java |   2 +-
 .../direct/ParDoMultiOverrideFactory.java   |   2 +-
 .../beam/runners/direct/SideInputContainer.java |   2 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   2 +-
 .../direct/TestStreamEvaluatorFactory.java  |   2 +-
 .../direct/WatermarkCallbackExecutor.java   |   2 +-
 .../runners/direct/CommittedResultTest.java |   2 +-
 .../runners/direct/EvaluationContextTest.java   |   2 +-
 .../direct/KeyedPValueTrackingVisitorTest.java  |   2 +-
 .../runners/direct/SideInputContainerTest.java  |   2 +-
 .../StatefulParDoEvaluatorFactoryTest.java  |   2 +-
 .../runners/direct/ViewOverrideFactoryTest.java |   2 +-
 .../direct/WatermarkCallbackExecutorTest.java   |   2 +-
 .../direct/WriteWithShardingFactoryTest.java|   2 +-
 .../flink/FlinkBatchTransformTranslators.java   |   2 +-
 .../flink/FlinkBatchTranslationContext.java |   2 +-
 .../FlinkStreamingTransformTranslators.java |   2 +-
 .../functions/AbstractFlinkCombineRunner.java   |   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |   2 +-
 .../functions/FlinkPartialReduceFunction.java   |   2 +-
 .../functions/FlinkReduceFunction.java  |   2 +-
 .../functions/FlinkSideInputReader.java |   2 +-
 .../functions/FlinkStatefulDoFnFunction.java|   2 +-
 .../functions/HashingFlinkCombineRunner.java|   2 +-
 .../functions/SortingFlinkCombineRunner.java|   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../streaming/SplittableDoFnOperator.java   |   2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   2 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   2 +-
 

[17/19] beam git commit: Move some PCollectionView bits out of util

[kenn]

Move some PCollectionView bits out of util


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c83cc744
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c83cc744
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c83cc744

Branch: refs/heads/master
Commit: c83cc744a69f735ac134471705e3403b9d5edd34
Parents: b2553ca
Author: Kenneth Knowles 
Authored: Wed May 3 20:34:54 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:56 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|   2 +-
 .../construction/PTransformMatchersTest.java|   2 +-
 .../runners/direct/PCollectionViewWindow.java   |  67 +++
 .../beam/runners/direct/SideInputContainer.java |   1 -
 .../runners/direct/SideInputContainerTest.java  |   2 +-
 .../direct/ViewEvaluatorFactoryTest.java|   2 +-
 .../runners/direct/ViewOverrideFactoryTest.java |   2 +-
 .../direct/WriteWithShardingFactoryTest.java|   2 +-
 .../flink/FlinkStreamingViewOverrides.java  |   2 +-
 .../runners/dataflow/BatchViewOverrides.java|   2 +-
 .../org/apache/beam/sdk/transforms/Combine.java |   2 +-
 .../org/apache/beam/sdk/transforms/View.java|   2 +-
 .../beam/sdk/util/PCollectionViewWindow.java|  67 ---
 .../apache/beam/sdk/util/PCollectionViews.java  | 497 ---
 .../beam/sdk/values/PCollectionViews.java   | 495 ++
 .../beam/sdk/transforms/DoFnTesterTest.java |   2 +-
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   2 +-
 17 files changed, 574 insertions(+), 577 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c83cc744/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index 1c845c6..e1828c3 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -60,9 +60,9 @@ import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.View.AsIterable;
 import org.apache.beam.sdk.transforms.View.AsSingleton;
-import org.apache.beam.sdk.util.PCollectionViews;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PCollectionViews;
 import org.apache.hadoop.conf.Configuration;
 
 /**

http://git-wip-us.apache.org/repos/asf/beam/blob/c83cc744/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
index cb28c34..6271234 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
@@ -60,13 +60,13 @@ import org.apache.beam.sdk.transforms.ViewFn;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.util.PCollectionViews;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollection.IsBounded;
 import org.apache.beam.sdk.values.PCollectionList;
 import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PCollectionViews;
 import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.PValue;
 import org.apache.beam.sdk.values.TupleTag;

http://git-wip-us.apache.org/repos/asf/beam/blob/c83cc744/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PCollectionViewWindow.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PCollectionViewWindow.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PCollectionViewWindow.java
new file mode 100644
index 000..7a7d8ff
--- /dev/null
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PCollectionViewWindow.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache 

[04/19] beam git commit: Move Java sdk.util.state to sdk.state

[kenn]

Move Java sdk.util.state to sdk.state


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/78e0acca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/78e0acca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/78e0acca

Branch: refs/heads/master
Commit: 78e0accad907420675aefd0ac617bcf5d2a82619
Parents: defb554
Author: Kenneth Knowles 
Authored: Tue May 2 10:29:33 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:34 2017 -0700

--
 .../translation/utils/ApexStateInternals.java   |  20 +-
 .../translation/utils/StateInternalsProxy.java  |   4 +-
 .../utils/ApexStateInternalsTest.java   |  12 +-
 .../construction/PTransformMatchersTest.java|   6 +-
 .../apache/beam/runners/core/DoFnAdapters.java  |   2 +-
 .../runners/core/InMemoryStateInternals.java|  22 +-
 .../runners/core/MergingActiveWindowSet.java|   2 +-
 .../beam/runners/core/MergingStateAccessor.java |   2 +-
 .../apache/beam/runners/core/NonEmptyPanes.java |   4 +-
 ...eBoundedSplittableProcessElementInvoker.java |   2 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 +-
 .../org/apache/beam/runners/core/ReduceFn.java  |   2 +-
 .../runners/core/ReduceFnContextFactory.java|   6 +-
 .../beam/runners/core/SideInputHandler.java |   4 +-
 .../beam/runners/core/SimpleDoFnRunner.java |   4 +-
 .../beam/runners/core/SplittableParDo.java  |   4 +-
 .../apache/beam/runners/core/StateAccessor.java |   2 +-
 .../beam/runners/core/StateInternals.java   |   4 +-
 .../apache/beam/runners/core/StateMerging.java  |  14 +-
 .../apache/beam/runners/core/StateTable.java|   4 +-
 .../org/apache/beam/runners/core/StateTag.java  |  16 +-
 .../org/apache/beam/runners/core/StateTags.java |  20 +-
 .../beam/runners/core/StatefulDoFnRunner.java   |   4 +-
 .../beam/runners/core/SystemReduceFn.java   |   8 +-
 .../core/TestInMemoryStateInternals.java|   4 +-
 .../apache/beam/runners/core/WatermarkHold.java |   4 +-
 .../AfterDelayFromFirstElementStateMachine.java |   4 +-
 .../core/triggers/AfterPaneStateMachine.java|   2 +-
 .../TriggerStateMachineContextFactory.java  |   2 +-
 .../triggers/TriggerStateMachineRunner.java |   2 +-
 .../core/InMemoryStateInternalsTest.java|  16 +-
 .../runners/core/StatefulDoFnRunnerTest.java|   6 +-
 .../CopyOnAccessInMemoryStateInternals.java |  18 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   2 +-
 .../CopyOnAccessInMemoryStateInternalsTest.java |  14 +-
 .../runners/direct/EvaluationContextTest.java   |   2 +-
 .../StatefulParDoEvaluatorFactoryTest.java  |   6 +-
 .../flink/FlinkDetachedRunnerResult.java|   1 -
 .../types/EncodedValueSerializer.java   |   2 -
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../state/FlinkBroadcastStateInternals.java |  20 +-
 .../state/FlinkKeyGroupStateInternals.java  |  20 +-
 .../state/FlinkSplitStateInternals.java |  20 +-
 .../streaming/state/FlinkStateInternals.java|  20 +-
 .../flink/streaming/DoFnOperatorTest.java   |   6 +-
 .../FlinkBroadcastStateInternalsTest.java   |  10 +-
 .../FlinkKeyGroupStateInternalsTest.java|   4 +-
 .../streaming/FlinkSplitStateInternalsTest.java |   4 +-
 .../streaming/FlinkStateInternalsTest.java  |  12 +-
 .../BatchStatefulParDoOverridesTest.java|   6 +-
 .../DataflowPipelineTranslatorTest.java |   6 +-
 .../spark/stateful/SparkStateInternals.java |  20 +-
 .../src/main/resources/beam/findbugs-filter.xml |   2 +-
 .../org/apache/beam/sdk/state/BagState.java |  28 +
 .../apache/beam/sdk/state/CombiningState.java   |  53 ++
 .../apache/beam/sdk/state/GroupingState.java|  42 ++
 .../org/apache/beam/sdk/state/MapState.java |  79 +++
 .../apache/beam/sdk/state/ReadableState.java|  51 ++
 .../apache/beam/sdk/state/ReadableStates.java   |  45 ++
 .../org/apache/beam/sdk/state/SetState.java |  45 ++
 .../java/org/apache/beam/sdk/state/State.java   |  32 +
 .../org/apache/beam/sdk/state/StateBinder.java  |  66 ++
 .../org/apache/beam/sdk/state/StateContext.java |  43 ++
 .../apache/beam/sdk/state/StateContexts.java|  51 ++
 .../org/apache/beam/sdk/state/StateSpec.java|  53 ++
 .../org/apache/beam/sdk/state/StateSpecs.java   | 629 +++
 .../org/apache/beam/sdk/state/ValueState.java   |  37 ++
 .../beam/sdk/state/WatermarkHoldState.java  |  42 ++
 .../org/apache/beam/sdk/state/package-info.java |  22 +
 .../apache/beam/sdk/testing/UsesMapState.java   |   2 +-
 .../apache/beam/sdk/testing/UsesSetState.java   |   2 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|   4 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   2 +-
 .../beam/sdk/transforms/GroupIntoBatches.java   |  10 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |   2 +-
 

[11/19] beam git commit: Move ValueWithRecordId to sdk.values, annotated

[kenn]

Move ValueWithRecordId to sdk.values, annotated


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b2553caf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b2553caf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b2553caf

Branch: refs/heads/master
Commit: b2553caf1350eaea3caefe55d5af414694c96424
Parents: e0b3f80
Author: Kenneth Knowles 
Authored: Wed May 3 20:19:51 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:55 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   |   2 +-
 .../beam/runners/spark/TestSparkRunner.java |   2 +-
 .../sdk/io/BoundedReadFromUnboundedSource.java  |   2 +-
 .../apache/beam/sdk/util/ValueWithRecordId.java | 134 --
 .../beam/sdk/values/ValueWithRecordId.java  | 138 +++
 .../beam/sdk/util/ValueWithRecordIdTest.java|  34 -
 .../beam/sdk/values/ValueWithRecordIdTest.java  |  34 +
 7 files changed, 175 insertions(+), 171 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b2553caf/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 7123316..57da61b 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -121,7 +121,6 @@ import org.apache.beam.sdk.util.InstanceBuilder;
 import org.apache.beam.sdk.util.MimeTypes;
 import org.apache.beam.sdk.util.NameUtils;
 import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.beam.sdk.util.ValueWithRecordId;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
@@ -132,6 +131,7 @@ import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.PInput;
 import org.apache.beam.sdk.values.PValue;
 import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.ValueWithRecordId;
 import org.apache.beam.sdk.values.WindowingStrategy;
 import org.joda.time.DateTimeUtils;
 import org.joda.time.DateTimeZone;

http://git-wip-us.apache.org/repos/asf/beam/blob/b2553caf/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
index 6808d7b..a6851c4 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
@@ -48,11 +48,11 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.util.ValueWithRecordId;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PValue;
 import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.ValueWithRecordId;
 import org.apache.commons.io.FileUtils;
 import org.joda.time.Duration;
 import org.joda.time.Instant;

http://git-wip-us.apache.org/repos/asf/beam/blob/b2553caf/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
index e54176f..d9adf92 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
@@ -37,9 +37,9 @@ import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.util.FluentBackoff;
 import org.apache.beam.sdk.util.NameUtils;
-import org.apache.beam.sdk.util.ValueWithRecordId;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.ValueWithRecordId;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 

[02/19] beam git commit: Move Java sdk.util.state to sdk.state

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/78e0acca/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java
deleted file mode 100644
index 48fa742..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util.state;
-
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.CombineWithContext;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-
-/**
- * Visitor for binding a {@link StateSpec} and to the associated {@link State}.
- */
-public interface StateBinder {
-   ValueState bindValue(
-  String id, StateSpec spec, Coder coder);
-
-   BagState bindBag(
-  String id, StateSpec spec, Coder elemCoder);
-
-   SetState bindSet(
-  String id, StateSpec spec, Coder elemCoder);
-
-   MapState bindMap(
-  String id,
-  StateSpec> spec,
-  Coder mapKeyCoder,
-  Coder mapValueCoder);
-
-   CombiningState 
bindCombining(
-  String id,
-  StateSpec> spec,
-  Coder accumCoder,
-  Combine.CombineFn combineFn);
-
-   CombiningState 
bindCombiningWithContext(
-  String id,
-  StateSpec> spec,
-  Coder accumCoder,
-  CombineWithContext.CombineFnWithContext 
combineFn);
-
-  /**
-   * Bind to a watermark {@link StateSpec}.
-   *
-   * This accepts the {@link TimestampCombiner} that dictates how watermark 
hold timestamps added
-   * to the returned {@link WatermarkHoldState} are to be combined.
-   */
-  WatermarkHoldState bindWatermark(
-  String id,
-  StateSpec spec,
-  TimestampCombiner timestampCombiner);
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/78e0acca/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContext.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContext.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContext.java
deleted file mode 100644
index 887a5f1..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContext.java
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util.state;
-
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.values.PCollectionView;
-
-/**
- * Information accessible the state API.
- */
-public interface StateContext {
-  /**
-   * Returns the {@code PipelineOptions} specified with the
-   * {@link org.apache.beam.sdk.runners.PipelineRunner}.
-   */
-  PipelineOptions getPipelineOptions();
-
-  /**
-   * Returns the value of the side input for the 

[05/19] beam git commit: Move PathValidator adjacent to other filesystem bits

[kenn]

Move PathValidator adjacent to other filesystem bits


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1d4b1ed4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1d4b1ed4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1d4b1ed4

Branch: refs/heads/master
Commit: 1d4b1ed49ce2212d961ad37325a3bfce8964f673
Parents: b61e5bb
Author: Kenneth Knowles 
Authored: Wed May 3 19:36:45 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:55 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   |  2 +-
 .../beam/sdk/io/fs/NoopPathValidator.java   | 52 ++
 .../apache/beam/sdk/io/fs/PathValidator.java| 58 
 .../apache/beam/sdk/util/NoopPathValidator.java |  4 +-
 .../org/apache/beam/sdk/util/PathValidator.java | 55 ---
 .../sdk/extensions/gcp/options/GcpOptions.java  |  2 +-
 .../sdk/extensions/gcp/options/GcsOptions.java  |  2 +-
 .../apache/beam/sdk/util/GcsPathValidator.java  |  1 +
 8 files changed, 117 insertions(+), 59 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1d4b1ed4/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index d5e650e..ce824c6 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -88,6 +88,7 @@ import org.apache.beam.sdk.io.FileSystems;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.io.WriteFiles;
+import org.apache.beam.sdk.io.fs.PathValidator;
 import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
 import org.apache.beam.sdk.io.fs.ResourceId;
 import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
@@ -118,7 +119,6 @@ import 
org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.InstanceBuilder;
 import org.apache.beam.sdk.util.MimeTypes;
 import org.apache.beam.sdk.util.NameUtils;
-import org.apache.beam.sdk.util.PathValidator;
 import org.apache.beam.sdk.util.ReleaseInfo;
 import org.apache.beam.sdk.util.Reshuffle;
 import org.apache.beam.sdk.util.ValueWithRecordId;

http://git-wip-us.apache.org/repos/asf/beam/blob/1d4b1ed4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/NoopPathValidator.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/NoopPathValidator.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/NoopPathValidator.java
new file mode 100644
index 000..d5be8f0
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/NoopPathValidator.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.fs;
+
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.options.PipelineOptions;
+
+/**
+ * For internal use only; no backwards compatibility guarantees.
+ *
+ * Noop implementation of {@link PathValidator}. All paths are allowed and 
returned unchanged.
+ */
+@Internal
+public class NoopPathValidator implements PathValidator {
+
+  private NoopPathValidator() {
+  }
+
+  public static PathValidator fromOptions(
+  @SuppressWarnings("unused") PipelineOptions options) {
+return new NoopPathValidator();
+  }
+
+  @Override
+  public void validateInputFilePatternSupported(String filepattern) {}
+
+  @Override
+  public void validateOutputFilePrefixSupported(String filePrefix) {}
+
+  @Override
+  public void validateOutputResourceSupported(ResourceId 

[15/19] beam git commit: Move WindowingStrategy from util to values

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
index c317182..fa95477 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
@@ -26,9 +26,9 @@ import javax.annotation.Nullable;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
 import org.apache.flink.api.common.functions.RuntimeContext;
 
 /**

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
index 9f000e0..6517bf2 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
@@ -40,10 +40,10 @@ import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
 import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
 import org.apache.flink.api.common.functions.RichGroupReduceFunction;
 import org.apache.flink.api.common.functions.RuntimeContext;
 import org.apache.flink.configuration.Configuration;

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
index 7ee2f69..942bf42 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
@@ -34,8 +34,8 @@ import 
org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.WindowingStrategy;
 import org.apache.flink.api.java.tuple.Tuple2;
 import org.apache.flink.util.Collector;
 import org.joda.time.Instant;

http://git-wip-us.apache.org/repos/asf/beam/blob/b61e5bb2/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
index 4aacb4a..fb4c678 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
@@ -31,8 +31,8 @@ import 
org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.KV;
+import 

[10/19] beam git commit: Mark TimeDomain experimental alongside Timers; improve javadoc

[kenn]

Mark TimeDomain experimental alongside Timers; improve javadoc


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b4c808a0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b4c808a0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b4c808a0

Branch: refs/heads/master
Commit: b4c808a045d493e93d83c815dc7520c6b213858a
Parents: ca41af8
Author: Kenneth Knowles 
Authored: Tue May 2 12:31:02 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:55 2017 -0700

--
 .../org/apache/beam/sdk/state/TimeDomain.java | 18 ++
 1 file changed, 10 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b4c808a0/sdks/java/core/src/main/java/org/apache/beam/sdk/state/TimeDomain.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/TimeDomain.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/TimeDomain.java
index e814915..0526453 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/TimeDomain.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/TimeDomain.java
@@ -17,27 +17,29 @@
  */
 package org.apache.beam.sdk.state;
 
+import org.apache.beam.sdk.annotations.Experimental;
+
 /**
- * {@code TimeDomain} specifies whether an operation is based on
- * timestamps of elements or current "real-world" time as reported while 
processing.
+ * {@link TimeDomain} specifies whether an operation is based on timestamps of 
elements or current
+ * "real-world" time as reported while processing.
  */
+@Experimental(Experimental.Kind.TIMERS)
 public enum TimeDomain {
   /**
-   * The {@code EVENT_TIME} domain corresponds to the timestamps on the 
elements. Time advances
-   * on the system watermark advances.
+   * The {@link #EVENT_TIME} domain corresponds to the timestamps on the 
elements. Time advances on
+   * the system watermark advances.
*/
   EVENT_TIME,
 
   /**
-   * The {@code PROCESSING_TIME} domain corresponds to the current to the 
current (system) time.
+   * The {@link #PROCESSING_TIME} domain corresponds to the current to the 
current (system) time.
* This is advanced during execution of the pipeline.
*/
   PROCESSING_TIME,
 
   /**
-   * Same as the {@code PROCESSING_TIME} domain, except it won't fire a timer 
set for time
-   * {@code T} until all timers from earlier stages set for a time earlier 
than {@code T} have
-   * fired.
+   * Same as the {@link #PROCESSING_TIME} domain, except it won't fire a timer 
set for time t
+   * until all timers from earlier stages set for a time earlier than t 
have fired.
*/
   SYNCHRONIZED_PROCESSING_TIME
 }

[18/19] beam git commit: Update Dataflow worker to 20170504-2

[kenn]

Update Dataflow worker to 20170504-2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/02e80a7d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/02e80a7d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/02e80a7d

Branch: refs/heads/master
Commit: 02e80a7d5890ac2281d970cb1977399f79431d5b
Parents: c83cc74
Author: Kenneth Knowles <k...@google.com>
Authored: Thu May 4 16:09:26 2017 -0700
Committer: Kenneth Knowles <k...@google.com>
Committed: Thu May 4 16:09:26 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/02e80a7d/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 0367bd8..e21b6de 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170504
+
beam-master-20170504-2
 
1
 
6
   

[GitHub] beam pull request #2891: [BEAM-1340,BEAM-1345] Vital moves of user-facing st...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2891


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[12/19] beam git commit: Move Reshuffle and ReshuffleTrigger out of util

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/e0b3f806/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
index ee3af0b..50be0bb 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
@@ -42,7 +42,6 @@ import com.google.api.services.bigquery.model.TableSchema;
 import com.google.common.collect.HashBasedTable;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
-
 import java.io.BufferedReader;
 import java.io.ByteArrayInputStream;
 import java.io.FileReader;
@@ -54,7 +53,6 @@ import java.nio.charset.StandardCharsets;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ThreadLocalRandom;
-
 import org.apache.avro.Schema;
 import org.apache.avro.file.DataFileWriter;
 import org.apache.avro.generic.GenericDatumWriter;
@@ -69,7 +67,6 @@ import 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
 import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
 import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.JobService;
 import org.apache.beam.sdk.util.FluentBackoff;
-
 import org.apache.beam.sdk.util.MimeTypes;
 import org.apache.beam.sdk.util.Transport;
 import org.joda.time.Duration;

http://git-wip-us.apache.org/repos/asf/beam/blob/e0b3f806/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
index d52723b..8915069 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
@@ -19,7 +19,6 @@ package org.apache.beam.sdk.io.gcp.bigquery;
 
 import com.google.api.services.bigquery.model.Table;
 import com.google.api.services.bigquery.model.TableRow;
-
 import java.util.ArrayList;
 import java.util.List;
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e0b3f806/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ConfigurableEmployeeInputFormat.java
--
diff --git 
a/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ConfigurableEmployeeInputFormat.java
 
b/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ConfigurableEmployeeInputFormat.java
index 40f949b..1d3190e 100644
--- 
a/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ConfigurableEmployeeInputFormat.java
+++ 
b/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ConfigurableEmployeeInputFormat.java
@@ -19,7 +19,6 @@ import java.io.DataOutput;
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
-
 import org.apache.hadoop.conf.Configurable;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.io.Text;

http://git-wip-us.apache.org/repos/asf/beam/blob/e0b3f806/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/EmployeeInputFormat.java
--
diff --git 
a/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/EmployeeInputFormat.java
 
b/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/EmployeeInputFormat.java
index 206f9ab..a641707 100644
--- 
a/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/EmployeeInputFormat.java
+++ 
b/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/EmployeeInputFormat.java
@@ -19,7 +19,6 @@ import java.io.DataOutput;
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
-
 import org.apache.beam.sdk.values.KV;
 import org.apache.hadoop.io.Text;
 import org.apache.hadoop.io.Writable;

http://git-wip-us.apache.org/repos/asf/beam/blob/e0b3f806/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ReuseObjectsEmployeeInputFormat.java

[13/19] beam git commit: Move Reshuffle and ReshuffleTrigger out of util

[kenn]

Move Reshuffle and ReshuffleTrigger out of util

These are two deprecated, but regularly used, transforms and triggers, to the
point where the Runner API proto even includes something akin to the
ReshuffleTrigger.

They are moved to where they "belong" and marked deprecated.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e0b3f806
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e0b3f806
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e0b3f806

Branch: refs/heads/master
Commit: e0b3f8064b97d8678e75bf6ba25244bca31e6a7d
Parents: 6542eaf
Author: Kenneth Knowles 
Authored: Wed May 3 19:57:02 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:55 2017 -0700

--
 .../runners/core/construction/Triggers.java |   2 +-
 .../triggers/ReshuffleTriggerStateMachine.java  |   2 +-
 .../flink/FlinkBatchTransformTranslators.java   |   2 +-
 .../FlinkStreamingTransformTranslators.java |   2 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   2 +-
 .../dataflow/ReshuffleOverrideFactory.java  |   4 +-
 .../translation/GroupCombineFunctions.java  |   3 +-
 .../spark/translation/TransformTranslator.java  |   2 +-
 .../streaming/StreamingTransformTranslator.java |   2 +-
 .../org/apache/beam/sdk/io/FileSystems.java |   1 -
 .../beam/sdk/transforms/ReifyTimestamps.java|  73 +
 .../apache/beam/sdk/transforms/Reshuffle.java   |  92 +++
 .../transforms/windowing/ReshuffleTrigger.java  |  58 
 .../apache/beam/sdk/util/ReifyTimestamps.java   |  76 --
 .../org/apache/beam/sdk/util/Reshuffle.java |  89 --
 .../apache/beam/sdk/util/ReshuffleTrigger.java  |  52 
 .../beam/sdk/transforms/GroupByKeyTest.java |   1 -
 .../sdk/transforms/ReifyTimestampsTest.java | 104 +++
 .../beam/sdk/transforms/ReshuffleTest.java  | 266 ++
 .../windowing/ReshuffleTriggerTest.java |  42 +++
 .../beam/sdk/util/ReifyTimestampsTest.java  | 109 
 .../org/apache/beam/sdk/util/ReshuffleTest.java | 271 ---
 .../beam/sdk/util/ReshuffleTriggerTest.java |  44 ---
 .../extensions/sorter/ExternalSorterTest.java   |   1 -
 .../apache/beam/sdk/io/common/HashingFn.java|   2 -
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java|   2 +-
 .../sdk/io/gcp/bigquery/BigQueryHelpers.java|   2 -
 .../io/gcp/bigquery/BigQueryQuerySource.java|   1 -
 .../sdk/io/gcp/bigquery/CalculateSchemas.java   |   1 -
 .../beam/sdk/io/gcp/bigquery/CreateTables.java  |   1 -
 .../io/gcp/bigquery/StreamingWriteTables.java   |   2 +-
 .../sdk/io/gcp/bigquery/TableDestination.java   |   1 -
 .../sdk/io/gcp/bigquery/WritePartition.java |   1 -
 .../beam/sdk/io/gcp/bigquery/WriteRename.java   |   1 -
 .../beam/sdk/io/gcp/bigquery/WriteTables.java   |   1 -
 .../io/gcp/bigquery/FakeBigQueryServices.java   |   2 -
 .../sdk/io/gcp/bigquery/FakeJobService.java |   3 -
 .../sdk/io/gcp/bigquery/TableContainer.java |   1 -
 .../ConfigurableEmployeeInputFormat.java|   1 -
 .../hadoop/inputformat/EmployeeInputFormat.java |   1 -
 .../ReuseObjectsEmployeeInputFormat.java|   1 -
 .../hadoop/inputformat/TestEmployeeDataSet.java |   2 -
 .../inputformat/HIFIOWithElasticTest.java   |   1 -
 .../HIFIOWithEmbeddedCassandraTest.java |   2 -
 .../integration/tests/HIFIOCassandraIT.java |   2 -
 .../integration/tests/HIFIOElasticIT.java   |   1 -
 .../org/apache/beam/sdk/io/jdbc/JdbcIO.java |   2 -
 .../org/apache/beam/sdk/io/jdbc/JdbcIOIT.java   |   1 -
 .../org/apache/beam/sdk/io/jms/JmsRecord.java   |   1 -
 .../beam/sdk/io/kafka/KafkaCheckpointMark.java  |   2 -
 .../apache/beam/sdk/io/kafka/KafkaRecord.java   |   1 -
 .../CoderBasedKafkaDeserializer.java|   1 -
 .../CoderBasedKafkaSerializer.java  |   1 -
 .../serialization/InstantDeserializer.java  |   1 -
 .../kafka/serialization/InstantSerializer.java  |   1 -
 .../beam/sdk/io/mongodb/MongoDbGridFSIO.java|   3 -
 .../sdk/io/mongodb/MongoDBGridFSIOTest.java |   3 -
 .../beam/sdk/io/mongodb/MongoDbIOTest.java  |   4 -
 .../org/apache/beam/sdk/io/mqtt/MqttIO.java |   3 -
 59 files changed, 648 insertions(+), 708 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e0b3f806/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Triggers.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Triggers.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Triggers.java
index 5e73571..df6c9ed 100644
--- 

[01/19] beam git commit: Add @Internal and @Experimental to state package

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 2f4dd8dfa -> 3042d761a


Add @Internal and @Experimental to state package


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ac01ec7a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ac01ec7a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ac01ec7a

Branch: refs/heads/master
Commit: ac01ec7afb43ae0bc4198234d1de16a830f95b10
Parents: 78e0acc
Author: Kenneth Knowles 
Authored: Tue May 2 10:41:01 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu May 4 16:06:34 2017 -0700

--
 .../org/apache/beam/sdk/state/BagState.java |  4 +
 .../apache/beam/sdk/state/CombiningState.java   |  3 +
 .../apache/beam/sdk/state/GroupingState.java|  3 +
 .../org/apache/beam/sdk/state/MapState.java |  3 +
 .../apache/beam/sdk/state/ReadableStates.java   |  7 +-
 .../org/apache/beam/sdk/state/SetState.java |  4 +
 .../java/org/apache/beam/sdk/state/State.java   |  4 +
 .../org/apache/beam/sdk/state/StateBinder.java  |  7 +-
 .../org/apache/beam/sdk/state/StateContext.java |  5 +-
 .../apache/beam/sdk/state/StateContexts.java|  5 +-
 .../org/apache/beam/sdk/state/StateSpec.java| 18 +++-
 .../org/apache/beam/sdk/state/StateSpecs.java   | 88 
 .../beam/sdk/state/WatermarkHoldState.java  | 11 ++-
 .../org/apache/beam/sdk/state/package-info.java |  2 +-
 14 files changed, 109 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ac01ec7a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
index 189d151..10fba05 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
@@ -17,11 +17,15 @@
  */
 package org.apache.beam.sdk.state;
 
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+
 /**
  * State containing a bag values. Items can be added to the bag and the 
contents read out.
  *
  * @param  The type of elements in the bag.
  */
+@Experimental(Kind.STATE)
 public interface BagState extends GroupingState {
   @Override
   BagState readLater();

http://git-wip-us.apache.org/repos/asf/beam/blob/ac01ec7a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
index 6080127..ddda255 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.state;
 
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 
 /**
@@ -27,6 +29,7 @@ import org.apache.beam.sdk.transforms.Combine.CombineFn;
  * @param  the type of accumulator
  * @param  the type of value extracted from the state
  */
+@Experimental(Kind.STATE)
 public interface CombiningState
 extends GroupingState {
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ac01ec7a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/GroupingState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/GroupingState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/GroupingState.java
index 3a12e79..d99ff25 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/GroupingState.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/GroupingState.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.state;
 
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 
 /**
@@ -26,6 +28,7 @@ import org.apache.beam.sdk.transforms.Combine.CombineFn;
  * @param  the type of values added to the state
  * @param  the type of value extracted from the state
  */
+@Experimental(Kind.STATE)
 public interface GroupingState extends 
ReadableState, State {
   /**
* Add a value to the buffer.

[06/19] beam git commit: Move SideInputReader to runners/core-java

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/6542eafc/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NullSideInputReader.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NullSideInputReader.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NullSideInputReader.java
deleted file mode 100644
index 3da2dc1..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NullSideInputReader.java
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import com.google.common.collect.Sets;
-import java.util.Collections;
-import java.util.Set;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.values.PCollectionView;
-
-/**
- * A {@link SideInputReader} representing a well-defined set of views, but not 
storing
- * any values for them. Used to check if a side input is present when the data 
itself
- * comes from elsewhere.
- */
-public class NullSideInputReader implements SideInputReader {
-
-  private Set views;
-
-  public static NullSideInputReader empty() {
-return new NullSideInputReader(Collections.emptySet());
-  }
-
-  public static NullSideInputReader of(Iterable> 
views) {
-return new NullSideInputReader(views);
-  }
-
-  private NullSideInputReader(Iterable> views) {
-this.views = Sets.newHashSet(views);
-  }
-
-  @Override
-  public  T get(PCollectionView view, BoundedWindow window) {
-throw new IllegalArgumentException("cannot call 
NullSideInputReader.get()");
-  }
-
-  @Override
-  public boolean isEmpty() {
-return views.isEmpty();
-  }
-
-  @Override
-  public  boolean contains(PCollectionView view) {
-return views.contains(view);
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/6542eafc/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReadyCheckingSideInputReader.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReadyCheckingSideInputReader.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReadyCheckingSideInputReader.java
deleted file mode 100644
index cb38a55..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReadyCheckingSideInputReader.java
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.util;
-
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.values.PCollectionView;
-
-/**
- * A {@link SideInputReader} that allows callers to check to see if a {@link 
PCollectionView} has
- * had its contents set in a window.
- */
-public interface ReadyCheckingSideInputReader extends SideInputReader {
-  /**
-   * Returns true if the {@link PCollectionView} is ready in the provided 
{@link BoundedWindow}.
-   */
-  boolean isReady(PCollectionView view, BoundedWindow window);
-}
-

http://git-wip-us.apache.org/repos/asf/beam/blob/6542eafc/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SideInputReader.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SideInputReader.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SideInputReader.java
deleted file mode 

[03/19] beam git commit: Move Java sdk.util.state to sdk.state

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/78e0acca/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml 
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index 28bbc3c..1db0e86 100644
--- a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
@@ -373,7 +373,7 @@
 
   
   
-
+
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/78e0acca/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
new file mode 100644
index 000..189d151
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/BagState.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.state;
+
+/**
+ * State containing a bag values. Items can be added to the bag and the 
contents read out.
+ *
+ * @param  The type of elements in the bag.
+ */
+public interface BagState extends GroupingState {
+  @Override
+  BagState readLater();
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/78e0acca/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
new file mode 100644
index 000..6080127
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/CombiningState.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.state;
+
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+
+/**
+ * State for a single value that is managed by a {@link CombineFn}. This is an 
internal extension
+ * to {@link GroupingState} that includes the {@code AccumT} type.
+ *
+ * @param  the type of values added to the state
+ * @param  the type of accumulator
+ * @param  the type of value extracted from the state
+ */
+public interface CombiningState
+extends GroupingState {
+
+  /**
+   * Read the merged accumulator for this combining value. It is implied that 
reading the
+   * state involes reading the accumulator, so {@link #readLater} is 
sufficient to prefetch for
+   * this.
+   */
+  AccumT getAccum();
+
+  /**
+   * Add an accumulator to this combining value. Depending on implementation 
this may immediately
+   * merge it with the previous accumulator, or may buffer this accumulator 
for a future merge.
+   */
+  void addAccum(AccumT accum);
+
+  /**
+   * Merge the given accumulators according to the underlying combiner.
+   */
+  AccumT mergeAccumulators(Iterable accumulators);
+
+  @Override
+  CombiningState readLater();
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/78e0acca/sdks/java/core/src/main/java/org/apache/beam/sdk/state/GroupingState.java
--
diff --git 

[jira] [Commented] (BEAM-2172) ProcessBundleHandlerTest.testCreatingAndProcessingDoFn fails at HEAD

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997605#comment-15997605
 ] 

ASF GitHub Bot commented on BEAM-2172:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2906

[BEAM-2172] Update expectation in ProcessBundleHandlerTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam process_bundle_expectation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2906.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2906


commit 7da2f61654509648e46cfa5009de6c7b6ed6e461
Author: Thomas Groh 
Date:   2017-05-05T00:01:24Z

Update expectation in ProcessBundleHandlerTest




> ProcessBundleHandlerTest.testCreatingAndProcessingDoFn fails at HEAD
> 
>
> Key: BEAM-2172
> URL: https://issues.apache.org/jira/browse/BEAM-2172
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
>
> java.lang.AssertionError: 
> Expected: iterable containing [ pane=PaneInfo.NO_FIRING}>]
>  but: item 0: was  timestamp=294247-01-09T04:00:54.775Z, pane=PaneInfo.NO_FIRING}>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:956)
>   at org.junit.Assert.assertThat(Assert.java:923)
>   at 
> org.apache.beam.fn.harness.control.ProcessBundleHandlerTest.testCreatingAndProcessingDoFn(ProcessBundleHandlerTest.java:436)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at 

[GitHub] beam pull request #2906: [BEAM-2172] Update expectation in ProcessBundleHand...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2906

[BEAM-2172] Update expectation in ProcessBundleHandlerTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam process_bundle_expectation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2906.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2906


commit 7da2f61654509648e46cfa5009de6c7b6ed6e461
Author: Thomas Groh 
Date:   2017-05-05T00:01:24Z

Update expectation in ProcessBundleHandlerTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PerformanceTests_Dataflow #373

[Apache Jenkins Server]

See 


Changes:

[dhalperi] GCP-IO: manage versions in top level pom

[dhalperi] Remove IOChannelUtil/Factory from BigQueryIO

[klk] Eliminate Pipeline.getOptions

[robertwb] Cleanup old cython files before testing again

[klk] [BEAM-1727] Add align and offset to Timer

[klk] Fix initial watermark of DoFnOperator in Flink runner

[dhalperi] [BEAM-59] Remove IOChannelUtils, IOChannelFactory

[dhalperi] Move thrown.expect to correct place

[tgroh] Note when checkpoints may be committed.

[tgroh] Split StartBundleContext and FinishBundleContext

[kirpichov] [BEAM-2170] PubsubMessageWithAttributesCoder should not NPE on 
messages

[kirpichov] Makes PubsubMessagePayloadOnlyCoder not require whole-stream context

[kirpichov] Moves coder choice into PubsubSource

[davor] [BEAM-2144] Remove transform.reflect and utils from Javadocs

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 2f4dd8dfaf9e94ff2a1f97edc695fbcd75be34cd (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2f4dd8dfaf9e94ff2a1f97edc695fbcd75be34cd
 > git rev-list b73918b55ab06e5a47ef9dc33ae3dbaebaed330a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson227170201862895542.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson5916819713713284246.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson5309455617960359779.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson7566161025314604258.sh
+ python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn