[jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855649#comment-15855649
 ] 

Jingsong Lee commented on BEAM-1393:


[~aljoscha] I found some APIs changed in Flink 1.2.0.
1.ProcessingTimeService replace registerTimer in StreamingRuntimeContext
2.ManualWatermarkContext move to StreamSourceContexts and become private, we 
can't use it to check
3.UnboundedSourceWrapper need use getOperatorState to snapshot instead of 
OutputStream. (change the parallelism)
4. GlobalConfiguration.getDynamicProperties() replace GlobalConfiguration. 
(maybe need deal with NPE)
5.use KeyedStateBackend instead of AbstractStateBackend.
6.need cast timerService to HeapInternalTimerService to advanceWatermark. (we 
override processWatermark)
7.side-input data: I have some question. I can understand split/merge of 
SPLIT_DISTRIBUTE, but not quite understand the split/merge of BROADCAST. Can 
you explain it？ Or give me some example? 

> Update Flink Runner to Flink 1.2.0
> --
>
> Key: BEAM-1393
> URL: https://issues.apache.org/jira/browse/BEAM-1393
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
>
> When we update to 1.2.0 we can use the new internal Timer API that is 
> available to Flink operators: {{InternalTimerService}} and also use broadcast 
> state to store side-input data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855665#comment-15855665
 ] 

Jingsong Lee commented on BEAM-25:
--

[~kenn] Thanks for reply. But I need scan all elements in BagState every time 
if I use BagState to deduplication. Has Beam a plan to support SetState?

> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1405) Spark runner should not close the spark context when set to reuse it

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-1405:
--

Assignee: Ismaël Mejía  (was: Jean-Baptiste Onofré)

> Spark runner should not close the spark context when set to reuse it
> 
>
> Key: BEAM-1405
> URL: https://issues.apache.org/jira/browse/BEAM-1405
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> It seems that we don't take into account the options when we decide to close 
> the spark context, this produces an exception when trying to reuse a provided 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1934: [BEAM-1405} Spark runner should not close the spark...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/1934

[BEAM-1405} Spark runner should not close the spark context when set to 
reuse it

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1405-spark-stop-context

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1934


commit 4b1ba982d0cd734fcfd007bf1507ffe8d791e670
Author: Ismaël Mejía 
Date:   2017-02-07T08:41:04Z

[BEAM-1405] Refactor to remove repeated code from test

commit 3b29d67c6fea02c7ed5bfa7c4a4702994e05204a
Author: Ismaël Mejía 
Date:   2017-02-07T10:35:47Z

[BEAM-1405] Skip stopping context when spark context is provided




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1405) Spark runner should not close the spark context when set to reuse it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855741#comment-15855741
 ] 

ASF GitHub Bot commented on BEAM-1405:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/1934

[BEAM-1405} Spark runner should not close the spark context when set to 
reuse it

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1405-spark-stop-context

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1934


commit 4b1ba982d0cd734fcfd007bf1507ffe8d791e670
Author: Ismaël Mejía 
Date:   2017-02-07T08:41:04Z

[BEAM-1405] Refactor to remove repeated code from test

commit 3b29d67c6fea02c7ed5bfa7c4a4702994e05204a
Author: Ismaël Mejía 
Date:   2017-02-07T10:35:47Z

[BEAM-1405] Skip stopping context when spark context is provided




> Spark runner should not close the spark context when set to reuse it
> 
>
> Key: BEAM-1405
> URL: https://issues.apache.org/jira/browse/BEAM-1405
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> It seems that we don't take into account the options when we decide to close 
> the spark context, this produces an exception when trying to reuse a provided 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Dataflow #2220

[Apache Jenkins Server]

See 


--
[...truncated 26 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 59570
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Dataflow] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 59570
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/google-cloud-dataflow-java -DforkCount=0 
-DrunnableOnServicePipelineOptions=[ 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner", 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-runnable-on-service-tests/" ]
2017-02-07T12:00:56.006 [INFO] Error stacktraces are turned on.
2017-02-07T12:00:56.101 [INFO] Scanning for projects...
2017-02-07T12:00:58.387 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T12:00:59.051 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 9.0 KB/sec)
2017-02-07T12:00:59.081 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T12:00:59.134 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 100.2 KB/sec)
2017-02-07T12:00:59.169 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T12:00:59.232 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 52.4 KB/sec)
2017-02-07T12:00:59.238 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T12:00:59.313 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 294.1 KB/sec)
2017-02-07T12:00:59.324 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T12:00:59.382 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 548.7 KB/sec)
2017-02-07T12:00:59.396 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T12:00:59.452 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 243.7 KB/sec)
2017-02-07T12:00:59.483 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T12:00:59.517 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 115.5 KB/sec)
2017-02-07T12:00:59.529 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T12:00:59.567 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 87.2 KB/sec)
2017-02-07T12:00:59.574 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T12:00:59.626 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 376.7 KB/sec)
2017-02-07T12:00:59.632 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T12:00:59.676 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 153.5 KB/sec)
2017-02-07T12:00:59.680 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T12:00:59.738 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
 (14 KB at 228.3 KB/sec)
2017-02-07T12:00:59.748 [INFO] Download

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #871

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1411) Unable to downgrade to lower guava version after upgrade to beam-0.5

[Michael Luckey (JIRA)]

Michael Luckey created BEAM-1411:


 Summary: Unable to downgrade to lower guava version after upgrade 
to beam-0.5
 Key: BEAM-1411
 URL: https://issues.apache.org/jira/browse/BEAM-1411
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.5.0
Reporter: Michael Luckey
Assignee: Davor Bonaci


While testing upgrading to 0.5 version of beam we ran into NoSuchMethodErrors

{noformat}
java.lang.NoSuchMethodError: 
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
 at 
org.apache.beam.sdk.io.hdfs.HDFSFileSource$SerializableSplit.(HDFSFileSource.java:473)
 at 
org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:81)
 at 
org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:78)
 at 
com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451)
 at java.util.AbstractList$Itr.next(AbstractList.java:358)
 at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:202)
 at 
org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:65)
 at 
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:168)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:329)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:71)
 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:178)
 at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:258)
{noformat}

This seems to be caused by the [HDFS 
IO|https://github.com/apache/beam/tree/master/sdks/java/io/hdfs] components 
guava dependency not being shaded - in contrast to the core components - and 
revealed by the [recent update to 
guava-20.0|https://github.com/apache/beam/commit/0b4b2becb45b9f637ba31f599ebe8be0331bd633]
 and the therefore incorporated api changes made by [overloading 
methods|https://github.com/google/guava/commit/892e323fca32945cdfb25395ca6e346dd0fffa5b#diff-fe7358934fa6eba23c2791eb40cec030]

Did not check, whether more components are affected to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1935: [BEAM-79] Fix Gearpump runner for WindowingTest

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/1935

[BEAM-79] Fix Gearpump runner for WindowingTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam 
gearpump-runner-window

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1935


commit 82994256798fe85c4e2fbbab8262836c51ae1ab8
Author: manuzhang 
Date:   2017-02-07T14:14:18Z

[BEAM-79] Fix Gearpump runner for WindowingTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856059#comment-15856059
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/1935

[BEAM-79] Fix Gearpump runner for WindowingTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam 
gearpump-runner-window

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1935


commit 82994256798fe85c4e2fbbab8262836c51ae1ab8
Author: manuzhang 
Date:   2017-02-07T14:14:18Z

[BEAM-79] Fix Gearpump runner for WindowingTest




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1405) Spark runner should not close the SparkContext when the context is provided

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-1405:
---
Summary: Spark runner should not close the SparkContext when the context is 
provided  (was: Spark runner should not close the spark context when set to 
reuse it)

> Spark runner should not close the SparkContext when the context is provided
> ---
>
> Key: BEAM-1405
> URL: https://issues.apache.org/jira/browse/BEAM-1405
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> It seems that we don't take into account the options when we decide to close 
> the spark context, this produces an exception when trying to reuse a provided 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: This closes #1934

[amitsela]

This closes #1934


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ed7b82e7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ed7b82e7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ed7b82e7

Branch: refs/heads/master
Commit: ed7b82e7ecee8de001e181b355ca8f1a284e5c6b
Parents: 9dad73c 5edcdff
Author: Sela 
Authored: Tue Feb 7 16:43:11 2017 +0200
Committer: Sela 
Committed: Tue Feb 7 16:43:11 2017 +0200

--
 .../spark/translation/SparkContextFactory.java  | 11 ++-
 .../runners/spark/ProvidedSparkContextTest.java | 72 +---
 2 files changed, 39 insertions(+), 44 deletions(-)
--

[2/3] beam git commit: [BEAM-1405] Skip stopping context when spark context is provided

[amitsela]

[BEAM-1405] Skip stopping context when spark context is provided


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5edcdff9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5edcdff9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5edcdff9

Branch: refs/heads/master
Commit: 5edcdff9ac6aee552cb95fe46f32a9f57dd813e1
Parents: 92707b9
Author: Ismaël Mejía 
Authored: Tue Feb 7 14:43:11 2017 +0100
Committer: Ismaël Mejía 
Committed: Tue Feb 7 14:47:04 2017 +0100

--
 .../runners/spark/translation/SparkContextFactory.java   | 11 ---
 .../beam/runners/spark/ProvidedSparkContextTest.java |  2 ++
 2 files changed, 10 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5edcdff9/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
index 67839a8..326838a 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
@@ -40,17 +40,21 @@ public final class SparkContextFactory {
*/
   static final String TEST_REUSE_SPARK_CONTEXT = 
"beam.spark.test.reuseSparkContext";
 
+  // Spark allows only one context for JVM so this can be static.
   private static JavaSparkContext sparkContext;
   private static String sparkMaster;
+  private static boolean usesProvidedSparkContext;
 
   private SparkContextFactory() {
   }
 
   public static synchronized JavaSparkContext 
getSparkContext(SparkPipelineOptions options) {
 SparkContextOptions contextOptions = options.as(SparkContextOptions.class);
+usesProvidedSparkContext = contextOptions.getUsesProvidedSparkContext();
 // reuse should be ignored if the context is provided.
 if (Boolean.getBoolean(TEST_REUSE_SPARK_CONTEXT)
-&& !contextOptions.getUsesProvidedSparkContext()) {
+&& !usesProvidedSparkContext) {
+
   // if the context is null or stopped for some reason, re-create it.
   if (sparkContext == null || sparkContext.sc().isStopped()) {
 sparkContext = createSparkContext(contextOptions);
@@ -67,13 +71,14 @@ public final class SparkContextFactory {
   }
 
   public static synchronized void stopSparkContext(JavaSparkContext context) {
-if (!Boolean.getBoolean(TEST_REUSE_SPARK_CONTEXT)) {
+if (!Boolean.getBoolean(TEST_REUSE_SPARK_CONTEXT)
+&& !usesProvidedSparkContext) {
   context.stop();
 }
   }
 
   private static JavaSparkContext createSparkContext(SparkContextOptions 
contextOptions) {
-if (contextOptions.getUsesProvidedSparkContext()) {
+if (usesProvidedSparkContext) {
   LOG.info("Using a provided Spark Context");
   JavaSparkContext jsc = contextOptions.getProvidedSparkContext();
   if (jsc == null || jsc.sc().isStopped()){

http://git-wip-us.apache.org/repos/asf/beam/blob/5edcdff9/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
index 00c894d..a4190a9 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
@@ -57,6 +57,8 @@ public class ProvidedSparkContextTest {
 public void testWithProvidedContext() throws Exception {
 JavaSparkContext jsc = new JavaSparkContext("local[*]", 
"Existing_Context");
 testWithValidProvidedContext(jsc);
+// A provided context must not be stopped after execution
+assertFalse(jsc.sc().isStopped());
 jsc.stop();
 }
 

[jira] [Commented] (BEAM-1405) Spark runner should not close the SparkContext when the context is provided

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856160#comment-15856160
 ] 

ASF GitHub Bot commented on BEAM-1405:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1934


> Spark runner should not close the SparkContext when the context is provided
> ---
>
> Key: BEAM-1405
> URL: https://issues.apache.org/jira/browse/BEAM-1405
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> It seems that we don't take into account the options when we decide to close 
> the spark context, this produces an exception when trying to reuse a provided 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1405) Spark runner should not close the SparkContext when the context is provided

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1405.
-
   Resolution: Fixed
Fix Version/s: 0.6.0

> Spark runner should not close the SparkContext when the context is provided
> ---
>
> Key: BEAM-1405
> URL: https://issues.apache.org/jira/browse/BEAM-1405
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
> Fix For: 0.6.0
>
>
> It seems that we don't take into account the options when we decide to close 
> the spark context, this produces an exception when trying to reuse a provided 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/3] beam git commit: [BEAM-1405] Refactor to remove repeated code from test

[amitsela]

Repository: beam
Updated Branches:
  refs/heads/master 9dad73c29 -> ed7b82e7e


[BEAM-1405] Refactor to remove repeated code from test


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/92707b9a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/92707b9a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/92707b9a

Branch: refs/heads/master
Commit: 92707b9a07b7bc367b375fb25293554f1de25d87
Parents: 9dad73c
Author: Ismaël Mejía 
Authored: Tue Feb 7 14:41:57 2017 +0100
Committer: Ismaël Mejía 
Committed: Tue Feb 7 14:46:40 2017 +0100

--
 .../runners/spark/ProvidedSparkContextTest.java | 70 
 1 file changed, 29 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/92707b9a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
index 2982844..00c894d 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
@@ -18,6 +18,7 @@
 
 package org.apache.beam.runners.spark;
 
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.fail;
 
 import com.google.common.collect.ImmutableSet;
@@ -48,15 +49,6 @@ public class ProvidedSparkContextTest {
 private static final String PROVIDED_CONTEXT_EXCEPTION =
 "The provided Spark context was not created or was stopped";
 
-private SparkContextOptions getSparkContextOptions(JavaSparkContext jsc) {
-final SparkContextOptions options = 
PipelineOptionsFactory.as(SparkContextOptions.class);
-options.setRunner(SparkRunner.class);
-options.setUsesProvidedSparkContext(true);
-options.setProvidedSparkContext(jsc);
-options.setEnableSparkMetricSinks(false);
-return options;
-}
-
 /**
  * Provide a context and call pipeline run.
  * @throws Exception
@@ -64,20 +56,7 @@ public class ProvidedSparkContextTest {
 @Test
 public void testWithProvidedContext() throws Exception {
 JavaSparkContext jsc = new JavaSparkContext("local[*]", 
"Existing_Context");
-
-SparkContextOptions options = getSparkContextOptions(jsc);
-
-Pipeline p = Pipeline.create(options);
-PCollection inputWords = 
p.apply(Create.of(WORDS).withCoder(StringUtf8Coder
-.of()));
-PCollection output = inputWords.apply(new 
WordCount.CountWords())
-.apply(MapElements.via(new WordCount.FormatAsTextFn()));
-
-PAssert.that(output).containsInAnyOrder(EXPECTED_COUNT_SET);
-
-// Run test from pipeline
-p.run().waitUntilFinish();
-
+testWithValidProvidedContext(jsc);
 jsc.stop();
 }
 
@@ -87,8 +66,22 @@ public class ProvidedSparkContextTest {
  */
 @Test
 public void testWithNullContext() throws Exception {
-JavaSparkContext jsc = null;
+testWithInvalidContext(null);
+}
+
+/**
+ * A SparkRunner with a stopped provided Spark context cannot run 
pipelines.
+ * @throws Exception
+ */
+@Test
+public void testWithStoppedProvidedContext() throws Exception {
+JavaSparkContext jsc = new JavaSparkContext("local[*]", 
"Existing_Context");
+// Stop the provided Spark context directly
+jsc.stop();
+testWithInvalidContext(jsc);
+}
 
+private void testWithValidProvidedContext(JavaSparkContext jsc) throws 
Exception {
 SparkContextOptions options = getSparkContextOptions(jsc);
 
 Pipeline p = Pipeline.create(options);
@@ -99,24 +92,11 @@ public class ProvidedSparkContextTest {
 
 PAssert.that(output).containsInAnyOrder(EXPECTED_COUNT_SET);
 
-try {
-p.run().waitUntilFinish();
-fail("Should throw an exception when The provided Spark context is 
null");
-} catch (RuntimeException e){
-assert(e.getMessage().contains(PROVIDED_CONTEXT_EXCEPTION));
-}
+// Run test from pipeline
+p.run().waitUntilFinish();
 }
 
-/**
- * A SparkRunner with a stopped provided Spark context cannot run 
pipelines.
- * @throws Exception
- */
-@Test
-public void testWithStoppedProvidedContext() throws Exception {
-JavaSparkContext jsc = new JavaSparkContext("local[*]", 
"Existing_Context");
-// Stop the provided Spark context directly
-jsc.stop();
-
+private void testWithInvalidContext(JavaSparkContext jsc) {
 S

[GitHub] beam pull request #1934: [BEAM-1405] Spark runner should not close the Spark...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1934


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2559

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-1405] Refactor to remove repeated code from test

[iemejia] [BEAM-1405] Skip stopping context when spark context is provided

--
[...truncated 7132 lines...]
2017-02-07T15:23:13.551 [INFO] 2017-02-07\T\15:23:13.550 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/apis/google-api-services-storage/v1-rev35-1.20.0/google-api-services-storage-v1-rev35-1.20.0.pom
 (3 KB at 80.2 KB/sec)
2017-02-07T15:23:13.556 [INFO] 2017-02-07\T\15:23:13.556 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/2.0.3/jsr305-2.0.3.pom
2017-02-07T15:23:13.585 [INFO] 2017-02-07\T\15:23:13.585 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/2.0.3/jsr305-2.0.3.pom
 (2 KB at 48.3 KB/sec)
2017-02-07T15:23:13.613 [INFO] 2017-02-07\T\15:23:13.613 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.pom
2017-02-07T15:23:13.643 [INFO] 2017-02-07\T\15:23:13.643 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.pom
 (2 KB at 45.8 KB/sec)
2017-02-07T15:23:13.830 [INFO] 2017-02-07\T\15:23:13.829 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.20.0/google-api-client-java6-1.20.0.jar
2017-02-07T15:23:13.830 [INFO] 2017-02-07\T\15:23:13.830 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.20.0/google-api-client-jackson2-1.20.0.jar
2017-02-07T15:23:13.831 [INFO] 2017-02-07\T\15:23:13.831 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.20.0/google-oauth-client-java6-1.20.0.jar
2017-02-07T15:23:13.865 [INFO] 2017-02-07\T\15:23:13.865 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.20.0/google-api-client-jackson2-1.20.0.jar
 (3 KB at 73.2 KB/sec)
2017-02-07T15:23:13.902 [INFO] 2017-02-07\T\15:23:13.902 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.20.0/google-oauth-client-java6-1.20.0.jar
 (11 KB at 148.0 KB/sec)
2017-02-07T15:23:13.909 [INFO] 2017-02-07\T\15:23:13.908 [INFO] 

2017-02-07T15:23:13.909 [INFO] 2017-02-07\T\15:23:13.909 [INFO] BUILD FAILURE
2017-02-07T15:23:13.909 [INFO] 2017-02-07\T\15:23:13.909 [INFO] 

2017-02-07T15:23:13.910 [INFO] 2017-02-07\T\15:23:13.910 [INFO] Total time: 
3.061 s
2017-02-07T15:23:13.911 [INFO] 2017-02-07\T\15:23:13.911 [INFO] Finished at: 
2017-02-07T15:23:13+00:00
2017-02-07T15:23:14.029 [INFO] 2017-02-07\T\15:23:14.029 [INFO] Final Memory: 
11M/193M
2017-02-07T15:23:14.036 [INFO] 2017-02-07\T\15:23:14.029 [INFO] 

2017-02-07T15:23:14.037 [INFO] 2017-02-07\T\15:23:14.037 [ERROR] Failed to 
execute goal on project basic: Could not resolve dependencies for project 
archetype.it:basic:jar:0.1: Could not transfer artifact 
com.google.api-client:google-api-client-java6:jar:1.20.0 from/to central 
(https://repo.maven.apache.org/maven2): handshake alert:  unrecognized_name -> 
[Help 1]
2017-02-07T15:23:14.037 [INFO] 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
on project basic: Could not resolve dependencies for project 
archetype.it:basic:jar:0.1: Could not transfer artifact 
com.google.api-client:google-api-client-java6:jar:1.20.0 from/to central 
(https://repo.maven.apache.org/maven2): handshake alert:  unrecognized_name
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
2017-02-07T15:23:14.038 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(Lifec

[2/2] beam git commit: This closes #1855: Add wrapping of lambda in a SimpleFunction

[kenn]

This closes #1855: Add wrapping of lambda in a SimpleFunction


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/144bffd4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/144bffd4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/144bffd4

Branch: refs/heads/master
Commit: 144bffd4024decbc586bf16efe05caeda9427d77
Parents: ed7b82e 2315217
Author: Kenneth Knowles 
Authored: Tue Feb 7 08:18:16 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 7 08:18:16 2017 -0800

--
 .../beam/sdk/transforms/SimpleFunction.java | 44 +++--
 .../beam/sdk/transforms/SimpleFunctionTest.java | 43 
 .../sdk/transforms/MapElementsJava8Test.java| 24 ++-
 .../sdk/transforms/SimpleFunctionJava8Test.java | 69 
 4 files changed, 170 insertions(+), 10 deletions(-)
--

[GitHub] beam pull request #1855: [BEAM-1339] Add wrapping of lambda in a SimpleFunct...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1855


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1339) There's an easy trick for getting a type descriptor attached to a lambda that seems nicer than what we have

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856292#comment-15856292
 ] 

ASF GitHub Bot commented on BEAM-1339:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1855


> There's an easy trick for getting a type descriptor attached to a lambda that 
> seems nicer than what we have
> ---
>
> Key: BEAM-1339
> URL: https://issues.apache.org/jira/browse/BEAM-1339
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Trivial
>  Labels: starter
>
> Demonstrated in https://github.com/apache/beam/pull/1855



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #428

[Apache Jenkins Server]

See 


Changes:

[klk] Add wrapping of lambda in a SimpleFunction

--
Started by GitHub push by asfgit
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 144bffd4024decbc586bf16efe05caeda9427d77 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 144bffd4024decbc586bf16efe05caeda9427d77
 > git rev-list ed7b82e7ecee8de001e181b355ca8f1a284e5c6b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Established TCP socket on 35688
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Apex] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 35688
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 clean verify --projects runners/apex --also-make --batch-mode --errors 
--activate-profiles runnable-on-service-tests --activate-profiles 
local-runnable-on-service-tests
2017-02-07T16:33:07.036 [INFO] Error stacktraces are turned on.
2017-02-07T16:33:07.108 [INFO] Scanning for projects...
2017-02-07T16:33:08.445 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T16:33:08.825 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 15.7 KB/sec)
2017-02-07T16:33:08.839 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T16:33:08.888 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 130.9 KB/sec)
2017-02-07T16:33:08.913 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T16:33:08.967 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 61.2 KB/sec)
2017-02-07T16:33:08.972 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T16:33:09.033 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 361.6 KB/sec)
2017-02-07T16:33:09.045 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T16:33:09.094 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 649.4 KB/sec)
2017-02-07T16:33:09.103 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T16:33:09.144 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 332.8 KB/sec)
2017-02-07T16:33:09.178 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T16:33:09.216 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/o

[jira] [Commented] (BEAM-1411) Unable to downgrade to lower guava version after upgrade to beam-0.5

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856291#comment-15856291
 ] 

Daniel Halperin commented on BEAM-1411:
---

Hi [~michel],

Thanks for the report, and good sleuthing!

Early on we were very good about shading, but newer modules maybe not as much. 
We need to get this cleaned up before the next release.

Would you mind sending a PR to implement the shading? Can model it after 
https://github.com/apache/beam/pull/1906

We could presumably also use a new unit test to catch these types of issues in 
the future.

Thanks,
Dan

> Unable to downgrade to lower guava version after upgrade to beam-0.5
> 
>
> Key: BEAM-1411
> URL: https://issues.apache.org/jira/browse/BEAM-1411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Michael Luckey
>Assignee: Davor Bonaci
>
> While testing upgrading to 0.5 version of beam we ran into NoSuchMethodErrors
> {noformat}
> java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>  at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$SerializableSplit.(HDFSFileSource.java:473)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:81)
>  at 
> org.apache.beam.sdk.io.hdfs.AvroHDFSFileSource$1.apply(AvroHDFSFileSource.java:78)
>  at 
> com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451)
>  at java.util.AbstractList$Itr.next(AbstractList.java:358)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:202)
>  at 
> org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:65)
>  at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:168)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:329)
>  at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:71)
>  at org.apache.beam.sdk.Pipeline.run(Pipeline.java:178)
>  at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:258)
> {noformat}
> This seems to be caused by the [HDFS 
> IO|https://github.com/apache/beam/tree/master/sdks/java/io/hdfs] components 
> guava dependency not being shaded - in contrast to the core components - and 
> revealed by the [recent update to 
> guava-20.0|https://github.com/apache/beam/commit/0b4b2becb45b9f637ba31f599ebe8be0331bd633]
>  and the therefore incorporated api changes made by [overloading 
> methods|https://github.com/google/guava/commit/892e323fca32945cdfb25395ca6e346dd0fffa5b#diff-fe7358934fa6eba23c2791eb40cec030]
> Did not check, whether more components are affected to.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2221

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2560

[Apache Jenkins Server]

See 


Changes:

[klk] Add wrapping of lambda in a SimpleFunction

--
Started by GitHub push by asfgit
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 144bffd4024decbc586bf16efe05caeda9427d77 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 144bffd4024decbc586bf16efe05caeda9427d77
 > git rev-list ed7b82e7ecee8de001e181b355ca8f1a284e5c6b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 54511
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_MavenInstall] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 54511
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 
 
-Dmaven.repo.local=
 -B -e -P release,dataflow-runner clean install coveralls:report 
-DrepoToken= -DskipITs=false -DintegrationTestPipelineOptions=[ 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]
2017-02-07T16:33:37.794 [INFO] Error stacktraces are turned on.
2017-02-07T16:33:37.885 [INFO] Scanning for projects...
2017-02-07T16:33:39.237 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T16:33:39.603 [ERROR] [ERROR] Some problems were encountered while 
processing the POMs:
[ERROR] Unresolveable build extension: Plugin 
kr.motd.maven:os-maven-plugin:1.4.0.Final or one of its dependencies could not 
be resolved: Failed to read artifact descriptor for 
kr.motd.maven:os-maven-plugin:jar:1.4.0.Final @ 
 @ 
2017-02-07T16:33:39.607 [ERROR] The build could not read 1 project -> [Help 1]
org.apache.maven.project.ProjectBuildingException: Some problems were 
encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin 
kr.motd.maven:os-maven-plugin:1.4.0.Final or one of its dependencies could not 
be resolved: Failed to read artifact descriptor for 
kr.motd.maven:os-maven-plugin:jar:1.4.0.Final @ 

at 
org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:422)
at 
org.apache.maven.graph.DefaultGraphBuilder.collectProjects(DefaultGraphBuilder.java:419)
at 
org.apache.maven.graph.DefaultGraphBuilder.getProjectsForMavenReactor(DefaultGraphBuilder.java:410)
at 
org.apache.maven.graph.DefaultGraphBuilder.build(DefaultGraphBuilder.java:83)
at org.apache.maven.DefaultMaven.buildGraph(DefaultMaven.java:491)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:219)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven32Launcher.main(Maven32Launcher.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
a

[jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856294#comment-15856294
 ] 

Aljoscha Krettek commented on BEAM-1393:


Yes, there are definitely some changes that need to be done while moving to 
Flink 1.2.

1. This should be covered by BEAM-1394.
2. I think we can remove the checks and use {{SourceFunction.SourceContext}} 
instead.
3. Definitely true, this will get us rescalability.
4. I think it should be {{GlobalConfiguration.loadConfiguration()}}, AFAIK the 
dynamic properties is only used in conjunction with Mesos code so far
5. true
6. I think we have to get a bit creative here and only call super when we know 
that the watermark should advance. Casting doesn't seem wise since we won't 
always be guaranteed that the timer service is in fact a 
{{HeapInternalTimerService}}.
7. What BROADCAST does is collect all checkpointed states into one "list" and 
then send out that list to all parallel subtasks when restoring. The way we 
would use it is to only checkpoint anything from the operator with subtask 
index 0 because we assume that the state is the same on all parallel instances 
of the operator.

That last part is still a bit vague, I think we can tackle that once the other 
parts are figured out.

> Update Flink Runner to Flink 1.2.0
> --
>
> Key: BEAM-1393
> URL: https://issues.apache.org/jira/browse/BEAM-1393
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
>
> When we update to 1.2.0 we can use the new internal Timer API that is 
> available to Flink operators: {{InternalTimerService}} and also use broadcast 
> state to store side-input data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1936: [BEAM-1332] Cleanup files after the tests end

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1936

[BEAM-1332] Cleanup files after the tests end

Snippet tests don't clean up the temporary files that are created these 
changes should address that.

R: @aaltay PTAL 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam 
BEAM-1332-clean_files_after_snippet_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1936


commit fad52f97c7455426145de6a6ba29f021d42dca2a
Author: Sourabh Bajaj 
Date:   2017-01-27T23:46:30Z

Cleanup files after the tests end




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1332) Temporary files in snippet tests are not cleaned up

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856297#comment-15856297
 ] 

ASF GitHub Bot commented on BEAM-1332:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1936

[BEAM-1332] Cleanup files after the tests end

Snippet tests don't clean up the temporary files that are created these 
changes should address that.

R: @aaltay PTAL 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam 
BEAM-1332-clean_files_after_snippet_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1936


commit fad52f97c7455426145de6a6ba29f021d42dca2a
Author: Sourabh Bajaj 
Date:   2017-01-27T23:46:30Z

Cleanup files after the tests end




> Temporary files in snippet tests are not cleaned up 
> 
>
> Key: BEAM-1332
> URL: https://issues.apache.org/jira/browse/BEAM-1332
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> Files created in 
> https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/examples/snippets/snippets_test.py#L426
>  are not deleted at teardown. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Add wrapping of lambda in a SimpleFunction

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master ed7b82e7e -> 144bffd40


Add wrapping of lambda in a SimpleFunction


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/23152178
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/23152178
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/23152178

Branch: refs/heads/master
Commit: 23152178f81e635db65a7aae71f47fa67b3dc065
Parents: ed7b82e
Author: Kenneth Knowles 
Authored: Thu Jan 26 11:19:42 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 7 08:16:39 2017 -0800

--
 .../beam/sdk/transforms/SimpleFunction.java | 44 +++--
 .../beam/sdk/transforms/SimpleFunctionTest.java | 43 
 .../sdk/transforms/MapElementsJava8Test.java| 24 ++-
 .../sdk/transforms/SimpleFunctionJava8Test.java | 69 
 4 files changed, 170 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/23152178/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/SimpleFunction.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/SimpleFunction.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/SimpleFunction.java
index 8604659..db44380 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/SimpleFunction.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/SimpleFunction.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.transforms;
 
+import java.lang.reflect.Method;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
 import org.apache.beam.sdk.values.TypeDescriptor;
@@ -29,6 +31,40 @@ import org.apache.beam.sdk.values.TypeDescriptor;
 public abstract class SimpleFunction
 implements SerializableFunction, HasDisplayData {
 
+  @Nullable
+  private final SerializableFunction fn;
+
+  protected SimpleFunction() {
+this.fn = null;
+// A subclass must override apply if using this constructor. Check that via
+// reflection.
+try {
+  Method methodThatMustBeOverridden =
+  SimpleFunction.class.getDeclaredMethod("apply", new Class[] 
{Object.class});
+  Method methodOnSubclass =
+  getClass().getMethod("apply", new Class[] {Object.class});
+
+  if (methodOnSubclass.equals(methodThatMustBeOverridden)) {
+throw new IllegalStateException(
+"Subclass of SimpleFunction must override 'apply' method"
++ " or pass a SerializableFunction to the constructor,"
++ " usually via a lambda or method reference.");
+  }
+
+} catch (NoSuchMethodException exc) {
+  throw new RuntimeException("Impossible state: missing 'apply' method 
entirely", exc);
+}
+  }
+
+  protected SimpleFunction(SerializableFunction fn) {
+this.fn = fn;
+  }
+
+  @Override
+  public OutputT apply(InputT input) {
+return fn.apply(input);
+  }
+
   public static 
   SimpleFunction fromSerializableFunctionWithOutputType(
   SerializableFunction fn, TypeDescriptor 
outputType) {
@@ -77,23 +113,17 @@ public abstract class SimpleFunction
   private static class SimpleFunctionWithOutputType
   extends SimpleFunction {
 
-private final SerializableFunction fn;
 private final TypeDescriptor outputType;
 
 public SimpleFunctionWithOutputType(
 SerializableFunction fn,
 TypeDescriptor outputType) {
-  this.fn = fn;
+  super(fn);
   this.outputType = outputType;
 }
 
 
 @Override
-public OutputT apply(InputT input) {
-  return fn.apply(input);
-}
-
-@Override
 public TypeDescriptor getOutputTypeDescriptor() {
   return outputType;
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/23152178/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SimpleFunctionTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SimpleFunctionTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SimpleFunctionTest.java
new file mode 100644
index 000..bcfb558
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SimpleFunctionTest.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You ma

[2/2] beam git commit: Check that Elements, Timers have permitted Timestamps

[tgroh]

Check that Elements, Timers have permitted Timestamps

Add a checkArgument to ImmutableListBundleFactory that elements must
be timestamped earlier than the maximum representable timestamp.

Add a checkArgument to TimerData that Timers must fire before the
maximum representable timestamp.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8c3f7773
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8c3f7773
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8c3f7773

Branch: refs/heads/master
Commit: 8c3f7773dffb12558c96fa72317bdcb18d576970
Parents: 144bffd
Author: Thomas Groh 
Authored: Mon Feb 6 15:49:05 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 7 08:59:06 2017 -0800

--
 .../direct/ImmutableListBundleFactory.java  |  6 +
 .../beam/runners/direct/WatermarkManager.java   |  6 +
 .../direct/ImmutableListBundleFactoryTest.java  | 23 +
 .../runners/direct/WatermarkManagerTest.java| 27 
 4 files changed, 62 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8c3f7773/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
index 6b342d6..36264ee 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
+import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkState;
 
 import com.google.auto.value.AutoValue;
@@ -94,6 +95,11 @@ class ImmutableListBundleFactory implements BundleFactory {
   "Can't add element %s to committed bundle in PCollection %s",
   element,
   pcollection);
+  checkArgument(
+  element.getTimestamp().isBefore(BoundedWindow.TIMESTAMP_MAX_VALUE),
+  "Can't add an element past the end of time (%s), got timestamp %s",
+  BoundedWindow.TIMESTAMP_MAX_VALUE,
+  element.getTimestamp());
   elements.add(element);
   if (element.getTimestamp().isBefore(minSoFar)) {
 minSoFar = element.getTimestamp();

http://git-wip-us.apache.org/repos/asf/beam/blob/8c3f7773/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
index 2526d2d..942e67c 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
+import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.auto.value.AutoValue;
@@ -1361,6 +1362,11 @@ public class WatermarkManager {
* it has previously been deleted. Returns this {@link 
TimerUpdateBuilder}.
*/
   public TimerUpdateBuilder setTimer(TimerData setTimer) {
+checkArgument(
+
setTimer.getTimestamp().isBefore(BoundedWindow.TIMESTAMP_MAX_VALUE),
+"Got a timer for after the end of time (%s), got %s",
+BoundedWindow.TIMESTAMP_MAX_VALUE,
+setTimer.getTimestamp());
 deletedTimers.remove(setTimer);
 setTimers.add(setTimer);
 return this;

http://git-wip-us.apache.org/repos/asf/beam/blob/8c3f7773/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
index 3327ccd..c5ad0cd 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
@@ -44,6 +44,7 @@ import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.hamcrest.Matcher;
 import org

[1/2] beam git commit: This closes #1931

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 144bffd40 -> 459a8f836


This closes #1931


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/459a8f83
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/459a8f83
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/459a8f83

Branch: refs/heads/master
Commit: 459a8f836bcd9c94f6c91e6327df67238ae8dd08
Parents: 144bffd 8c3f777
Author: Thomas Groh 
Authored: Tue Feb 7 08:59:06 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 7 08:59:06 2017 -0800

--
 .../direct/ImmutableListBundleFactory.java  |  6 +
 .../beam/runners/direct/WatermarkManager.java   |  6 +
 .../direct/ImmutableListBundleFactoryTest.java  | 23 +
 .../runners/direct/WatermarkManagerTest.java| 27 
 4 files changed, 62 insertions(+)
--

[GitHub] beam pull request #1931: [BEAM-1409] Check that Elements, Timers have permit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1931


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #429

[Apache Jenkins Server]

See 


Changes:

[tgroh] Check that Elements, Timers have permitted Timestamps

--
Started by GitHub push by asfgit
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/beam.git +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 459a8f836bcd9c94f6c91e6327df67238ae8dd08 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 459a8f836bcd9c94f6c91e6327df67238ae8dd08
 > git rev-list 144bffd4024decbc586bf16efe05caeda9427d77 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Established TCP socket on 37932
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Apex] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 37932
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 clean verify --projects runners/apex --also-make --batch-mode --errors 
--activate-profiles runnable-on-service-tests --activate-profiles 
local-runnable-on-service-tests
2017-02-07T17:00:04.667 [INFO] Error stacktraces are turned on.
2017-02-07T17:00:04.747 [INFO] Scanning for projects...
2017-02-07T17:00:06.230 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T17:00:06.827 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 10.0 KB/sec)
2017-02-07T17:00:06.843 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T17:00:06.894 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 125.8 KB/sec)
2017-02-07T17:00:06.909 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T17:00:06.961 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 63.5 KB/sec)
2017-02-07T17:00:06.964 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T17:00:07.052 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 250.7 KB/sec)
2017-02-07T17:00:07.087 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T17:00:07.150 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 505.1 KB/sec)
2017-02-07T17:00:07.167 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T17:00:07.217 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 272.9 KB/sec)
2017-02-07T17:00:07.235 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T17:00:07.277 [INFO] Downloaded: 
https://repo.maven.apache

[jira] [Commented] (BEAM-1409) The DirectRunner should forbid elements outside of representable time.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856332#comment-15856332
 ] 

ASF GitHub Bot commented on BEAM-1409:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1931


> The DirectRunner should forbid elements outside of representable time.
> --
>
> Key: BEAM-1409
> URL: https://issues.apache.org/jira/browse/BEAM-1409
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Davor Bonaci
>
> This ensures that even if the timestamps are representable in Java, they are 
> also representable within the model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1409) The DirectRunner should forbid elements outside of representable time.

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1409.
---
   Resolution: Fixed
 Assignee: Thomas Groh  (was: Davor Bonaci)
Fix Version/s: 0.6.0

> The DirectRunner should forbid elements outside of representable time.
> --
>
> Key: BEAM-1409
> URL: https://issues.apache.org/jira/browse/BEAM-1409
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.6.0
>
>
> This ensures that even if the timestamps are representable in Java, they are 
> also representable within the model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1409) The DirectRunner should forbid elements outside of representable time.

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856343#comment-15856343
 ] 

Thomas Groh commented on BEAM-1409:
---

The DirectRunner enforces that element timestamps and timer timestamps must be 
within representable time.

> The DirectRunner should forbid elements outside of representable time.
> --
>
> Key: BEAM-1409
> URL: https://issues.apache.org/jira/browse/BEAM-1409
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.6.0
>
>
> This ensures that even if the timestamps are representable in Java, they are 
> also representable within the model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1409) The DirectRunner forbids elements outside of representable time.

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-1409:
--
Summary: The DirectRunner forbids elements outside of representable time.  
(was: The DirectRunner should forbid elements outside of representable time.)

> The DirectRunner forbids elements outside of representable time.
> 
>
> Key: BEAM-1409
> URL: https://issues.apache.org/jira/browse/BEAM-1409
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.6.0
>
>
> This ensures that even if the timestamps are representable in Java, they are 
> also representable within the model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1369) some unit tests in python take longer than 10s to run

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1369.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> some unit tests in python take longer than 10s to run
> -
>
> Key: BEAM-1369
> URL: https://issues.apache.org/jira/browse/BEAM-1369
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> 4 or 5 take up 50% of the time to run the unit tests.
> apache_beam.transforms.combiners_test.CombineTest.test_sample: 17.1545s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive: 
> 16.s
> apache_beam.examples.complete.estimate_pi_test.EstimatePiTest.test_basics: 
> 8.3667s
> apache_beam.io.textio_test.TextSourceTest.test_dynamic_work_rebalancing: 
> 1.9172s
> apache_beam.io.avroio_test.TestAvro.test_read_with_splitting_multiple_blocks: 
> 1.3939s
> apache_beam.io.textio_test.TextSourceTest.test_read_gzip_large: 1.2832s
> apache_beam.io.source_test_utils_test.SourceTestUtilsTest.test_split_at_fraction_exhaustive:
>  1.2787s
> apache_beam.io.bigquery_test.TestBigQueryReader.test_read_from_table_and_job_complete_retry:
>  1.0082s



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1331) Add documentation from reading compressed text sources

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1331.
-
Resolution: Fixed

> Add documentation from reading compressed text sources
> --
>
> Key: BEAM-1331
> URL: https://issues.apache.org/jira/browse/BEAM-1331
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Trivial
> Fix For: Not applicable
>
>
> Add a snippet for reading from compressed text sources.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1332) Temporary files in snippet tests are not cleaned up

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1332.
-
Resolution: Fixed

> Temporary files in snippet tests are not cleaned up 
> 
>
> Key: BEAM-1332
> URL: https://issues.apache.org/jira/browse/BEAM-1332
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> Files created in 
> https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/examples/snippets/snippets_test.py#L426
>  are not deleted at teardown. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-877) Allow disabling flattening of results when using BigQuery source

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-877.
-
Resolution: Fixed

> Allow disabling flattening of results when using BigQuery source
> 
>
> Key: BEAM-877
> URL: https://issues.apache.org/jira/browse/BEAM-877
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Sourabh Bajaj
> Fix For: Not applicable
>
>
> Java SDK supports disabling results flattening when creating a BQ source 
> using a query.
> https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L477
> Python SDK should be updated to support this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-852) Validate sources when they are created

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-852.
-
Resolution: Fixed

> Validate sources when they are created
> --
>
> Key: BEAM-852
> URL: https://issues.apache.org/jira/browse/BEAM-852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Sourabh Bajaj
> Fix For: Not applicable
>
>
> We currently do not validate some sources at creation time. For example text, 
> Avro. Validating sources early will improve user experience since it will  
> help catch issues early. For example, we can fail before submitting a job to 
> a runner.
> It should also be possible to disable validation to support environments 
> where users do not have access to the input at job submission. Java SDK 
> already follows a similar model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1406) Remove deprecated fileio.TextFileSink

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-1406.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Remove deprecated fileio.TextFileSink
> -
>
> Key: BEAM-1406
> URL: https://issues.apache.org/jira/browse/BEAM-1406
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Users should be using textio.WriteToText() instead of fileio.TextFileSink.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2561

[Apache Jenkins Server]

See 


Changes:

[tgroh] Check that Elements, Timers have permitted Timestamps

--
[...truncated 7134 lines...]
2017-02-07T17:19:34.601 [INFO] 2017-02-07\T\17:19:34.601 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/apis/google-api-services-storage/v1-rev35-1.20.0/google-api-services-storage-v1-rev35-1.20.0.pom
 (3 KB at 80.2 KB/sec)
2017-02-07T17:19:34.607 [INFO] 2017-02-07\T\17:19:34.606 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/2.0.3/jsr305-2.0.3.pom
2017-02-07T17:19:34.636 [INFO] 2017-02-07\T\17:19:34.636 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/2.0.3/jsr305-2.0.3.pom
 (2 KB at 45.1 KB/sec)
2017-02-07T17:19:34.671 [INFO] 2017-02-07\T\17:19:34.671 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.pom
2017-02-07T17:19:34.701 [INFO] 2017-02-07\T\17:19:34.701 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.pom
 (2 KB at 45.8 KB/sec)
2017-02-07T17:19:34.850 [INFO] 2017-02-07\T\17:19:34.850 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.20.0/google-api-client-java6-1.20.0.jar
2017-02-07T17:19:34.851 [INFO] 2017-02-07\T\17:19:34.851 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.20.0/google-api-client-jackson2-1.20.0.jar
2017-02-07T17:19:34.856 [INFO] 2017-02-07\T\17:19:34.856 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.20.0/google-oauth-client-java6-1.20.0.jar
2017-02-07T17:19:34.888 [INFO] 2017-02-07\T\17:19:34.888 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.20.0/google-api-client-java6-1.20.0.jar
 (3 KB at 59.8 KB/sec)
2017-02-07T17:19:34.924 [INFO] 2017-02-07\T\17:19:34.924 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.20.0/google-oauth-client-java6-1.20.0.jar
 (11 KB at 146.0 KB/sec)
2017-02-07T17:19:34.929 [INFO] 2017-02-07\T\17:19:34.929 [INFO] 

2017-02-07T17:19:34.929 [INFO] 2017-02-07\T\17:19:34.929 [INFO] BUILD FAILURE
2017-02-07T17:19:34.929 [INFO] 2017-02-07\T\17:19:34.929 [INFO] 

2017-02-07T17:19:34.930 [INFO] 2017-02-07\T\17:19:34.930 [INFO] Total time: 
3.436 s
2017-02-07T17:19:34.930 [INFO] 2017-02-07\T\17:19:34.930 [INFO] Finished at: 
2017-02-07T17:19:34+00:00
2017-02-07T17:19:35.051 [INFO] 2017-02-07\T\17:19:35.051 [INFO] Final Memory: 
10M/192M
2017-02-07T17:19:35.052 [INFO] 2017-02-07\T\17:19:35.051 [INFO] 

2017-02-07T17:19:35.060 [INFO] 2017-02-07\T\17:19:35.060 [ERROR] Failed to 
execute goal on project basic: Could not resolve dependencies for project 
archetype.it:basic:jar:0.1: Could not transfer artifact 
com.google.api-client:google-api-client-jackson2:jar:1.20.0 from/to central 
(https://repo.maven.apache.org/maven2): handshake alert:  unrecognized_name -> 
[Help 1]
2017-02-07T17:19:35.061 [INFO] 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
on project basic: Could not resolve dependencies for project 
archetype.it:basic:jar:0.1: Could not transfer artifact 
com.google.api-client:google-api-client-jackson2:jar:1.20.0 from/to central 
(https://repo.maven.apache.org/maven2): handshake alert:  unrecognized_name
2017-02-07T17:19:35.061 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
2017-02-07T17:19:35.062 [INFO]  at 
org.apache.maven

[jira] [Updated] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1384:
--
Priority: Major  (was: Minor)

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-1384:
-

Assignee: Daniel Halperin  (was: Jean-Baptiste Onofré)

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Daniel Halperin
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1937: [BEAM-1384] JmsIO: better errors during start, bett...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1937

[BEAM-1384] JmsIO: better errors during start, better testing

For BEAM-1384, the test has been failing because the error may be surfaced 
in either
start() or close(), depending on execution path. The underlying bug is that 
start
was implemented so that close might fail (this.connection would be set, but
this.connection.close would fail in a bad way).

Rewrite start() to fix the invariant needed by close() and also to provide
better error messages. Fixup the tests. Unfortunately, expectedException 
doesn't
really support testing causes and nested caused, so the rewrite dropped its 
use.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam jms-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1937.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1937


commit 522ca570e9556e41f84327adfe8f3fb1c96f47e6
Author: Dan Halperin 
Date:   2017-02-07T17:35:35Z

[BEAM-1384] JmsIO: better errors during start, better testing

For BEAM-1384, the test has been failing because the error may be surfaced 
in either
start() or close(), depending on execution path. The underlying bug is that 
start
was implemented so that close might fail (this.connection would be set, but
this.connection.close would fail in a bad way).

Rewrite start() to fix the invariant needed by close() and also to provide
better error messages. Fixup the tests. Unfortunately, expectedException 
doesn't
really support testing causes and nested caused, so the rewrite dropped its 
use.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856374#comment-15856374
 ] 

ASF GitHub Bot commented on BEAM-1384:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1937

[BEAM-1384] JmsIO: better errors during start, better testing

For BEAM-1384, the test has been failing because the error may be surfaced 
in either
start() or close(), depending on execution path. The underlying bug is that 
start
was implemented so that close might fail (this.connection would be set, but
this.connection.close would fail in a bad way).

Rewrite start() to fix the invariant needed by close() and also to provide
better error messages. Fixup the tests. Unfortunately, expectedException 
doesn't
really support testing causes and nested caused, so the rewrite dropped its 
use.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam jms-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1937.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1937


commit 522ca570e9556e41f84327adfe8f3fb1c96f47e6
Author: Dan Halperin 
Date:   2017-02-07T17:35:35Z

[BEAM-1384] JmsIO: better errors during start, better testing

For BEAM-1384, the test has been failing because the error may be surfaced 
in either
start() or close(), depending on execution path. The underlying bug is that 
start
was implemented so that close might fail (this.connection would be set, but
this.connection.close would fail in a bad way).

Rewrite start() to fix the invariant needed by close() and also to provide
better error messages. Fixup the tests. Unfortunately, expectedException 
doesn't
really support testing causes and nested caused, so the rewrite dropped its 
use.




> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Daniel Halperin
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-886) Support new DoFn in Python SDK

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-886.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
> Fix For: Not applicable
>
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2562

[Apache Jenkins Server]

See 

--
[...truncated 4755 lines...]
2017-02-07T18:10:37.366 [INFO] 

2017-02-07T18:10:37.395 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/datastore/datastore-v1-proto-client/1.4.0/datastore-v1-proto-client-1.4.0.pom
2017-02-07T18:10:37.438 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/datastore/datastore-v1-proto-client/1.4.0/datastore-v1-proto-client-1.4.0.pom
 (4 KB at 73.4 KB/sec)
2017-02-07T18:10:37.441 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/datastore/datastore-v1-proto-client-parent/1.4.0/datastore-v1-proto-client-parent-1.4.0.pom
2017-02-07T18:10:37.469 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/datastore/datastore-v1-proto-client-parent/1.4.0/datastore-v1-proto-client-parent-1.4.0.pom
 (7 KB at 245.2 KB/sec)
2017-02-07T18:10:37.473 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-protos/0.9.2/bigtable-protos-0.9.2.pom
2017-02-07T18:10:37.502 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-protos/0.9.2/bigtable-protos-0.9.2.pom
 (8 KB at 242.4 KB/sec)
2017-02-07T18:10:37.506 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-core-parent/0.9.2/bigtable-client-core-parent-0.9.2.pom
2017-02-07T18:10:37.535 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-core-parent/0.9.2/bigtable-client-core-parent-0.9.2.pom
 (2 KB at 48.4 KB/sec)
2017-02-07T18:10:37.537 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-parent/0.9.2/bigtable-client-parent-0.9.2.pom
2017-02-07T18:10:37.569 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-parent/0.9.2/bigtable-client-parent-0.9.2.pom
 (30 KB at 911.9 KB/sec)
2017-02-07T18:10:37.588 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-core/0.9.2/bigtable-client-core-0.9.2.pom
2017-02-07T18:10:37.618 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigtable/bigtable-client-core/0.9.2/bigtable-client-core-0.9.2.pom
 (5 KB at 156.4 KB/sec)
2017-02-07T18:10:37.622 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.pom
2017-02-07T18:10:37.652 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.pom
 (19 KB at 625.2 KB/sec)
2017-02-07T18:10:37.657 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-appengine/0.4.0/google-auth-library-appengine-0.4.0.pom
2017-02-07T18:10:37.686 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-appengine/0.4.0/google-auth-library-appengine-0.4.0.pom
 (3 KB at 78.9 KB/sec)
2017-02-07T18:10:37.689 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-parent/0.4.0/google-auth-library-parent-0.4.0.pom
2017-02-07T18:10:37.718 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-parent/0.4.0/google-auth-library-parent-0.4.0.pom
 (6 KB at 206.8 KB/sec)
2017-02-07T18:10:37.722 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/appengine/appengine-api-1.0-sdk/1.9.34/appengine-api-1.0-sdk-1.9.34.pom
2017-02-07T18:10:37.749 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/appengine/appengine-api-1.0-sdk/1.9.34/appengine-api-1.0-sdk-1.9.34.pom
 (607 B at 22.0 KB/sec)
2017-02-07T18:10:37.751 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/appengine/appengine/1.9.34/appengine-1.9.34.pom
2017-02-07T18:10:37.779 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/appengine/appengine/1.9.34/appengine-1.9.34.pom
 (845 B at 29.5 KB/sec)
2017-02-07T18:10:37.782 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/4/oss-parent-4.pom
2017-02-07T18:10:37.809 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/4/oss-parent-4.pom
 (4 KB at 144.7 KB/sec)
2017-02-07T18:10:37.837 [INFO] Downloading: 
https://repository.apache.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-02-07T18:10:37.837 [INFO] Downloading: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-02-07T18:10:38.056 [INFO] Downloaded: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
 (405 B at 1.8 KB/sec)
2017-02-07T18:10:38.059 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io

[2/2] beam git commit: This closes #1936

[altay]

This closes #1936


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/074031ca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/074031ca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/074031ca

Branch: refs/heads/master
Commit: 074031cac6ba58109e959395eaab73e797602420
Parents: 459a8f8 a732f22
Author: Ahmet Altay 
Authored: Tue Feb 7 09:58:38 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 7 09:58:38 2017 -0800

--
 sdks/python/apache_beam/examples/snippets/snippets_test.py | 6 ++
 1 file changed, 6 insertions(+)
--

[1/2] beam git commit: Cleanup files after the tests end

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 459a8f836 -> 074031cac


Cleanup files after the tests end


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a732f227
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a732f227
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a732f227

Branch: refs/heads/master
Commit: a732f227b2f768855bbe5ecac32bd357aeda2a55
Parents: 459a8f8
Author: Sourabh Bajaj 
Authored: Fri Jan 27 15:46:30 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 7 09:58:28 2017 -0800

--
 sdks/python/apache_beam/examples/snippets/snippets_test.py | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a732f227/sdks/python/apache_beam/examples/snippets/snippets_test.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py 
b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index 4827e94..95c959b 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -427,14 +427,18 @@ class SnippetsTest(unittest.TestCase):
 # real data.
 beam.io.ReadFromText = SnippetsTest.DummyReadTransform
 beam.io.WriteToText = SnippetsTest.DummyWriteTransform
+self.temp_files = []
 
   def tearDown(self):
 beam.io.ReadFromText = self.old_read_from_text
 beam.io.WriteToText = self.old_write_to_text
+# Cleanup all the temporary files created in the test
+map(os.remove, self.temp_files)
 
   def create_temp_file(self, contents=''):
 with tempfile.NamedTemporaryFile(delete=False) as f:
   f.write(contents)
+  self.temp_files.append(f.name)
   return f.name
 
   def get_output(self, path, sorted_output=True, suffix=''):
@@ -560,6 +564,8 @@ class SnippetsTest(unittest.TestCase):
 gzip_file_name = temp_path + '.gz'
 with open(temp_path) as src, gzip.open(gzip_file_name, 'wb') as dst:
   dst.writelines(src)
+  # Add the temporary gzip file to be cleaned up as well.
+  self.temp_files.append(gzip_file_name)
 snippets.model_textio_compressed(
 {'read': gzip_file_name}, ['aa', 'bb', 'cc'])
 

[jira] [Commented] (BEAM-1332) Temporary files in snippet tests are not cleaned up

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856454#comment-15856454
 ] 

ASF GitHub Bot commented on BEAM-1332:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1936


> Temporary files in snippet tests are not cleaned up 
> 
>
> Key: BEAM-1332
> URL: https://issues.apache.org/jira/browse/BEAM-1332
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> Files created in 
> https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/examples/snippets/snippets_test.py#L426
>  are not deleted at teardown. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856453#comment-15856453
 ] 

Jingsong Lee commented on BEAM-1393:


(y) 
But there is a problem with the 6 point. 
AbstractStreamOperator.processWatermark did two things: advanceTimerWatermark 
and emitWatermark. 
Now in Beam, advanceTimerWatermark is determined by inputWatermark, 
emitWatermark is determined by Min(inputWatermark, 
StateInternals.watermarkHold). Their watermarks is not the same. 
Even, advancing watermarkHold need fireTimer to clear early namespace.

> Update Flink Runner to Flink 1.2.0
> --
>
> Key: BEAM-1393
> URL: https://issues.apache.org/jira/browse/BEAM-1393
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
>
> When we update to 1.2.0 we can use the new internal Timer API that is 
> available to Flink operators: {{InternalTimerService}} and also use broadcast 
> state to store side-input data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1936: [BEAM-1332] Cleanup files after the tests end

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1936


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-588) All runners should support ProfilingOptions

[Younghee Kwon (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856468#comment-15856468
 ] 

Younghee Kwon commented on BEAM-588:


The PR is about to be merged. 

Several things to do in a successive PR: 
 1. integrate the memory reporter into DirectRunner using PipelineOptions
 2. having an option to dump full profile into disk (as opposed to only logging 
the biggest 10 as now).
 3. (optional) experiment with other profilers for the platforms that guppy is 
not available.


> All runners should support ProfilingOptions
> ---
>
> Key: BEAM-588
> URL: https://issues.apache.org/jira/browse/BEAM-588
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py#L366
> This is useful for profiling pipelines in different environments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1412) Maven install postcommit broken (to be investigated)

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1412:
-

 Summary: Maven install postcommit broken (to be investigated)
 Key: BEAM-1412
 URL: https://issues.apache.org/jira/browse/BEAM-1412
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Kenneth Knowles
Priority: Blocker


https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/ went 
red / yellow / red around here: 
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/2556/

I've just looked in on two that were both a failure to transfer a dependency. 
There may be other things going on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #1928

[altay]

This closes #1928


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/63dc08ee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/63dc08ee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/63dc08ee

Branch: refs/heads/master
Commit: 63dc08ee1638fedfe0465c8894c7211aeb1c7a6d
Parents: 074031c 0a59205
Author: Ahmet Altay 
Authored: Tue Feb 7 10:18:22 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 7 10:18:22 2017 -0800

--
 sdks/python/apache_beam/utils/profiler.py | 79 +-
 1 file changed, 78 insertions(+), 1 deletion(-)
--

[1/2] beam git commit: To add sdks/python/utils/profiler a MemoryReporter that tracks heap profiles.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 074031cac -> 63dc08ee1


To add sdks/python/utils/profiler a MemoryReporter that tracks heap profiles.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a592053
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a592053
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a592053

Branch: refs/heads/master
Commit: 0a59205314d21f452b91848496d6ae21e369d7df
Parents: 074031c
Author: Younghee Kwon 
Authored: Mon Feb 6 12:35:50 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 7 10:16:42 2017 -0800

--
 sdks/python/apache_beam/utils/profiler.py | 79 +-
 1 file changed, 78 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0a592053/sdks/python/apache_beam/utils/profiler.py
--
diff --git a/sdks/python/apache_beam/utils/profiler.py 
b/sdks/python/apache_beam/utils/profiler.py
index 1214172..3599f98 100644
--- a/sdks/python/apache_beam/utils/profiler.py
+++ b/sdks/python/apache_beam/utils/profiler.py
@@ -24,7 +24,8 @@ import pstats
 import StringIO
 import tempfile
 import time
-
+from threading import Timer
+import warnings
 
 from apache_beam.utils.dependency import _dependency_file_copy
 
@@ -67,3 +68,79 @@ class Profile(object):
   self.profile, stream=s).sort_stats(Profile.SORTBY)
   self.stats.print_stats()
   logging.info('Profiler data: [%s]', s.getvalue())
+
+
+class MemoryReporter(object):
+  """A memory reporter that reports the memory usage and heap profile.
+  Usage:
+mr = MemoryReporter(interval_second=30.0)
+mr.start()
+while ...
+  
+  # this will report continuously with 30 seconds between reports.
+mr.stop()
+
+  NOTE: A reporter with start() should always stop(), or the parent process can
+  never finish.
+
+  Or simply the following which does star() and stop():
+with MemoryReporter(interval_second=100):
+  while ...
+
+
+  Also it could report on demand without continuous reporting.
+mr = MemoryReporter()  # default interval 60s but not started.
+
+mr.report_once()
+  """
+
+  def __init__(self, interval_second=60.0):
+# guppy might not have installed. http://pypi.python.org/pypi/guppy/0.1.10
+# The reporter can be set up only when guppy is installed (and guppy cannot
+# be added to the required packages in setup.py, since it's not available
+# in all platforms).
+try:
+  from guppy import hpy  # pylint: disable=import-error
+  self._hpy = hpy
+  self._interval_second = interval_second
+  self._timer = None
+except ImportError:
+  warnings.warn('guppy is not installed; MemoryReporter not available.')
+  self._hpy = None
+self._enabled = False
+
+  def __enter__(self):
+self.start()
+return self
+
+  def __exit__(self, *args):
+self.stop()
+
+  def start(self):
+if self._enabled or not self._hpy:
+  return
+self._enabled = True
+
+def report_with_interval():
+  if not self._enabled:
+return
+  self.report_once()
+  self._timer = Timer(self._interval_second, report_with_interval)
+  self._timer.start()
+
+self._timer = Timer(self._interval_second, report_with_interval)
+self._timer.start()
+
+  def stop(self):
+if not self._enabled:
+  return
+self._timer.cancel()
+self._enabled = False
+
+  def report_once(self):
+if not self._hpy:
+  return
+report_start_time = time.time()
+heap_profile = self._hpy().heap()
+logging.info('*** MemoryReport Heap:\n %s\n MemoryReport took %.1f 
seconds',
+ heap_profile, time.time() - report_start_time)

[GitHub] beam pull request #1928: [BEAM-588] Add MemoryReporter to python-sdk

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1928


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856501#comment-15856501
 ] 

Kenneth Knowles commented on BEAM-25:
-

We do have a vague plan to support SetState and MapState directly. I don't know 
if there are JIRAs filed, but they were mentioned only briefly here: 
https://s.apache.org/beam-state#heading=h.80ytngnwpr7h

Definitely good ideas.

> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-588) All runners should support ProfilingOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856502#comment-15856502
 ] 

ASF GitHub Bot commented on BEAM-588:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1928


> All runners should support ProfilingOptions
> ---
>
> Key: BEAM-588
> URL: https://issues.apache.org/jira/browse/BEAM-588
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py#L366
> This is useful for profiling pipelines in different environments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Apex #430

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2564

[Apache Jenkins Server]

See 


Changes:

[altay] To add sdks/python/utils/profiler a MemoryReporter that tracks heap

--
[...truncated 26 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 36289
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_MavenInstall] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 36289
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 
 
-Dmaven.repo.local=
 -B -e -P release,dataflow-runner clean install coveralls:report 
-DrepoToken= -DskipITs=false -DintegrationTestPipelineOptions=[ 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]
2017-02-07T18:37:37.913 [INFO] Error stacktraces are turned on.
2017-02-07T18:37:38.223 [INFO] Scanning for projects...
2017-02-07T18:37:39.467 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T18:37:39.789 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 18.5 KB/sec)
2017-02-07T18:37:39.799 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T18:37:39.838 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 164.5 KB/sec)
2017-02-07T18:37:39.850 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T18:37:39.885 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 94.4 KB/sec)
2017-02-07T18:37:39.887 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T18:37:39.939 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 424.2 KB/sec)
2017-02-07T18:37:39.944 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T18:37:39.985 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 776.2 KB/sec)
2017-02-07T18:37:39.992 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T18:37:40.025 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 413.5 KB/sec)
2017-02-07T18:37:40.035 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T18:37:40.067 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 126.4 KB/sec)
2017-02-07T18:37:40.073 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T18:37:40.103 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 110.5 KB/sec)
2017-02-07T18:37:40.106 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T18:37:40.141 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 570.5 KB/sec)
2017-02-07T18:37:40.145 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T18:37:40.177 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 206.3 KB/sec)
2017-02-07T18:37:40.179 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T18:37:40.212 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
 (14 KB at 401.3

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Dataflow #2224

[Apache Jenkins Server]

See 


Changes:

[altay] Cleanup files after the tests end

[altay] To add sdks/python/utils/profiler a MemoryReporter that tracks heap

--
[...truncated 28 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 52461
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Dataflow] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 52461
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/google-cloud-dataflow-java -DforkCount=0 
-DrunnableOnServicePipelineOptions=[ 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner", 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-runnable-on-service-tests/" ]
2017-02-07T18:42:26.626 [INFO] Error stacktraces are turned on.
2017-02-07T18:42:26.832 [INFO] Scanning for projects...
2017-02-07T18:42:29.372 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T18:42:29.830 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 13.0 KB/sec)
2017-02-07T18:42:29.846 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T18:42:29.891 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 145.8 KB/sec)
2017-02-07T18:42:29.935 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T18:42:29.981 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 71.8 KB/sec)
2017-02-07T18:42:29.987 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T18:42:30.055 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 324.4 KB/sec)
2017-02-07T18:42:30.061 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T18:42:30.119 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 548.7 KB/sec)
2017-02-07T18:42:30.125 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T18:42:30.160 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 389.9 KB/sec)
2017-02-07T18:42:30.173 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T18:42:30.214 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 101.1 KB/sec)
2017-02-07T18:42:30.223 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T18:42:30.254 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 106.9 KB/sec)
2017-02-07T18:42:30.258 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T18:42:30.298 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 499.2 KB/sec)
2017-02-07T18:42:30.304 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T18:42:30.337 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 200.0 KB/sec)
2017-02-07T18:42:30.341 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T18:42:30.377 [INFO] Downloaded: 
https://repo.mave

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #877

[Apache Jenkins Server]

See 


Changes:

[altay] To add sdks/python/utils/profiler a MemoryReporter that tracks heap

--
[...truncated 179 lines...]
2017-02-07T18:46:55.287 [INFO] Apache Beam :: Runners
2017-02-07T18:46:55.287 [INFO] Apache Beam :: Runners :: Core Java
2017-02-07T18:46:55.287 [INFO] Apache Beam :: Runners :: Direct Java
2017-02-07T18:46:55.287 [INFO] Apache Beam :: SDKs :: Java :: IO
2017-02-07T18:46:55.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka
2017-02-07T18:46:55.287 [INFO] Apache Beam :: Runners :: Spark
2017-02-07T18:46:55.525 [INFO]  
   
2017-02-07T18:46:55.525 [INFO] 

2017-02-07T18:46:55.525 [INFO] Building Apache Beam :: Parent 0.6.0-SNAPSHOT
2017-02-07T18:46:55.526 [INFO] 

2017-02-07T18:46:55.529 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-remote-resources-plugin/1.5/maven-remote-resources-plugin-1.5.pom
2017-02-07T18:46:55.561 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-remote-resources-plugin/1.5/maven-remote-resources-plugin-1.5.pom
 (14 KB at 407.5 KB/sec)
2017-02-07T18:46:55.564 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/24/maven-plugins-24.pom
2017-02-07T18:46:55.594 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/24/maven-plugins-24.pom
 (11 KB at 351.8 KB/sec)
2017-02-07T18:46:55.599 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-remote-resources-plugin/1.5/maven-remote-resources-plugin-1.5.jar
2017-02-07T18:46:55.644 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-remote-resources-plugin/1.5/maven-remote-resources-plugin-1.5.jar
 (68 KB at 1505.3 KB/sec)
2017-02-07T18:46:55.655 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-site-plugin/3.5.1/maven-site-plugin-3.5.1.pom
2017-02-07T18:46:55.687 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-site-plugin/3.5.1/maven-site-plugin-3.5.1.pom
 (18 KB at 547.7 KB/sec)
2017-02-07T18:46:55.692 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/28/maven-plugins-28.pom
2017-02-07T18:46:55.723 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/28/maven-plugins-28.pom
 (12 KB at 391.6 KB/sec)
2017-02-07T18:46:55.726 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/27/maven-parent-27.pom
2017-02-07T18:46:55.762 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/27/maven-parent-27.pom
 (40 KB at 1104.3 KB/sec)
2017-02-07T18:46:55.766 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/17/apache-17.pom
2017-02-07T18:46:55.798 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/17/apache-17.pom (16 KB 
at 490.0 KB/sec)
2017-02-07T18:46:55.803 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-site-plugin/3.5.1/maven-site-plugin-3.5.1.jar
2017-02-07T18:46:55.867 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-site-plugin/3.5.1/maven-site-plugin-3.5.1.jar
 (129 KB at 2008.4 KB/sec)
2017-02-07T18:46:55.875 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.pom
2017-02-07T18:46:55.909 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.pom
 (12 KB at 344.4 KB/sec)
2017-02-07T18:46:55.912 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/27/maven-plugins-27.pom
2017-02-07T18:46:55.953 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/27/maven-plugins-27.pom
 (12 KB at 270.7 KB/sec)
2017-02-07T18:46:55.959 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.jar
2017-02-07T18:46:56.018 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.jar
 (157 KB at 2647.3 KB/sec)
2017-02-07T18:46:56.031 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-enforcer-plugin/1.3.1/maven-enforcer-plugin-1.3.1.pom
2017-02-07T18:46:56.062 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-enforcer-pl

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856576#comment-15856576
 ] 

Kenneth Knowles commented on BEAM-25:
-

The JIRAs are BEAM-41 and BEAM-42.

> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-41) State for "Set" data, supporting efficient membership checks

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856591#comment-15856591
 ] 

Kenneth Knowles commented on BEAM-41:
-

Yes, this seems about right. Just to be clear, you may want to override `add()` 
to note in the javadoc that it is still a "blind write" that does not require 
reading. We can work out the details in a PR if you like.

> State for "Set" data, supporting efficient membership checks
> 
>
> Key: BEAM-41
> URL: https://issues.apache.org/jira/browse/BEAM-41
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: State
>
> Analogous to how BagState supports efficient blind writes, a SetState would 
> support efficient membership checking without reading the entire set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-41) State for "Set" data, supporting efficient membership checks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-41:
---

Assignee: Jingsong Lee  (was: Kenneth Knowles)

> State for "Set" data, supporting efficient membership checks
> 
>
> Key: BEAM-41
> URL: https://issues.apache.org/jira/browse/BEAM-41
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
>Priority: Minor
>  Labels: State
>
> Analogous to how BagState supports efficient blind writes, a SetState would 
> support efficient membership checking without reading the entire set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-41) State for "Set" data, supporting efficient membership checks

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856595#comment-15856595
 ] 

Kenneth Knowles commented on BEAM-41:
-

One thing to consider is whether there is a version of {{readLater}} for the 
{{contains}} query on a few elements, and also a bulk {{contains}} query. You 
shouldn't have to read the whole set to ask these queries, and you shouldn't 
have to do separate reads for them either.

> State for "Set" data, supporting efficient membership checks
> 
>
> Key: BEAM-41
> URL: https://issues.apache.org/jira/browse/BEAM-41
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
>Priority: Minor
>  Labels: State
>
> Analogous to how BagState supports efficient blind writes, a SetState would 
> support efficient membership checking without reading the entire set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1413) DoFn javadoc has @literal where it shouldn't

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1413:
-

 Summary: DoFn javadoc has @literal where it shouldn't
 Key: BEAM-1413
 URL: https://issues.apache.org/jira/browse/BEAM-1413
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


In particular, the docs for StateId and TimerId annotations have an extra layer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1938: [BEAM-646] Add PTransformMatcher

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1938

[BEAM-646] Add PTransformMatcher

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This interface is used to identify PTransforms to replace during
Pipeline Surgery.

Add ClassPTransformMatcher as the initial simple matcher.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam ptransform_matchers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1938.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1938


commit 6ca7d4ac186e4d11606e0cf7dd767eaf31a1da2d
Author: Thomas Groh 
Date:   2017-02-07T17:35:25Z

Add PTransformMatcher

This interface is used to identify PTransforms to replace during
Pipeline Surgery.

Add ClassPTransformMatcher as the initial simple matcher.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856646#comment-15856646
 ] 

ASF GitHub Bot commented on BEAM-646:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1938

[BEAM-646] Add PTransformMatcher

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This interface is used to identify PTransforms to replace during
Pipeline Surgery.

Add ClassPTransformMatcher as the initial simple matcher.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam ptransform_matchers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1938.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1938


commit 6ca7d4ac186e4d11606e0cf7dd767eaf31a1da2d
Author: Thomas Groh 
Date:   2017-02-07T17:35:25Z

Add PTransformMatcher

This interface is used to identify PTransforms to replace during
Pipeline Surgery.

Add ClassPTransformMatcher as the initial simple matcher.




> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1407) Support multiple Kafka client in KakaIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856653#comment-15856653
 ] 

Xu Mingmin commented on BEAM-1407:
--

add a design doc 
https://docs.google.com/document/d/1YlCWws4SYCqUWAtVz9mrmVdFNM8b3E8DLCImYep-I6k/edit?usp=sharing
 
[~davor] could you help for a review?

> Support multiple Kafka client in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, (maybe 0.8 as well 
> ?), 
> 1). to fully leverage new features in each version, like external 
> authentication, timestamp in 0.10;
> 2). hide kafka API changed, to support steamless switch between different 
> cluster versions;
> Scope of change:
> 1). add an abstract API for Kafka Consumer in existing KafkaIO. By default 
> it's kafka 0.9 to keep compatible, with an option for extension;
> 2). Wrapped API for Kafka 0.10, with Kafka message timestamp support;
> 3). wrapped API for Kafka 0.8 (TBD);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #1937

[dhalperi]

This closes #1937


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8c1a577f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8c1a577f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8c1a577f

Branch: refs/heads/master
Commit: 8c1a577fe52ffd02e57f3e65924aced10e38a82a
Parents: 63dc08e 8ef78d7
Author: Dan Halperin 
Authored: Tue Feb 7 12:49:05 2017 -0800
Committer: Dan Halperin 
Committed: Tue Feb 7 12:49:05 2017 -0800

--
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  | 24 +
 .../org/apache/beam/sdk/io/jms/JmsIOTest.java   | 37 +---
 2 files changed, 42 insertions(+), 19 deletions(-)
--

[GitHub] beam pull request #1937: [BEAM-1384] JmsIO: better errors during start, bett...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1937


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1384] JmsIO: better errors during start, better testing

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 63dc08ee1 -> 8c1a577fe


[BEAM-1384] JmsIO: better errors during start, better testing

For BEAM-1384, the test has been failing because the error may be surfaced in 
either
start() or close(), depending on execution path. The underlying bug is that 
start
was implemented so that close might fail (this.connection would be set, but
this.connection.close would fail in a bad way).

Rewrite start() to fix the invariant needed by close() and also to provide
better error messages. Fixup the tests. Unfortunately, expectedException doesn't
really support testing causes and nested caused, so the rewrite dropped its use.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8ef78d72
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8ef78d72
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8ef78d72

Branch: refs/heads/master
Commit: 8ef78d72b903bded2c8db4125286e6f07e4a4a0b
Parents: 63dc08e
Author: Dan Halperin 
Authored: Tue Feb 7 09:35:35 2017 -0800
Committer: Dan Halperin 
Committed: Tue Feb 7 12:49:02 2017 -0800

--
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  | 24 +
 .../org/apache/beam/sdk/io/jms/JmsIOTest.java   | 37 +---
 2 files changed, 42 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8ef78d72/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
--
diff --git 
a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java 
b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
index 270fe31..a935b56 100644
--- a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
+++ b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
@@ -405,14 +405,26 @@ public class JmsIO {
   Read spec = source.spec;
   ConnectionFactory connectionFactory = spec.getConnectionFactory();
   try {
+Connection connection;
 if (spec.getUsername() != null) {
-  this.connection =
+  connection =
   connectionFactory.createConnection(spec.getUsername(), 
spec.getPassword());
 } else {
-  this.connection = connectionFactory.createConnection();
+  connection = connectionFactory.createConnection();
 }
-this.connection.start();
+connection.start();
+this.connection = connection;
+  } catch (Exception e) {
+throw new IOException("Error connecting to JMS", e);
+  }
+
+  try {
 this.session = this.connection.createSession(false, 
Session.AUTO_ACKNOWLEDGE);
+  } catch (Exception e) {
+throw new IOException("Error creating JMS session", e);
+  }
+
+  try {
 if (spec.getTopic() != null) {
   this.consumer =
   
this.session.createConsumer(this.session.createTopic(spec.getTopic()));
@@ -420,11 +432,11 @@ public class JmsIO {
   this.consumer =
   
this.session.createConsumer(this.session.createQueue(spec.getQueue()));
 }
-
-return advance();
   } catch (Exception e) {
-throw new IOException(e);
+throw new IOException("Error creating JMS consumer", e);
   }
+
+  return advance();
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/8ef78d72/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
--
diff --git 
a/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java 
b/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
index a06bba3..f07247d 100644
--- a/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
+++ b/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
@@ -17,14 +17,20 @@
  */
 package org.apache.beam.sdk.io.jms;
 
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.instanceOf;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
-
 import javax.jms.Connection;
 import javax.jms.ConnectionFactory;
+import javax.jms.JMSException;
 import javax.jms.Message;
 import javax.jms.MessageConsumer;
 import javax.jms.MessageProducer;
@@ -49,7 +55,6 @@ import org.junit.Before;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
-import org.junit.rules.ExpectedException;
 import org.junit.runner.RunWith;
 import org.junit.runn

[jira] [Commented] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856757#comment-15856757
 ] 

ASF GitHub Bot commented on BEAM-1384:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1937


> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Daniel Halperin
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1384) JmsIO postcommit failure: testAuthenticationWithBadPassword

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1384.
---
   Resolution: Fixed
Fix Version/s: 0.6.0

Rewrote racy code in start() so that close() doesn't throw.

> JmsIO postcommit failure: testAuthenticationWithBadPassword
> ---
>
> Key: BEAM-1384
> URL: https://issues.apache.org/jira/browse/BEAM-1384
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Daniel Halperin
> Fix For: 0.6.0
>
>
> Postcommit failure here: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2517/org.apache.beam$beam-sdks-java-io-jms/testReport/junit/org.apache.beam.sdk.io.jms/JmsIOTest/testAuthenticationWithBadPassword/
> It seems to be looking for a message which is actually there but buried deep 
> in the cause stack.
> Filing as minor, but if this recurs it should be moved up to Major or Blocker.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1413) DoFn javadoc has @literal where it shouldn't

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856782#comment-15856782
 ] 

Daniel Halperin commented on BEAM-1413:
---

Grepping the website:

{code}
[12:58:23 dhalperi@dhalperi:0.5.0 903fd828 ] grep literal -R .  
[]
./org/apache/beam/runners/dataflow/util/TimeUtil.html: that is, seconds with 
optional fractional seconds and a literal 's' at the end.
./org/apache/beam/runners/core/BaseExecutionContext.html: {@literal @}Override
./org/apache/beam/sdk/io/kinesis/KinesisIO.html:   {@literal @}Override
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}TimerId("my-timer-id")
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}TimerId("my-timer-id") Timer myTimer) {
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}OnTimer("my-timer-id")
./org/apache/beam/sdk/transforms/DoFn.html: {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/display/DisplayData.html: {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/CombineFns.html:   {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/GroupByKey.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal 
@}StateId("my-state-id")
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal 
@}StateId("my-state-id") ValueState myState) {
./org/apache/beam/sdk/transforms/Latest.html:  {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/View.html: {@literal @}Override
./org/apache/beam/sdk/transforms/join/CoGroupByKey.html:   {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/Aggregator.html:   {@literal @}ProcessElement
{code}

> DoFn javadoc has @literal where it shouldn't
> 
>
> Key: BEAM-1413
> URL: https://issues.apache.org/jira/browse/BEAM-1413
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> In particular, the docs for StateId and TimerId annotations have an extra 
> layer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Issue Comment Deleted] (BEAM-1413) DoFn javadoc has @literal where it shouldn't

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1413:
--
Comment: was deleted

(was: Grepping the website:

{code}
[12:58:23 dhalperi@dhalperi:0.5.0 903fd828 ] grep literal -R .  
[]
./org/apache/beam/runners/dataflow/util/TimeUtil.html: that is, seconds with 
optional fractional seconds and a literal 's' at the end.
./org/apache/beam/runners/core/BaseExecutionContext.html: {@literal @}Override
./org/apache/beam/sdk/io/kinesis/KinesisIO.html:   {@literal @}Override
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}TimerId("my-timer-id")
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}TimerId("my-timer-id") Timer myTimer) {
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}OnTimer("my-timer-id")
./org/apache/beam/sdk/transforms/DoFn.html: {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/display/DisplayData.html: {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/CombineFns.html:   {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/GroupByKey.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal 
@}StateId("my-state-id")
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal 
@}StateId("my-state-id") ValueState myState) {
./org/apache/beam/sdk/transforms/Latest.html:  {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/View.html: {@literal @}Override
./org/apache/beam/sdk/transforms/join/CoGroupByKey.html:   {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/Aggregator.html:   {@literal @}ProcessElement
{code})

> DoFn javadoc has @literal where it shouldn't
> 
>
> Key: BEAM-1413
> URL: https://issues.apache.org/jira/browse/BEAM-1413
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> In particular, the docs for StateId and TimerId annotations have an extra 
> layer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1413) DoFn javadoc has @literal where it shouldn't

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856781#comment-15856781
 ] 

Daniel Halperin commented on BEAM-1413:
---

Grepping the website:

{code}
[12:58:23 dhalperi@dhalperi:0.5.0 903fd828 ] grep literal -R .  
[]
./org/apache/beam/runners/dataflow/util/TimeUtil.html: that is, seconds with 
optional fractional seconds and a literal 's' at the end.
./org/apache/beam/runners/core/BaseExecutionContext.html: {@literal @}Override
./org/apache/beam/sdk/io/kinesis/KinesisIO.html:   {@literal @}Override
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}TimerId("my-timer-id")
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}TimerId("my-timer-id") Timer myTimer) {
./org/apache/beam/sdk/transforms/DoFn.TimerId.html:   {@literal 
@}OnTimer("my-timer-id")
./org/apache/beam/sdk/transforms/DoFn.html: {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/display/DisplayData.html: {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/CombineFns.html:   {@literal @}Override
./org/apache/beam/sdk/transforms/CombineFns.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal @}ProcessElement
./org/apache/beam/sdk/transforms/ParDo.html:{@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/GroupByKey.html:  {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal 
@}StateId("my-state-id")
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/DoFn.StateId.html:   {@literal 
@}StateId("my-state-id") ValueState myState) {
./org/apache/beam/sdk/transforms/Latest.html:  {@literal @}ProcessElement
./org/apache/beam/sdk/transforms/View.html: {@literal @}Override
./org/apache/beam/sdk/transforms/join/CoGroupByKey.html:   {@literal 
@}ProcessElement
./org/apache/beam/sdk/transforms/Aggregator.html:   {@literal @}ProcessElement
{code}

> DoFn javadoc has @literal where it shouldn't
> 
>
> Key: BEAM-1413
> URL: https://issues.apache.org/jira/browse/BEAM-1413
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> In particular, the docs for StateId and TimerId annotations have an extra 
> layer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #878

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2565

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1402) Make TextIO and AvroIO use best-practice types.

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856841#comment-15856841
 ] 

Eugene Kirpichov commented on BEAM-1402:


More specifically, the necessary changes are:

- Specify type parameters explicitly: TextIO.read() etc.
- Get rid of classes like TextIO.Read.Bound - instead, TextIO.Read should be 
the transform class itself
- Get rid of static builder-like .withBlah() methods - everything should be 
built using non-static methods on top of .read(), .write() and alike
- Remove coders from the TextIO API
- (optional) use AutoValue

The PR already does the first 3 of these items.

> Make TextIO and AvroIO use best-practice types.
> ---
>
> Key: BEAM-1402
> URL: https://issues.apache.org/jira/browse/BEAM-1402
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Davor Bonaci
>
> Replace static Read/Write classes with type-instantiated classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1414) CountingInput should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1414:
--

 Summary: CountingInput should comply with PTransform style guide
 Key: BEAM-1414
 URL: https://issues.apache.org/jira/browse/BEAM-1414
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes:

- Rename the whole class and its inner transforms to sound more verb-like, 
e.g.: GenerateRange.Bounded/Unbounded (as opposed to current 
CountingInput.BoundedCountingInput)
- Provide a more unified API between bounded and unbounded cases: 
GenerateRange.from(100) should return a GenerateRange.Unbounded; 
GenerateRange.from(100).to(200) should return a GenerateRange.Bounded. They 
both should accept a timestampFn. The unbounded one _should not_ have a 
withMaxNumRecords builder - that's redundant with specifying the range.
- (optional) Use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1415) PubsubIO should comply with PTransfrom style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1415:
--

 Summary: PubsubIO should comply with PTransfrom style guide
 Key: BEAM-1415
 URL: https://issues.apache.org/jira/browse/BEAM-1415
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Eugene Kirpichov
Assignee: Daniel Halperin


Suggested changes:

- Rename builder methods such as .subscription(), .topic() etc. to 
.withSubscription, .withTopic()
- Replace use of Coder from the API (.withCoder()) with a SerializableFunction
- Rename .withAttributes() to something else, because it sounds like this is a 
function that sets attributes.
- (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1416) Write transform should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1416:
--

 Summary: Write transform should comply with PTransform style guide
 Key: BEAM-1416
 URL: https://issues.apache.org/jira/browse/BEAM-1416
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Eugene Kirpichov
Assignee: Daniel Halperin


Suggested change: remove Bound class - Write itself should be the transform 
class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1417) Count should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1417:
--

 Summary: Count should comply with PTransform style guide
 Key: BEAM-1417
 URL: https://issues.apache.org/jira/browse/BEAM-1417
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested change (backward-compatible): methods globally(), perKey(), 
perElement() should return respective transform classes - Count.Globally, 
Count.PerKey, Count.PerElement. (currently some of them return a PTransform 
which is actually a Combine - the suggestion is to wrap them into classes).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1418) MapElements and FlatMapElements should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1418:
--

 Summary: MapElements and FlatMapElements should comply with 
PTransform style guide
 Key: BEAM-1418
 URL: https://issues.apache.org/jira/browse/BEAM-1418
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Type structure of these classes can be improved by slightly changing the API:

FlatMapElements.into(TypeDescriptor).via(as usual).

Likewise for MapElements. This allows getting rid of the awkward 
MissingOutputTypeDescriptor intermediate class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1419) Flatten should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1419:
--

 Summary: Flatten should comply with PTransform style guide
 Key: BEAM-1419
 URL: https://issues.apache.org/jira/browse/BEAM-1419
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes: rename classes Flatten.FlattenIterables and 
Flatten.FlattenPCollectionList respectively to Flatten.Iterables and 
Flatten.PCollectionList



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1420) GroupByKey should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1420:
--

 Summary: GroupByKey should comply with PTransform style guide
 Key: BEAM-1420
 URL: https://issues.apache.org/jira/browse/BEAM-1420
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested change: rename GroupByKey.create(boolean fewKeys) to 
GroupByKey.createWithFewKeys().



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1421) Latest should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1421:
--

 Summary: Latest should comply with PTransform style guide
 Key: BEAM-1421
 URL: https://issues.apache.org/jira/browse/BEAM-1421
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes: Latest.globally() and Latest.perKey() should return 
respectively a Latest.Globally and a Latest.PerKey, rather than generic 
PTransform.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2225

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1422) ParDo should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1422:
--

 Summary: ParDo should comply with PTransform style guide
 Key: BEAM-1422
 URL: https://issues.apache.org/jira/browse/BEAM-1422
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes:

- Get rid of ParDo.Unbound and UnboundMulti classes completely
- Get rid of static methods such as withSideInputs/Outputs() - the only entry 
point should be via ParDo.of(). Respectively, get rid of non-static .of().
- Rename ParDo.Bound and ParDo.BoundMulti respectively to simply ParDo and 
ParDoWithSideOutputs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1423) Sample should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1423:
--

 Summary: Sample should comply with PTransform style guide
 Key: BEAM-1423
 URL: https://issues.apache.org/jira/browse/BEAM-1423
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes:
- Rename Sample.SampleAny to Sample.Any
- Methods any() and fixedSizeGlobally() should return respective classes: 
Sample.Any and (new wrapper PTransform class) Sample.FixedSizeGlobally



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1407) Support multiple Kafka client in KakaIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856891#comment-15856891
 ] 

Davor Bonaci commented on BEAM-1407:


It looks good to me, but I'll plus-in a few more folks to take a peek as well.

> Support multiple Kafka client in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, (maybe 0.8 as well 
> ?), 
> 1). to fully leverage new features in each version, like external 
> authentication, timestamp in 0.10;
> 2). hide kafka API changed, to support steamless switch between different 
> cluster versions;
> Scope of change:
> 1). add an abstract API for Kafka Consumer in existing KafkaIO. By default 
> it's kafka 0.9 to keep compatible, with an option for extension;
> 2). Wrapped API for Kafka 0.10, with Kafka message timestamp support;
> 3). wrapped API for Kafka 0.8 (TBD);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1424) ToString should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1424:
--

 Summary: ToString should comply with PTransform style guide
 Key: BEAM-1424
 URL: https://issues.apache.org/jira/browse/BEAM-1424
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes:

- Rename of() to elements(), kv() to kvs(), iterable() to iterables() for 
consistency between each other and with other Beam similar transforms e.g. 
Flatten.Iterables
- These methods should return respectively named transforms: ToString.Elements, 
ToString.KVs, ToString.Iterables



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1425) Window should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1425:
--

 Summary: Window should comply with PTransform style guide
 Key: BEAM-1425
 URL: https://issues.apache.org/jira/browse/BEAM-1425
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes:

- Remove static builder-like methods such as triggering(), 
discardingFiredPanes() - the only static entry point should be .into().
- (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1426) SortValues should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1426:
--

 Summary: SortValues should comply with PTransform style guide
 Key: BEAM-1426
 URL: https://issues.apache.org/jira/browse/BEAM-1426
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Eugene Kirpichov
Assignee: Davor Bonaci


Suggested changes: BufferedExternalSorter.Options should name its builder 
methods .withBlah(), rather than .setBlah().



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)