[1/2] incubator-beam git commit: Fix mvn command args in Apex postcommit Jenkins job

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4323247a3 -> b38c9e9eb


Fix mvn command args in Apex postcommit Jenkins job


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8c00f63c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8c00f63c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8c00f63c

Branch: refs/heads/master
Commit: 8c00f63ca264e73c7ab7cfc760ef7748edbb6f60
Parents: 4323247
Author: Kenneth Knowles 
Authored: Thu Dec 15 10:20:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 10:20:03 2016 -0800

--
 .jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c00f63c/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
--
diff --git a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy 
b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
index 79d12be..232c94e 100644
--- a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
+++ b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
@@ -36,6 +36,6 @@ mavenJob('beam_PostCommit_Java_RunnableOnService_Apex') {
   --also-make \
   --batch-mode \
   --errors \
-  --profile runnable-on-service-tests \
-  --profile local-runnable-on-service-tests''')
+  --activate-profiles runnable-on-service-tests \
+  --activate-profiles local-runnable-on-service-tests''')
 }

[jira] [Commented] (BEAM-1160) Disabling Read transform validation cause empty file patterns to unexpected succeed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752113#comment-15752113
 ] 

ASF GitHub Bot commented on BEAM-1160:
--

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1627

[BEAM-1160] Add option to disable failures if filePattern resolves to empty

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam read-allow-empty-glob

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1627


commit 43195ba27b4303c852e81fa1de493ec4fec641bb
Author: Scott Wegner 
Date:   2016-12-15T18:12:25Z

Add option to disable failures if filePattern resolves to empty

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.




> Disabling Read transform validation cause empty file patterns to unexpected 
> succeed
> ---
>
> Key: BEAM-1160
> URL: https://issues.apache.org/jira/browse/BEAM-1160
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>
> Typically, input file patterns are validated during Pipeline construction, 
> but standard Read transforms include an option to disable validation. This is 
> generally useful but can lead to cases where a Pipeline executes successfully 
> with empty inputs.
> We should fail execution on empty file-based inputs even when validation is 
> disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1627: [BEAM-1160] Add option to disable failure...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1627

[BEAM-1160] Add option to disable failures if filePattern resolves to empty

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam read-allow-empty-glob

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1627


commit 43195ba27b4303c852e81fa1de493ec4fec641bb
Author: Scott Wegner 
Date:   2016-12-15T18:12:25Z

Add option to disable failures if filePattern resolves to empty

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1626: Fix typo in mvn command args in Apex post...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1626

Fix typo in mvn command args in Apex postcommit Jenkins job

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: anyone

Tested that this actually runs so kicking the seed job should fix up the 
postcommit for Apex.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam apex-mvn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1626


commit 8c00f63ca264e73c7ab7cfc760ef7748edbb6f60
Author: Kenneth Knowles 
Date:   2016-12-15T18:20:03Z

Fix mvn command args in Apex postcommit Jenkins job




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1625: Don't incorrectly log error in MetricsEnv...

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1625

Don't incorrectly log error in MetricsEnvironment

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam fix-metrics-warning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1625.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1625


commit 125223f8feb3576d3ff5ccdffa58a5e80808286c
Author: bchambers 
Date:   2016-12-14T19:23:39Z

Don't incorrectly log error in MetricsEnvironment

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #2

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/incubator-beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4323247a3a8ea6fc06b99f66f0001f6956b494c9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4323247a3a8ea6fc06b99f66f0001f6956b494c9
 > git rev-list 4323247a3a8ea6fc06b99f66f0001f6956b494c9 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Established TCP socket on 32896
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Apex] $ 
/home/jenkins/tools/java/latest1.8/bin/java -Xmx2g -Xms256m 
-XX:MaxPermSize=512m -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 32896
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 clean verify --projects runners/apex --also-make --batch-mode --errors 
--profile runnable-on-service-tests --profile local-runnable-on-service-tests
Unable to parse command line options: Unrecognized option: --profile

usage: mvn [options] [] []

Options:
 -am,--also-makeIf project list is specified, also
build projects required by the
list
 -amd,--also-make-dependentsIf project list is specified, also
build projects that depend on
projects on the list
 -B,--batch-modeRun in non-interactive (batch)
mode
 -b,--builder  The id of the build strategy to
use.
 -C,--strict-checksums  Fail the build if checksums don't
match
 -c,--lax-checksums Warn if checksums don't match
 -cpu,--check-plugin-updatesIneffective, only kept for
backward compatibility
 -D,--define   Define a system property
 -e,--errorsProduce execution error messages
 -emp,--encrypt-master-passwordEncrypt master security password
 -ep,--encrypt-passwordEncrypt server password
 -f,--file Force the use of an alternate POM
file (or directory with pom.xml).
 -fae,--fail-at-end Only fail the build afterwards;
allow all non-impacted builds to

[GitHub] incubator-beam pull request #1624: [BEAM-647] Fault-tolerant sideInputs via ...

[kobisalant]

GitHub user kobisalant opened a pull request:

https://github.com/apache/incubator-beam/pull/1624

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kobisalant/incubator-beam 
BEAM-647-Fault-tolerant-sideInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 228c613c70c4edf1b01294a487d49f0a5492136e
Author: ksalant 
Date:   2016-12-15T17:42:47Z

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-647) Fault-tolerant sideInputs via Broadcast variables

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752015#comment-15752015
 ] 

ASF GitHub Bot commented on BEAM-647:
-

GitHub user kobisalant opened a pull request:

https://github.com/apache/incubator-beam/pull/1624

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kobisalant/incubator-beam 
BEAM-647-Fault-tolerant-sideInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 228c613c70c4edf1b01294a487d49f0a5492136e
Author: ksalant 
Date:   2016-12-15T17:42:47Z

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables




> Fault-tolerant sideInputs via Broadcast variables
> -
>
> Key: BEAM-647
> URL: https://issues.apache.org/jira/browse/BEAM-647
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Kobi Salant
>
> Following https://github.com/apache/incubator-beam/pull/909 which enables 
> checkpointing to recover from failures, sideInputs (being implemented by 
> broadcast variables) should be handled in a specific manner as described 
> here: 
> http://spark.apache.org/docs/1.6.2/streaming-programming-guide.html#accumulators-and-broadcast-variables.
> This is a bit more complicated than Aggregators (via Accumulators) as they 
> are implemented using a single "aggregating"  Accumulator, while a pipeline 
> may contain multiple sideInputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-647) Fault-tolerant sideInputs via Broadcast variables

[Kobi Salant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kobi Salant reassigned BEAM-647:


Assignee: Kobi Salant  (was: Amit Sela)

> Fault-tolerant sideInputs via Broadcast variables
> -
>
> Key: BEAM-647
> URL: https://issues.apache.org/jira/browse/BEAM-647
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Kobi Salant
>
> Following https://github.com/apache/incubator-beam/pull/909 which enables 
> checkpointing to recover from failures, sideInputs (being implemented by 
> broadcast variables) should be handled in a specific manner as described 
> here: 
> http://spark.apache.org/docs/1.6.2/streaming-programming-guide.html#accumulators-and-broadcast-variables.
> This is a bit more complicated than Aggregators (via Accumulators) as they 
> are implemented using a single "aggregating"  Accumulator, while a pipeline 
> may contain multiple sideInputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #1

[Apache Jenkins Server]

See 


--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/incubator-beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4323247a3a8ea6fc06b99f66f0001f6956b494c9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4323247a3a8ea6fc06b99f66f0001f6956b494c9
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Discovered a new module org.apache.beam:beam-parent Apache Beam :: Parent
Discovered a new module org.apache.beam:beam-sdks-java-build-tools Apache Beam 
:: SDKs :: Java :: Build Tools
Discovered a new module org.apache.beam:beam-sdks-parent Apache Beam :: SDKs
Discovered a new module org.apache.beam:beam-sdks-java-parent Apache Beam :: 
SDKs :: Java
Discovered a new module org.apache.beam:beam-sdks-java-core Apache Beam :: SDKs 
:: Java :: Core
Discovered a new module org.apache.beam:beam-sdks-java-io-parent Apache Beam :: 
SDKs :: Java :: IO
Discovered a new module org.apache.beam:beam-sdks-java-io-google-cloud-platform 
Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
Discovered a new module org.apache.beam:beam-sdks-java-io-hdfs Apache Beam :: 
SDKs :: Java :: IO :: HDFS
Discovered a new module org.apache.beam:beam-sdks-java-io-jms Apache Beam :: 
SDKs :: Java :: IO :: JMS
Discovered a new module org.apache.beam:beam-sdks-java-io-kafka Apache Beam :: 
SDKs :: Java :: IO :: Kafka
Discovered a new module org.apache.beam:beam-sdks-java-io-kinesis Apache Beam 
:: SDKs :: Java :: IO :: Kinesis
Discovered a new module org.apache.beam:beam-sdks-java-io-mongodb Apache Beam 
:: SDKs :: Java :: IO :: MongoDB
Discovered a new module org.apache.beam:beam-sdks-java-io-jdbc Apache Beam :: 
SDKs :: Java :: IO :: JDBC
Discovered a new module org.apache.beam:beam-sdks-java-maven-archetypes-parent 
Apache Beam :: SDKs :: Java :: Maven Archetypes
Discovered a new module org.apache.beam:beam-sdks-java-maven-archetypes-starter 
Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
Discovered a new module 
org.apache.beam:beam-sdks-java-maven-archetypes-examples Apache Beam :: SDKs :: 
Java :: Maven Archetypes :: Examples
Discovered a new module 
org.apache.beam:beam-sdks-java-maven-archetypes-examples-java8 Apache Beam :: 
SDKs :: Java :: Maven Archetypes :: Examples - Java 8
Discovered a new module org.apache.beam:beam-sdks-java-extensions-parent Apache 
Beam :: SDKs :: Java :: Extensions
Discovered a new module org.apache.beam:beam-sdks-java-extensions-join-library 
Apache Beam :: SDKs :: Java :: Extensions :: Join library
Discovered a new module org.apache.beam:beam-sdks-java-extensions-sorter Apache 
Beam :: SDKs :: Java :: Extensions :: Sorter
Discovered a new module org.apache.beam:beam-runners-parent Apache Beam :: 
Runners
Discovered a new module org.apache.beam:beam-runners-core-java Apache Beam :: 
Runners :: Core Java
Discovered a new module org.apache.beam:beam-runners-direct-java Apache Beam :: 
Runners :: Direct Java
Discovered a new module org.apache.beam:beam-runners-flink-parent Apache Beam 
:: Runners :: Flink
Discovered a new module org.apache.beam:beam-runners-flink_2.10 Apache Beam :: 
Runners :: Flink :: Core
Discovered a new module org.apache.beam:beam-runners-flink_2.10-examples Apache 
Beam :: Runners :: Flink :: Examples

[jira] [Created] (BEAM-1163) Add signature keys to the release guide vote template

[JIRA]

Ismaël Mejía created BEAM-1163:
--

 Summary: Add signature keys to the release guide vote template
 Key: BEAM-1163
 URL: https://issues.apache.org/jira/browse/BEAM-1163
 Project: Beam
  Issue Type: Improvement
  Components: website
Affects Versions: Not applicable
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía
Priority: Trivial


A small improvement, the idea is to add just the fingerprint of the person who 
signed the release in the template (for validation purposes:

The release artifacts are signed with the key with fingerprint XXX
https://dist.apache.org/repos/dist/release/incubator/beam/KEYS




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1162) Create RedisIO

[JIRA]

Ismaël Mejía created BEAM-1162:
--

 Summary: Create RedisIO
 Key: BEAM-1162
 URL: https://issues.apache.org/jira/browse/BEAM-1162
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Affects Versions: Not applicable
Reporter: Ismaël Mejía
Assignee: Jean-Baptiste Onofré
Priority: Minor


As discussed in the mailing list there is interest in this IO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-690) Backoff in the DirectRunner Monitor if no work is Available

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751670#comment-15751670
 ] 

Stas Levin commented on BEAM-690:
-

Hi,

I'm trying to look into it, correct me if I'm wrong, but won't having such a 
back-off result in tests taking longer to complete? 

Backing-off the scheduling of {{MonitorRunnable#run}} delays the detection of a 
shut down performed in {{MonitorRunnable#shouldShutdown}}, called by 
{{MonitorRunnable#run}}.

> Backoff in the DirectRunner Monitor if no work is Available
> ---
>
> Key: BEAM-690
> URL: https://issues.apache.org/jira/browse/BEAM-690
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>
> When a Pipeline has no elements available to process, the Monitor Runnable 
> will be repeatedly scheduled. Given that there is no work to be done, this 
> will loop over the steps in the transform looking for timers, and prompt the 
> sources to perform additional work, even though there is no work to be done. 
> This consumes the entirety of a single core.
> Add a bounded backoff to rescheduling the monitor runnable if no work has 
> been done since it last ran. This will reduce resource consumption on 
> low-throughput Pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-932) Findbugs doesn't pass in Spark runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751544#comment-15751544
 ] 

ASF GitHub Bot commented on BEAM-932:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1463


> Findbugs doesn't pass in Spark runner
> -
>
> Key: BEAM-932
> URL: https://issues.apache.org/jira/browse/BEAM-932
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Ismaël Mejía
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-runners-spark 
> ---
> [INFO] BugInstance size is 19
> [INFO] Error size is 0
> [INFO] Total bugs: 19
> [INFO] instanceof will always return false in 
> org.apache.beam.runners.spark.SparkRunner.run(Pipeline), since a 
> RuntimeException can't be a org.apache.spark.SparkException 
> [org.apache.beam.runners.spark.SparkRunner] At SparkRunner.java:[line 161]
> [INFO] 
> org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1.apply(Map$Entry)
>  makes inefficient use of keySet iterator instead of entrySet iterator 
> [org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1]
>  At WithNamedAggregatorsSupport.java:[line 125]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink shadows the 
> simple name of the superclass org.apache.spark.metrics.sink.CsvSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink] At 
> CsvSink.java:[lines 37-38]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink shadows 
> the simple name of the superclass org.apache.spark.metrics.sink.GraphiteSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink] At 
> GraphiteSink.java:[lines 37-38]
> [INFO] t must be non-null but is marked as nullable 
> [org.apache.beam.runners.spark.translation.WindowingHelpers$4] At 
> WindowingHelpers.java:[line 88]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1.evaluate(ConsoleIO$Write$Unbound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1]
>  At StreamingTransformTranslator.java:[line 92]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10.evaluate(ParDo$Bound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10]
>  At StreamingTransformTranslator.java:[line 360]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11.evaluate(ParDo$BoundMulti,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11]
>  At StreamingTransformTranslator.java:[line 395]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$2.evaluate(Read$Unbounded,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$2]
>  At StreamingTransformTranslator.java:[line 104]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$3.evaluate(CreateStream$QueuedValues,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$3]
>  At StreamingTransformTranslator.java:[line 115]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$4.evaluate(Flatten$FlattenPCollectionList,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$4]
>  At 

[GitHub] incubator-beam pull request #1463: [BEAM-932] Enable findbugs validation (an...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1463


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-932] Enable findbugs validation (and fix existing issues)

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1ad638e51 -> 4323247a3


[BEAM-932] Enable findbugs validation (and fix existing issues)


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bba3700a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bba3700a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bba3700a

Branch: refs/heads/master
Commit: bba3700aebd9dcaf88402b6845008cf7e5950cfe
Parents: 1ad638e
Author: Ismaël Mejía 
Authored: Thu Dec 15 14:34:20 2016 +0100
Committer: Ismaël Mejía 
Committed: Thu Dec 15 14:34:20 2016 +0100

--
 runners/spark/pom.xml   | 10 
 .../metrics/WithNamedAggregatorsSupport.java|  4 +-
 .../spark/translation/WindowingHelpers.java |  3 +-
 .../src/main/resources/beam/findbugs-filter.xml | 49 +---
 4 files changed, 47 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index e8fffa2..5a2fe87 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -146,7 +146,6 @@
 
   com.google.code.findbugs
   jsr305
-  1.3.9
 
 
   com.google.guava
@@ -317,15 +316,6 @@
   
 
   
-
-
-  org.codehaus.mojo
-  findbugs-maven-plugin
-  
-true
-  
-
-
 
   org.apache.maven.plugins
   maven-surefire-plugin

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
index 6932ae6..5e71280 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
@@ -121,8 +121,8 @@ public class WithNamedAggregatorsSupport extends 
MetricRegistry {
 final String parentName = entry.getKey();
 final Map gaugeMap = 
Maps.transformEntries(agg.renderAll(), toGauge());
 final Map fullNameGaugeMap = Maps.newLinkedHashMap();
-for (String shortName : gaugeMap.keySet()) {
-  fullNameGaugeMap.put(parentName + "." + shortName, 
gaugeMap.get(shortName));
+for (Map.Entry gaugeEntry : gaugeMap.entrySet()) {
+  fullNameGaugeMap.put(parentName + "." + gaugeEntry.getKey(), 
gaugeEntry.getValue());
 }
 return Maps.filterValues(fullNameGaugeMap, Predicates.notNull());
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
index ec94f3e..0acff71 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
@@ -18,6 +18,7 @@
 
 package org.apache.beam.runners.spark.translation;
 
+import javax.annotation.Nonnull;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.spark.api.java.function.Function;
 
@@ -84,7 +85,7 @@ public final class WindowingHelpers {
   public static  com.google.common.base.Function 
unwindowValueFunction() {
 return new com.google.common.base.Function() {
   @Override
-  public T apply(WindowedValue t) {
+  public T apply(@Nonnull WindowedValue t) {
 return t.getValue();
   }
 };

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml 
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index a696aeb..bfb4988 100644
--- 

[2/2] incubator-beam git commit: This closes #1463

[amitsela]

This closes #1463


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4323247a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4323247a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4323247a

Branch: refs/heads/master
Commit: 4323247a3a8ea6fc06b99f66f0001f6956b494c9
Parents: 1ad638e bba3700
Author: Sela 
Authored: Thu Dec 15 16:16:57 2016 +0200
Committer: Sela 
Committed: Thu Dec 15 16:16:57 2016 +0200

--
 runners/spark/pom.xml   | 10 
 .../metrics/WithNamedAggregatorsSupport.java|  4 +-
 .../spark/translation/WindowingHelpers.java |  3 +-
 .../src/main/resources/beam/findbugs-filter.xml | 49 +---
 4 files changed, 47 insertions(+), 19 deletions(-)
--

[jira] [Reopened] (BEAM-855) Remove the need for --streaming option in the spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela reopened BEAM-855:

  Assignee: (was: Jean-Baptiste Onofré)

> Remove the need for --streaming option in the spark runner
> --
>
> Key: BEAM-855
> URL: https://issues.apache.org/jira/browse/BEAM-855
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-855) Remove the need for --streaming option in the spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-855.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

> Remove the need for --streaming option in the spark runner
> --
>
> Key: BEAM-855
> URL: https://issues.apache.org/jira/browse/BEAM-855
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-757) The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-757:
---
Fix Version/s: (was: 0.4.0-incubating)
   0.5.0-incubating

> The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's 
> own.
> 
>
> Key: BEAM-757
> URL: https://issues.apache.org/jira/browse/BEAM-757
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.5.0-incubating
>
>
> The SDK now provides DoFnRunner implementations, and so to avoid maintaining 
> against the SDK, the runner should leverage the runner API instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-807) [SparkRunner] Replace OldDoFn with DoFn

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-807:
---
Fix Version/s: (was: 0.4.0-incubating)
   0.5.0-incubating

> [SparkRunner] Replace OldDoFn with DoFn
> ---
>
> Key: BEAM-807
> URL: https://issues.apache.org/jira/browse/BEAM-807
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-855) Remove the need for --streaming option in the spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-855:
---
Fix Version/s: (was: 0.4.0-incubating)

> Remove the need for --streaming option in the spark runner
> --
>
> Key: BEAM-855
> URL: https://issues.apache.org/jira/browse/BEAM-855
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #110: Update website pull request instructi...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam-site/pull/110

Update website pull request instructions in the Release Guide



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam-site 
RELEASE_GUIDE_WEBSITE_PR_UPDATE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/110.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #110


commit 8ddeada95adafa0949da8313ff6ab1f780a3a493
Author: Jean-Baptiste Onofré 
Date:   2016-12-15T11:58:30Z

Update website pull request instructions in the Release Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #108: Exclude apex runner translation packa...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam-site/pull/108

Exclude apex runner translation package from the generated javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam-site 
RELEASE_GUIDE_UPDATE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #108


commit c7ac43e62f01950614dc7cc1ec5137cb2bd7a8e6
Author: Jean-Baptiste Onofré 
Date:   2016-12-15T09:35:32Z

Update Release Guide: exclude apex runner translation package from the 
generated javadoc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---