date:20180617

[jira] [Work logged] (BEAM-2810) Consider a faster Avro library in Python

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2810?focusedWorklogId=112685=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112685
 ]

ASF GitHub Bot logged work on BEAM-2810:


Author: ASF GitHub Bot
Created on: 18/Jun/18 05:00
Start Date: 18/Jun/18 05:00
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5496: do not merge! 
[BEAM-2810] use fastavro in Avro IO
URL: https://github.com/apache/beam/pull/5496#issuecomment-397942365
 
 
   OK, I think this is ready for a proper review!
   
   - [x] block-iteration code [merged into 
fastavro](https://github.com/fastavro/fastavro/pull/208) and released in 
[0.19.7](https://github.com/fastavro/fastavro/releases/tag/0.19.7)
   - [x] fastavro vs apache/avro is configurable in `avroio.py` via 
`use_fastavro` argument to relevant `PTransform`s 
([`ReadFromAvro`](https://github.com/apache/beam/pull/5496/files#diff-04fef9e0550df0b0c4e1cd0264406eb5R73),
 
[`WriteToAvro`](https://github.com/apache/beam/pull/5496/files#diff-04fef9e0550df0b0c4e1cd0264406eb5R464),
 etc; default: `False`).
   - [x] `avroio_test` runs all tests against 
[apache/avro](https://github.com/apache/beam/pull/5496/files#diff-5282dd5fac1c35c3a7b556447eb694aaR52)
 and 
[fastavro](https://github.com/apache/beam/pull/5496/files#diff-5282dd5fac1c35c3a7b556447eb694aaR447)
   - [x] significant speed boost (**4-5x**) demonstrated in an example pipeline 
(discussion below)
   
   ## Example pipeline: 
[`sdks/python/examples/avro_bitcoin.py`](https://github.com/apache/beam/pull/5496/files#diff-3d963380bd7941037fba4ef3932a0cec)
   
   - I exported [the `bigquery-public-data:bitcoin_blockchain.transactions` 
table](https://bigquery.cloud.google.com/table/bigquery-public-data:bitcoin_blockchain.transactions)
 to public Avro files at 
[`gs://beam-avro-test/bitcoin/txns`](https://console.cloud.google.com/storage/browser/beam-avro-test/bitcoin/txns)
   - I ran the `avro_bitcoin` example pipeline (using DataflowRunner) on 
{fastavro,apache/avro} x {compressed,uncompressed}:
 - for example:
   
   ```bash
   python \
 -m apache_beam.examples.bitcoin \
 --runner DataflowRunner \
 --project  \
 --temp_location gs:/// \
 --sdk_location $PWD/python/dist/apache-beam-2.6.0.dev0.tar.gz \
 --output gs://beam-avro-test/bitcoin/txn-counts/fastavro-compressed \
 --fastavro \
 --compressed
   ```
   
 - outputs can be found at 
[`gs://beam-avro-test/bitcoin/txn-counts`](https://console.cloud.google.com/storage/browser/beam-avro-test/bitcoin/txn-counts/)
   
   
   ### Performance Measurements
   
   | Run | Elapsed time | Workers | vCPU-hrs | mem (GB-hrs) | PD time |
   | -- | -- | -- | -- | -- | -- |
   | fastavro (compressed) | 11m31s | 1 | 0.147 | 0.553 | 36.8 |
   | fastavro (uncompressed) | 11m9s | 1 | 0.149 | 0.558 | 37.2 |
   | apache/avro (compressed) | 17m15s | 1→4 | 0.643 | 2.413 | 160.9 |
   | apache/avro (uncompressed) | 17m30s | 1→4 | 0.684 | 2.566 | 171.0 |
   
   The collected metrics were the same in all cases, but the apache/avro 
outputs had 9 shards where the fastavro outputs had 5, I'm guessing due to the 
former having used up to 4 workers where the latter used 1?
   
   ### Relevant screenshots from job-pages
   
    fastavro compressed:
   
 ![fastavro 
compressed](https://cl.ly/2q1u2b1I381E/Screen%20Shot%202018-06-17%20at%2011.59.15%20PM.png)
 
   
    fastavro uncompressed:
   
 ![fastavro 
uncompressed](https://cl.ly/3D1y2p2L242A/Screen%20Shot%202018-06-17%20at%2011.59.39%20PM.png)
 
   
    apache/avro compressed:
   
 ![apache 
compressed](https://cl.ly/1c093X0n440t/Screen%20Shot%202018-06-18%20at%2012.00.09%20AM.png)
 
   
    apache/avro uncompressed:
   
 ![apache 
uncompressed](https://cl.ly/2C263E3S3w2v/Screen%20Shot%202018-06-18%20at%2012.00.48%20AM.png)
   
   ## Open Questions
   - I'd discussed adding an integration test of this functionality with 
@chamikaramj, but I'm thinking the example pipeline (and off-by-default flag) 
may be enough to make us comfortable w this feature; open to others' thoughts 
there.
   - Is there a more appropriate bucket that the test-data should live in?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112685)
Time Spent: 1.5h  (was: 1h 20m)

> Consider a faster Avro library in Python
> 
>
> Key: BEAM-2810
> URL: https://issues.apache.org/jira/browse/BEAM-2810
> Project: Beam
>

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112680
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 04:26
Start Date: 18/Jun/18 04:26
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397938913
 
 
   +R: @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112680)
Time Spent: 57.5h  (was: 57h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 57.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #523

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 54.42 MB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
java.lang.AssertionError: OutputSideInputs/ParMultiDo(Anonymous).output: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.testing.PAssert$DefaultConcludeFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn.processElement(PAssert.java:1215)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:42)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:131)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn.processElement(PAssert.java:1049)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at

Jenkins build is back to normal : beam_Dependency_Check #96

2018-06-17 Thread Apache Jenkins Server

See

[beam] 01/01: [Gradle Release Plugin] - pre tag commit: 'v2.5.0-RC2'.

2018-06-17 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to annotated tag v2.5.0-RC2
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 29bf6b7ef7092dc11385af935a60bfad2db1a531
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Jun 16 11:29:35 2018 +0200

[Gradle Release Plugin] - pre tag commit:  'v2.5.0-RC2'.
---
 gradle.properties | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gradle.properties b/gradle.properties
index a91170d..e0fdb00 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -22,4 +22,4 @@ offlineRepositoryRoot=offline-repository
 signing.gnupg.executable=gpg
 signing.gnupg.useLegacyGpg=true
 
-version=2.5.0-SNAPSHOT
+version=2.5.0-RC2

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] annotated tag v2.5.0-RC2 created (now 2fe5438)

2018-06-17 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to annotated tag v2.5.0-RC2
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at 2fe5438  (tag)
 tagging 29bf6b7ef7092dc11385af935a60bfad2db1a531 (commit)
 replaces v2.2.0
  by Jean-Baptiste Onofré
  on Sat Jun 16 11:29:35 2018 +0200

- Log -
[Gradle Release Plugin] - creating tag:  'v2.5.0-RC2'.
---

This annotated tag includes the following new commits:

 new 29bf6b7  [Gradle Release Plugin] - pre tag commit:  'v2.5.0-RC2'.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112679
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 04:18
Start Date: 18/Jun/18 04:18
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397938161
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112679)
Time Spent: 57h 20m  (was: 57h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 57h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112677
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 04:14
Start Date: 18/Jun/18 04:14
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397937760
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112677)
Time Spent: 57h 10m  (was: 57h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 57h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112676
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 04:09
Start Date: 18/Jun/18 04:09
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397937317
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112676)
Time Spent: 57h  (was: 56h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 57h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_Dependency_Check #95

2018-06-17 Thread Apache Jenkins Server

See 

--
GitHub pull request #5406 of commit c64916ec67185e7d4225f0333e881172f5c61bbe, 
no merge conflicts.
Setting status of c64916ec67185e7d4225f0333e881172f5c61bbe to PENDING with url 
https://builds.apache.org/job/beam_Dependency_Check/95/ and message: 'Build 
started sha1 is merged.'
Using context: Jenkins: Beam Dependency Check
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5406/*:refs/remotes/origin/pr/5406/*
 > git rev-parse refs/remotes/origin/pr/5406/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5406/merge^{commit} # timeout=10
Checking out Revision 19ad7bf2a1b06d8ff81e82aa0db91cb68899a75c 
(refs/remotes/origin/pr/5406/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 19ad7bf2a1b06d8ff81e82aa0db91cb68899a75c
Commit message: "Merge c64916ec67185e7d4225f0333e881172f5c61bbe into 
6af4561d32ae96d60c4aa3257ec56494cb9be154"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
No emails were triggered.
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_Dependency_Check] $ /bin/bash -xe /tmp/jenkins3211009723222995981.sh
+ cd src
+ bash .test-infra/jenkins/dependency_check/generate_report.sh

PROJECT_ID='apache-beam-testing'
DATASET_ID='beam_dependency_states'
PYTHON_DEP_TABLE_ID='python_dependency_states'
JAVA_DEP_TABLE_ID='java_dependency_states'
REPORT_DESCRIPTION="
 In Beam, we make a best-effort attempt at keeping all dependencies 
up-to-date. 
 A dependency update is high priority if it satisfies one of following 
criteria: 

 It has major versions update available;
  
 e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0 
  



 It is over 3 minor versions behind the latest version;
  
 e.g. org.tukaani:xz 1.5 -> 1.8 
  



 The current version is behind the later version for over 180 days.
  
 e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11 

  



 In the future, issues will be filed and tracked for these automatically, 
but in the meantime you can search for existing issues or open a new one. 
 Read more about our dependency update policy: https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit#\;>
 Beam Dependency Update Policy "


# Virtualenv for the rest of the script to run setup
/usr/bin/virtualenv dependency/check
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
. dependency/check/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if !

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112672
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 03:29
Start Date: 18/Jun/18 03:29
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397933363
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112672)
Time Spent: 56h 50m  (was: 56h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 56h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4325) Enforce ErrorProne analysis in the SQL project

2018-06-17 Thread Cade Markegard (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515319#comment-16515319
 ] 

Cade Markegard commented on BEAM-4325:
--

[~vectorijk] thanks for reaching out, I've made progress on fixing mostly 
everything just working through how to unravel this: 
[https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlExpression.java#L53]
 without just placing a SuppressWarning for 
[http://errorprone.info/bugpattern/TypeParameterUnusedInFormals] .

> Enforce ErrorProne analysis in the SQL project
> --
>
> Key: BEAM-4325
> URL: https://issues.apache.org/jira/browse/BEAM-4325
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
> # In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} 
> to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112671
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 03:22
Start Date: 18/Jun/18 03:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397932658
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112671)
Time Spent: 56h 40m  (was: 56.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 56h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112670
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 03:06
Start Date: 18/Jun/18 03:06
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397931100
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112670)
Time Spent: 56.5h  (was: 56h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 56.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112669
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 02:40
Start Date: 18/Jun/18 02:40
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397928441
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112669)
Time Spent: 56h 20m  (was: 56h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 56h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #798

2018-06-17 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-4325) Enforce ErrorProne analysis in the SQL project

2018-06-17 Thread Kai Jiang (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515288#comment-16515288
 ] 

Kai Jiang commented on BEAM-4325:
-

[~cademarkegard] Any process on this one? Let me know if you have any questions.

> Enforce ErrorProne analysis in the SQL project
> --
>
> Key: BEAM-4325
> URL: https://issues.apache.org/jira/browse/BEAM-4325
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
> # In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} 
> to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4292) Add streaming wordcount example

2018-06-17 Thread James Wilson (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515275#comment-16515275
 ] 

James Wilson commented on BEAM-4292:


Thanks, I'll keep the 3rd one in mind.  I should have several solid nights this 
week to work on this and can hopefully get it all up and running.

> Add streaming wordcount example
> ---
>
> Key: BEAM-4292
> URL: https://issues.apache.org/jira/browse/BEAM-4292
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: James Wilson
>Priority: Major
>
> It is referenced on the Beam website as part of the Wordcount progression:
> https://beam.apache.org/get-started/wordcount-example/#streamingwordcount-example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112664
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 18/Jun/18 01:06
Start Date: 18/Jun/18 01:06
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397919514
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112664)
Time Spent: 56h 10m  (was: 56h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 56h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #522

2018-06-17 Thread Apache Jenkins Server

See 


Changes:

[thw] Fix ValidatesRunner_Apex #5604

--
[...truncated 54.98 MB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
java.lang.AssertionError: OutputSideInputs/ParMultiDo(Anonymous).output: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.testing.PAssert$DefaultConcludeFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn.processElement(PAssert.java:1215)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:42)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:131)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn.processElement(PAssert.java:1049)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=112663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112663
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 17/Jun/18 23:36
Start Date: 17/Jun/18 23:36
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-397914030
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112663)
Time Spent: 56h  (was: 55h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 56h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1682

2018-06-17 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6af4561d32ae96d60c4aa3257ec56494cb9be154 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6af4561d32ae96d60c4aa3257ec56494cb9be154
Commit message: "Merge pull request #5666: Fix ValidatesRunner_Apex #5604"
 > git rev-list --no-walk 6af4561d32ae96d60c4aa3257ec56494cb9be154 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins7780963483394217457.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user amaliu...@163.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user ringw...@google.com
Not sending mail to unregistered user lukasz.gaj...@polidea.com
Not sending mail to unregistered user dariusz.aniszew...@polidea.com
Not sending mail to unregistered user amyrv...@google.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user eh...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user boyu...@google.com
Not sending mail to unregistered user timrobertson...@gmail.com
Not sending mail to unregistered user 
yifan...@yifanzou-linuxworkstation.sea.corp.google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user cademarkeg...@gmail.com
Not sending mail to unregistered user amm...@google.com
Not sending mail to unregistered user 
robi...@rodete-desktop-imager.corp.google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user samu...@google.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user jiang...@gmail.com
Not sending mail to unregistered user robbe.sneyd...@ml6.eu
Not sending mail to unregistered user ajam...@ajamato-linux0.sea.corp.google.com
Not sending mail to unregistered user schroed...@google.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user fvoich...@gmail.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user aromanenko@gmail.com
Not sending mail to unregistered user jhs...@opengov.com
Not sending mail to unregistered user kpeter...@nestlabs.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user ke...@google.com

[jira] [Work logged] (BEAM-3998) Futurize and fix python 2 compatibility for examples subpackage

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3998?focusedWorklogId=112660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112660
 ]

ASF GitHub Bot logged work on BEAM-3998:


Author: ASF GitHub Bot
Created on: 17/Jun/18 21:51
Start Date: 17/Jun/18 21:51
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5652: 
[BEAM-3998] Futurize examples subpackage
URL: https://github.com/apache/beam/pull/5652#discussion_r195941921
 
 

 ##
 File path: sdks/python/apache_beam/examples/snippets/snippets.py
 ##
 @@ -1048,7 +1055,7 @@ def to_entity(content):
 entity = entity_pb2.Entity()
 googledatastore.helper.add_key_path(entity.key, kind, str(uuid.uuid4()))
 googledatastore.helper.add_properties(entity,
-  {'content': six.text_type(content)})
+  {'content': unicode(content)})
 
 Review comment:
   I removed all the remaining `six` imports with the latest commit. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112660)
Time Spent: 1h 20m  (was: 1h 10m)

> Futurize and fix python 2 compatibility for examples subpackage
> ---
>
> Key: BEAM-3998
> URL: https://issues.apache.org/jira/browse/BEAM-3998
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3479) Add a regression test for the DoFn classloader selection

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3479?focusedWorklogId=112659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112659
 ]

ASF GitHub Bot logged work on BEAM-3479:


Author: ASF GitHub Bot
Created on: 17/Jun/18 21:51
Start Date: 17/Jun/18 21:51
Worklog Time Spent: 10m 
  Work Description: stale[bot] closed pull request #4412: [BEAM-3479] 
adding a test to ensure the right classloader is used to defined the dofninvoker
URL: https://github.com/apache/beam/pull/4412
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
index 91d9de70c3a..6aaccb500aa 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
@@ -20,6 +20,7 @@
 import static com.google.common.base.Preconditions.checkArgument;
 import static org.apache.beam.sdk.util.common.ReflectHelpers.findClassLoader;
 
+import com.google.common.annotations.VisibleForTesting;
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
 import java.lang.reflect.Method;
@@ -95,6 +96,9 @@
   public static final String STATE_PARAMETER_METHOD = "state";
   public static final String TIMER_PARAMETER_METHOD = "timer";
 
+  @VisibleForTesting
+  static final String PROXY_CLASSNAME_SUFFIX = 
DoFnInvoker.class.getSimpleName();
+
   /**
* Returns a {@link ByteBuddyDoFnInvokerFactory} shared with all other 
invocations, so that its
* cache of generated classes is global.
@@ -284,7 +288,7 @@ public static RestrictionTracker invokeNewTracker(Object 
restriction) {
 // private and package-private bits
 .with(
 StableInvokerNamingStrategy.forDoFnClass(fnClass)
-.withSuffix(DoFnInvoker.class.getSimpleName()))
+.withSuffix(PROXY_CLASSNAME_SUFFIX))
 
 // class  extends DoFnInvokerBase {
 .subclass(DoFnInvokerBase.class, 
ConstructorStrategy.Default.NO_CONSTRUCTORS)
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
index 42b9381bc42..7c3550c3003 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
@@ -20,6 +20,7 @@
 import static com.google.common.base.MoreObjects.firstNonNull;
 
 import com.google.auto.value.AutoValue;
+import com.google.common.annotations.VisibleForTesting;
 import javax.annotation.Nullable;
 import net.bytebuddy.NamingStrategy;
 import net.bytebuddy.description.type.TypeDescription;
@@ -31,6 +32,9 @@
  */
 @AutoValue
 abstract class StableInvokerNamingStrategy extends NamingStrategy.AbstractBase 
{
+  /** $ is for a nested class so use as most proxying framework $$. */
+  @VisibleForTesting
+  static final Object PROXY_NAME_DELIMITER = "$$";
 
   public abstract Class> getFnClass();
 
@@ -48,7 +52,9 @@ public StableInvokerNamingStrategy withSuffix(String 
newSuffix) {
   @Override
   protected String name(TypeDescription superClass) {
 return String.format(
-"%s$%s",
-getFnClass().getName(), firstNonNull(getSuffix(), 
superClass.getName().replace(".", "_")));
+"%s%s%s",
+getFnClass().getName(),
+PROXY_NAME_DELIMITER,
+firstNonNull(getSuffix(), superClass.getName().replace(".", "_")));
   }
 }
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactoryTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactoryTest.java
new file mode 100644
index 000..2ce5c005ae8
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactoryTest.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a

[jira] [Work logged] (BEAM-3479) Add a regression test for the DoFn classloader selection

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3479?focusedWorklogId=112658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112658
 ]

ASF GitHub Bot logged work on BEAM-3479:


Author: ASF GitHub Bot
Created on: 17/Jun/18 21:51
Start Date: 17/Jun/18 21:51
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #4412: [BEAM-3479] adding 
a test to ensure the right classloader is used to defined the dofninvoker
URL: https://github.com/apache/beam/pull/4412#issuecomment-397908943
 
 
   This pull request has been closed due to lack of activity. If you think that 
is incorrect, or the pull request requires review, you can revive the PR at any 
time.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112658)
Time Spent: 2h 50m  (was: 2h 40m)

> Add a regression test for the DoFn classloader selection
> 
>
> Key: BEAM-3479
> URL: https://issues.apache.org/jira/browse/BEAM-3479
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Follow up task after https://github.com/apache/beam/pull/4235 merge. This 
> task is about ensuring we test that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #521

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 54.50 MB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
java.lang.AssertionError: OutputSideInputs/ParMultiDo(Anonymous).output: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.testing.PAssert$DefaultConcludeFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn.processElement(PAssert.java:1215)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:42)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:131)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn.processElement(PAssert.java:1049)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at

[jira] [Work logged] (BEAM-3998) Futurize and fix python 2 compatibility for examples subpackage

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3998?focusedWorklogId=112655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112655
 ]

ASF GitHub Bot logged work on BEAM-3998:


Author: ASF GitHub Bot
Created on: 17/Jun/18 21:25
Start Date: 17/Jun/18 21:25
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5652: 
[BEAM-3998] Futurize examples subpackage
URL: https://github.com/apache/beam/pull/5652#discussion_r195941921
 
 

 ##
 File path: sdks/python/apache_beam/examples/snippets/snippets.py
 ##
 @@ -1048,7 +1055,7 @@ def to_entity(content):
 entity = entity_pb2.Entity()
 googledatastore.helper.add_key_path(entity.key, kind, str(uuid.uuid4()))
 googledatastore.helper.add_properties(entity,
-  {'content': six.text_type(content)})
+  {'content': unicode(content)})
 
 Review comment:
   I removed all the remaining ´six´ imports with the latest commit. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112655)
Time Spent: 1h 10m  (was: 1h)

> Futurize and fix python 2 compatibility for examples subpackage
> ---
>
> Key: BEAM-3998
> URL: https://issues.apache.org/jira/browse/BEAM-3998
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #761

2018-06-17 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=112654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112654
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 17/Jun/18 20:39
Start Date: 17/Jun/18 20:39
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-397904986
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112654)
Time Spent: 23h 20m  (was: 23h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 23h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #797

2018-06-17 Thread Apache Jenkins Server

See 


Changes:

[thw] Fix ValidatesRunner_Apex #5604

--
[...truncated 27.01 MB...]
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager - 
Finished task 3.0 in stage 436.0 (TID 409) in 17 ms on localhost (executor 
driver) (2/5)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSetManager - 
Finished task 2.0 in stage 436.0 (TID 408) in 19 ms on localhost (executor 
driver) (3/5)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSetManager - 
Finished task 1.0 in stage 436.0 (TID 407) in 19 ms on localhost (executor 
driver) (4/5)
[Executor task launch worker for task 410] INFO 
org.apache.spark.executor.Executor - Finished task 4.0 in stage 436.0 (TID 
410). 59466 bytes result sent to driver
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Finished task 4.0 in stage 436.0 (TID 410) in 17 ms on localhost (executor 
driver) (5/5)
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSchedulerImpl - 
Removed TaskSet 436.0, whose tasks have all completed, from pool 
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
ShuffleMapStage 436 (mapToPair at GroupCombineFunctions.java:59) finished in 
0.045 s
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
looking for newly runnable stages
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
running: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
waiting: Set(ResultStage 438)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
failed: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
Submitting ResultStage 438 (MapPartitionsRDD[2458] at map at 
TranslationUtils.java:129), which has no missing parents
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore 
- Block broadcast_95 stored as values in memory (estimated size 199.6 KB, free 
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore 
- Block broadcast_95_piece0 stored as bytes in memory (estimated size 50.6 KB, 
free 13.5 GB)
[dispatcher-event-loop-1] INFO org.apache.spark.storage.BlockManagerInfo - 
Added broadcast_95_piece0 in memory on localhost:42659 (size: 50.6 KB, free: 
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.SparkContext - Created 
broadcast 95 from broadcast at DAGScheduler.scala:1039
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
Submitting 4 missing tasks from ResultStage 438 (MapPartitionsRDD[2458] at map 
at TranslationUtils.java:129) (first 15 tasks are for partitions Vector(0, 1, 
2, 3))
[dag-scheduler-event-loop] INFO 
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 438.0 with 4 
tasks
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 0.0 in stage 438.0 (TID 411, localhost, executor driver, 
partition 0, PROCESS_LOCAL, 8248 bytes)
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 1.0 in stage 438.0 (TID 412, localhost, executor driver, 
partition 1, PROCESS_LOCAL, 8248 bytes)
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 2.0 in stage 438.0 (TID 413, localhost, executor driver, 
partition 2, PROCESS_LOCAL, 8248 bytes)
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 3.0 in stage 438.0 (TID 414, localhost, executor driver, 
partition 3, PROCESS_LOCAL, 8248 bytes)
[Executor task launch worker for task 411] INFO 
org.apache.spark.executor.Executor - Running task 0.0 in stage 438.0 (TID 411)
[Executor task launch worker for task 412] INFO 
org.apache.spark.executor.Executor - Running task 1.0 in stage 438.0 (TID 412)
[Executor task launch worker for task 414] INFO 
org.apache.spark.executor.Executor - Running task 3.0 in stage 438.0 (TID 414)
[Executor task launch worker for task 413] INFO 
org.apache.spark.executor.Executor - Running task 2.0 in stage 438.0 (TID 413)
[Executor task launch worker for task 414] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 414] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 414] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2126_3 locally
[Executor task launch worker for task 414] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2441_3 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2441_3 in

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1681

2018-06-17 Thread Apache Jenkins Server

See 


Changes:

[kpeterson] Allow a GCP project to be explicitly set for a load job, instead of

[aljoscha.krettek] Move _CreateSource to its own package to make it pickelable

[github] Undefined names: import MetricKey, MetricName

[herohde] Add Go integration tests

[matthias] Futurize metrics subpackage

[robbe.sneyders] Add py27-lint3 test to gradle.build

[timrobertson100] Fix error-prone warnings for io/hadoop-common

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[timrobertson100] Fix error-prone and some static analysis warnings in

[kpeterson] Adding a ValueProvider interface.

[kpeterson] Fixing checkstyle error.

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[ringwalt] Improve the TaggedOutput error when a tuple is passed as a tag.

[lukasz.gajowy] [BEAM-214] Add ParquetIO

[lukasz.gajowy] [BEAM-214] Add gradle build in ParquetIO

[lukasz.gajowy] [BEAM-214] Refactore the reads to leverage FileIO

[lukasz.gajowy] [BEAM-214] Use Parquet 1.10.0 version in ParquetIO.Sink

[lukasz.gajowy] [BEAM-214] Add ParquetIOIT

[lukasz.gajowy] [BEAM-214] Update ReadFn to an non deprecated, newest Parquet 
API

[lukasz.gajowy] [BEAM-214] Simplify the IO by removing redundant PTransforms

[lukasz.gajowy] [BEAM-214] Implement writing slices of bytes in BeamOutputStream

[apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset

[Pablo] [BEAM-4331] Disable integrationTest task caching

[tgroh] Add a new DockerEnvironmentFactory Constructor

[Pablo] Fixing Seed Job

[samuelw] [BEAM-3776] Fix issue with merging late windows where a watermark hold

[Pablo] Add future dependency to the Dockerfile

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[github] Mention adding a reviewer in PR template. (#5397)

[kedin] Fix Maven build

[rober] Update generated protos in Go SDK

[herohde] Invoke Go Dataflow integration tests from post-commit

[herohde] Add the Go SDK to the README

[timrobertson100] [BEAM-4342] Enforce ErrorProne analysis in hadoop IO, and 
static

[timrobertson100] [BEAM-4340] Enforce ErrorProne analysis in file-based-io-tests

[timrobertson100] [BEAM-4341] Enforce ErrorProne analysis in 
google-cloud-platform IO

[timrobertson100] [BEAM-4335] Enforce ErrorProne analysis in 
amazon-web-services IO

[timrobertson100] [BEAM-4339] Enforce ErrorProne analysis in elasticsearch IO

[timrobertson100] [BEAM-4355] Enforce ErrorProne analysis in XML IO

[timrobertson100] [BEAM-4338] Enforce ErrorProne analysis in common IO

[timrobertson100] [BEAM-4337] Enforce ErrorProne analysis in cassandra IO

[timrobertson100] [BEAM-4355] Reduces scope of findbugs annotations to build 
time only

[timrobertson100] [BEAM-4353] Enforce ErrorProne analysis in solr IO

[timrobertson100] [BEAM-4345] Enforce ErrorProne analysis in JDBC IO

[timrobertson100] [BEAM-4336] Enforce ErrorProne analysis in AMQP IO

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[dariusz.aniszewski] [BEAM-4041] Increase timeout for getting K8s LoadBalancer 
external IP

[timrobertson100] [BEAM-4346] Enforce ErrorProne analysis in JMS IO

[timrobertson100] [BEAM-4352] Enforce ErrorProne analysis in Redis IO

[lcwik] [BEAM-4271] Support side inputs for ExecutableStage and provide runner

[daniel.o.programmer] [BEAM-2937] Add new Combine URNs.

[tgroh] Update worker_id Documentation

[robbe.sneyders] Futurize internal subpackage

[dariusz.aniszewski] fixed typo

[mariagh] Add validation for streaming wordcount (streaming_wordcount_debugging)

[Pablo] Increasing the concurrent test execution count (#5408)

[tgroh] Add a Default RemoteEnvironment Implementation

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

[tgroh] Add a Window-only State Context

[lcwik] Address Ben's comments.

[github] Make hash function in Coder base class more conservative.

[github] updated right documentation for HCatalogIO.write

[sidhom] Fix DataStreamsTest checkstyle error

[ankurgoenka] Updating python container version to beam-master-20180518

[wcn] Fix the test to properly run.

[ankurgoenka] Reverting fnapi container version

[ankurgoenka] Copyting dependency.py and dependency_test.py

[ankurgoenka] reformatting code

[ankurgoenka] Introducing classes

[ankurgoenka] gruping public and non public methods

[ankurgoenka] Renaming method and updating reference for file_copy and 
file_download

[ankurgoenka] Removing Google specific code and unused code in stager.

[beam] branch master updated (e9cfd3e -> 6af4561)

2018-06-17 Thread iemejia

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e9cfd3e  Merge pull request #5665: Fix Maven build 7
 add 19e9c15  Fix ValidatesRunner_Apex #5604
 new 6af4561  Merge pull request #5666: Fix ValidatesRunner_Apex #5604

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/apex/build.gradle | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] 01/01: Merge pull request #5666: Fix ValidatesRunner_Apex #5604

2018-06-17 Thread iemejia

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6af4561d32ae96d60c4aa3257ec56494cb9be154
Merge: e9cfd3e 19e9c15
Author: Ismaël Mejía 
AuthorDate: Sun Jun 17 22:07:31 2018 +0200

Merge pull request #5666: Fix ValidatesRunner_Apex #5604

 runners/apex/build.gradle | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Updated] (BEAM-2661) Add KuduIO

2018-06-17 Thread Tim Robertson (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson updated BEAM-2661:

Description: 
New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
design aspects documented below.
h2. The API

The {{KuduIO}} API requires the user to provide a function to convert objects 
into operations. This is similar to the {{JdbcIO}} but different to others, 
such as {{HBaseIO}} which requires a pre-transform stage beforehand to convert 
into the mutations to apply. It was originally intended to copy the {{HBaseIO}} 
approach, but this was not possible:
 # The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
 # An alternative could be to introduce a new object to beam (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
 ## It is not a familiar API to those already knowing Kudu
 ## It still requires serialization and deserialization of the operations. 
Using the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e].
 This is possible but has fragilities given the Kudu code itself continues to 
evolve. 
 ## It becomes a trivial codebase in Beam to maintain by defer the object to 
mutation mapping to within the KuduIO transform. {{JdbcIO}} gives us the 
precedent to do this.

h2. Testing framework

{{Kudu}} is written in C++. While a 
[TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
 does exist in Java, it requires binaries to be available for the target 
environment which is not portable. Therefore we opt for the following:
 # Unit tests will use a mock Kudu client
 # Integration tests will cover the full aspects of the {{KuduIO}} and use a 
Docker based Kudu instance

  was:
New IO for Apache Kudu (https://kudu.apache.org/overview.html).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
design aspects documented below.

h2. The API

The {{KuduIO}} API requires the user to provide a function to convert objects 
into operations. This is similar to the {{JdbcIO}} but different to others, 
such as {{HBaseIO}} which requires a pre-transform stage beforehand to convert 
into the mutations to apply. It was originally intended to copy the {{HBaseIO}} 
approach, but this was not possible:

# The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
# An alternative could be to introduce a new object to beam  (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
## It is not a familiar API to those already knowing Kudu
## It still requires serialization and deserialization of the operations. Using 
the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e]

h2. Testing framework

{{Kudu}} is written in C++. While a 
[TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
 does exist in Java, it requires binaries to be available for the target 
environment which is not portable.  Therefore we opt for the following:

# Unit tests will use a mock Kudu client 
# Integration tests will cover the full aspects of the {{KuduIO}} and use a 
Docker based Kudu instance



> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: https://issues.apache.org/jira/browse/BEAM-2661
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Tim Robertson
>Priority: Major
>
> New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).
> This work is in progress [on this 
>

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=112652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112652
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 17/Jun/18 20:00
Start Date: 17/Jun/18 20:00
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-397902561
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112652)
Time Spent: 23h 10m  (was: 23h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 23h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2661) Add KuduIO

2018-06-17 Thread Tim Robertson (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson updated BEAM-2661:

Description: 
New IO for Apache Kudu (https://kudu.apache.org/overview.html).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
design aspects documented below.

h2. The API

The {{KuduIO}} API requires the user to provide a function to convert objects 
into operations. This is similar to the {{JdbcIO}} but different to others, 
such as {{HBaseIO}} which requires a pre-transform stage beforehand to convert 
into the mutations to apply. It was originally intended to copy the {{HBaseIO}} 
approach, but this was not possible:

# The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
# An alternative could be to introduce a new object to beam  (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
## It is not a familiar API to those already knowing Kudu
## It still requires serialization and deserialization of the operations. Using 
the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e]

h2. Testing framework

{{Kudu}} is written in C++. While a 
[TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
 does exist in Java, it requires binaries to be available for the target 
environment which is not portable.  Therefore we opt for the following:

# Unit tests will use a mock Kudu client 
# Integration tests will cover the full aspects of the {{KuduIO}} and use a 
Docker based Kudu instance


  was:
New IO for Apache Kudu (https://kudu.apache.org/overview.html).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO].

Design aspects are documented below.

The API
# The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
# An alternative could be to introduce a new object to beam  (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
## It is not a familiar API to those already knowing Kudu
## It still requires serialization and deserialization of the operations. Using 
the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e]





> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: https://issues.apache.org/jira/browse/BEAM-2661
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Tim Robertson
>Priority: Major
>
> New IO for Apache Kudu (https://kudu.apache.org/overview.html).
> This work is in progress [on this 
> branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
> design aspects documented below.
> h2. The API
> The {{KuduIO}} API requires the user to provide a function to convert objects 
> into operations. This is similar to the {{JdbcIO}} but different to others, 
> such as {{HBaseIO}} which requires a pre-transform stage beforehand to 
> convert into the mutations to apply. It was originally intended to copy the 
> {{HBaseIO}} approach, but this was not possible:
> # The Kudu 
> [Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
>  is a fat class, and is a subclass of {{KuduRpc}}. It 
> holds RPC logic, callbacks and a Kudu client. Because of this the 
> {{Operation}} does not serialize and furthermore, the logic for encoding the 
> operations (Insert, Upsert etc) in the Kudu Java API are one way only (no 
> decode) because the server is written in C++.
> # An alternative could be to introduce a new object to beam  (e.g. 
> {{o.a.b.sdk.io.kudu.KuduOperation}}) to enable 
> {{PCollection}}. This was considered but was discounted 
> because:
> ## It is not a

[jira] [Updated] (BEAM-2661) Add KuduIO

2018-06-17 Thread Tim Robertson (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson updated BEAM-2661:

Description: 
New IO for Apache Kudu (https://kudu.apache.org/overview.html).

This work is in progress [on this 
branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO].

Design aspects are documented below.

The API
# The Kudu 
[Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
 is a fat class, and is a subclass of {{KuduRpc}}. It holds 
RPC logic, callbacks and a Kudu client. Because of this the {{Operation}} does 
not serialize and furthermore, the logic for encoding the operations (Insert, 
Upsert etc) in the Kudu Java API are one way only (no decode) because the 
server is written in C++.
# An alternative could be to introduce a new object to beam  (e.g. 
{{o.a.b.sdk.io.kudu.KuduOperation}}) to enable {{PCollection}}. 
This was considered but was discounted because:
## It is not a familiar API to those already knowing Kudu
## It still requires serialization and deserialization of the operations. Using 
the existing Kudu approach of serializing into compact byte arrays would 
require a decoder along the lines of [this almost complete 
example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e]




  was:New IO for Apache Kudu (https://kudu.apache.org/overview.html).


> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: https://issues.apache.org/jira/browse/BEAM-2661
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Tim Robertson
>Priority: Major
>
> New IO for Apache Kudu (https://kudu.apache.org/overview.html).
> This work is in progress [on this 
> branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO].
> Design aspects are documented below.
> The API
> # The Kudu 
> [Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
>  is a fat class, and is a subclass of {{KuduRpc}}. It 
> holds RPC logic, callbacks and a Kudu client. Because of this the 
> {{Operation}} does not serialize and furthermore, the logic for encoding the 
> operations (Insert, Upsert etc) in the Kudu Java API are one way only (no 
> decode) because the server is written in C++.
> # An alternative could be to introduce a new object to beam  (e.g. 
> {{o.a.b.sdk.io.kudu.KuduOperation}}) to enable 
> {{PCollection}}. This was considered but was discounted 
> because:
> ## It is not a familiar API to those already knowing Kudu
> ## It still requires serialization and deserialization of the operations. 
> Using the existing Kudu approach of serializing into compact byte arrays 
> would require a decoder along the lines of [this almost complete 
> example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #760

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 27.65 MB...]
INFO: Container container-74 terminating.
Jun 17, 2018 7:34:12 PM com.datatorrent.bufferserver.server.Server$3 run
INFO: Removing ln 
LogicalNode@c93c042identifier=tcp://localhost:40255/81.output.83, 
upstream=81.output.83, group=stream51/82.data2, partitions=[], 
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@233eb81c{da=com.datatorrent.bufferserver.internal.DataList$Block@329392d7{identifier=81.output.83,
 data=1048576, readingOffset=0, writingOffset=84, 
starting_window=5b26b7b10001, ending_window=5b26b7b10008, refCount=2, 
uniqueIdentifier=0, next=null, future=null}}} from dl 
DataList@49d32cb9[identifier=81.output.83]
Jun 17, 2018 7:34:12 PM com.datatorrent.bufferserver.server.Server$3 run
INFO: Removing ln 
LogicalNode@58be36f8identifier=tcp://localhost:40255/41.output.42, 
upstream=41.output.42, group=stream21/42.data2, partitions=[], 
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@209d536a{da=com.datatorrent.bufferserver.internal.DataList$Block@76b5f13d{identifier=41.output.42,
 data=1048576, readingOffset=0, writingOffset=84, 
starting_window=5b26b7b10001, ending_window=5b26b7b10008, refCount=2, 
uniqueIdentifier=0, next=null, future=null}}} from dl 
DataList@589ec016[identifier=41.output.42]
Jun 17, 2018 7:34:12 PM com.datatorrent.bufferserver.server.Server$3 run
INFO: Removing ln 
LogicalNode@7fdf3940identifier=tcp://localhost:40255/137.output.139, 
upstream=137.output.139, group=stream55/138.sideInput1, partitions=[], 
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@8788958{da=com.datatorrent.bufferserver.internal.DataList$Block@42b49d3e{identifier=137.output.139,
 data=1048576, readingOffset=0, writingOffset=96, 
starting_window=5b26b7b10001, ending_window=5b26b7b10008, refCount=2, 
uniqueIdentifier=0, next=null, future=null}}} from dl 
DataList@1bd1b04c[identifier=137.output.139]
Jun 17, 2018 7:34:12 PM com.datatorrent.bufferserver.server.Server$3 run
INFO: Removing ln 
LogicalNode@24f01644identifier=tcp://localhost:40255/67.output.68, 
upstream=67.output.68, group=stream81/68.input, partitions=[], 
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@35615d50{da=com.datatorrent.bufferserver.internal.DataList$Block@791f20ed{identifier=67.output.68,
 data=1048576, readingOffset=0, writingOffset=96, 
starting_window=5b26b7b10001, ending_window=5b26b7b10008, refCount=2, 
uniqueIdentifier=0, next=null, future=null}}} from dl 
DataList@1fa41f9d[identifier=67.output.68]
Jun 17, 2018 7:34:12 PM com.datatorrent.bufferserver.server.Server$3 run
INFO: Removing ln 
LogicalNode@3733b240identifier=tcp://localhost:40255/18.out.18, 
upstream=18.out.18, group=stream133/19.data1, partitions=[], 
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@9618ff5{da=com.datatorrent.bufferserver.internal.DataList$Block@36d5d854{identifier=18.out.18,
 data=1048576, readingOffset=0, writingOffset=18341, 
starting_window=5b26b7b10001, ending_window=5b26b7b10008, refCount=2, 
uniqueIdentifier=0, next=null, future=null}}} from dl 
DataList@364d8e2b[identifier=18.out.18]
Jun 17, 2018 7:34:12 PM com.datatorrent.stram.util.LoggerUtil 
getFileAppender
WARNING: Log information is unavailable. To enable log information 
log4j/logging should be configured with single FileAppender that has 
immediateFlush set to true and log level set to ERROR or greater.
Jun 17, 2018 7:34:12 PM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=1,name=split9,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream0,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:34:12 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-25 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #520

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 54.75 MB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
java.lang.AssertionError: OutputSideInputs/ParMultiDo(Anonymous).output: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.testing.PAssert$DefaultConcludeFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn.processElement(PAssert.java:1215)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:42)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:131)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn.processElement(PAssert.java:1049)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #759

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 27.63 MB...]
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-72 terminating.
Jun 17, 2018 1:44:45 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-0 terminating.
Jun 17, 2018 1:44:45 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-71 msg: [container-71] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-2 msg: [container-2] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-71 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-5 msg: [container-5] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-5 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-35 msg: [container-35] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-35 terminating.
Jun 17, 2018 1:44:45 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
Jun 17, 2018 1:44:45 PM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-70 msg: [container-70] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-4 msg: [container-4] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-2 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-70 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-37 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-4 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-21 msg: [container-21] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-65 msg: [container-65] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-31 msg: [container-31] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-21 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-58 msg: [container-58] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-11 msg: [container-11] Exiting heartbeat loop..
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-58 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-28 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-11 terminating.
Jun 17, 2018 1:44:45 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-34

Jenkins build is back to normal : beam_PerformanceTests_JDBC #733

2018-06-17 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-3863) AfterProcessingTime trigger doesn't fire reliably

2018-06-17 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3863?focusedWorklogId=112615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-112615
 ]

ASF GitHub Bot logged work on BEAM-3863:


Author: ASF GitHub Bot
Created on: 17/Jun/18 11:36
Start Date: 17/Jun/18 11:36
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #4875: BEAM-3863: 
AfterProcessingTime trigger firing at delayedUntil time
URL: https://github.com/apache/beam/pull/4875#issuecomment-397872801
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 112615)
Time Spent: 1.5h  (was: 1h 20m)

> AfterProcessingTime trigger doesn't fire reliably
> -
>
> Key: BEAM-3863
> URL: https://issues.apache.org/jira/browse/BEAM-3863
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0, 2.2.0, 2.3.0
>Reporter: Pawel Bartoszek
>Assignee: Aljoscha Krettek
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> *Issue*
> Beam AfterProcessingTime trigger doesn't fire always reliably after a 
> configured delay.
> The following job triggers should fire after watermark passes the end of the 
> window and then every 5 seconds for late data and the finally at the end of 
> allowed lateness.
> *Expected behaviour*
> Late firing after processing time trigger should fire after 5 seconds since 
> first late records arrive in the pane.
> *Actual behaviour*
> From my testings late triggers works for some keys but not for the other - 
> it's pretty random which keys are affected. The DummySource generates 15 
> distinct keys AA,BB,..., PP. For each key it sends 5 on time records and one 
> late record. In case late trigger firing is missed it won't fire until the 
> allowed lateness period. 
> *Job code*
> {code:java}
> String[] runnerArgs = {"--runner=FlinkRunner", "--parallelism=8"};
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(runnerArgs).as(FlinkPipelineOptions.class);
> Pipeline pipeline = Pipeline.create(options);
> PCollection apply = pipeline.apply(Read.from(new DummySource()))
> 
> .apply(Window.into(FixedWindows.of(Duration.standardSeconds(10)))
> .triggering(AfterWatermark.pastEndOfWindow()
> .withLateFirings(
> AfterProcessingTime
> 
> .pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> .withAllowedLateness(Duration.standardMinutes(2), 
> Window.ClosingBehavior.FIRE_IF_NON_EMPTY)
> );
> apply.apply(Count.perElement())
> .apply(ParDo.of(new DoFn, Long>() {
> @ProcessElement
> public void process(ProcessContext context, BoundedWindow window) 
> {
> LOG.info("Count: {}. For window {}, Pane {}", 
> context.element(), window, context.pane());
> }
> }));
> pipeline.run().waitUntilFinish();{code}
>  
> *How can you replicate the issue?*
>  I've created a github repo 
> [https://github.com/pbartoszek/BEAM-3863_late_trigger] with the code shown 
> above. Please check out the README file for details how to replicate the 
> issue.
> *What's is causing the issue?*
> I explained the cause in PR.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #519

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 54.75 MB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
java.lang.AssertionError: OutputSideInputs/ParMultiDo(Anonymous).output: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.testing.PAssert$DefaultConcludeFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn.processElement(PAssert.java:1215)
at 
org.apache.beam.sdk.testing.PAssert$GroupedValuesCheckerDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:42)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:131)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:149)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:219)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:69)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:517)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:505)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn.processElement(PAssert.java:1049)
at 
org.apache.beam.sdk.testing.PAssert$ConcatFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #758

2018-06-17 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #757

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 27.64 MB...]
WARNING: Log information is unavailable. To enable log information 
log4j/logging should be configured with single FileAppender that has 
immediateFlush set to true and log level set to ERROR or greater.
Jun 17, 2018 8:30:09 AM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=8,name=split2,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream87,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 8:30:09 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-67 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 8:30:09 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-67 terminating.
Jun 17, 2018 8:30:09 AM com.datatorrent.bufferserver.server.Server$3 run
INFO: Removing ln 
LogicalNode@73c4ecdidentifier=tcp://localhost:35477/8.output.6, 
upstream=8.output.6, group=stream87/9.data1, partitions=[], 
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@4902e3f7{da=com.datatorrent.bufferserver.internal.DataList$Block@194a0f3e{identifier=8.output.6,
 data=1048576, readingOffset=0, writingOffset=2381, 
starting_window=5b261c0d0001, ending_window=5b261c0d000a, refCount=2, 
uniqueIdentifier=0, next=null, future=null}}} from dl 
DataList@73dd743e[identifier=8.output.6]
Jun 17, 2018 8:30:09 AM com.datatorrent.stram.util.LoggerUtil 
getFileAppender
WARNING: Log information is unavailable. To enable log information 
log4j/logging should be configured with single FileAppender that has 
immediateFlush set to true and log level set to ERROR or greater.
Jun 17, 2018 8:30:09 AM com.datatorrent.stram.util.LoggerUtil 
getFileAppender
WARNING: Log information is unavailable. To enable log information 
log4j/logging should be configured with single FileAppender that has 
immediateFlush set to true and log level set to ERROR or greater.
Jun 17, 2018 8:30:09 AM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=14,name=split5,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream113,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 8:30:09 AM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=15,name=split4,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream114,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 8:30:09 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-4 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #756

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 27.66 MB...]
Jun 17, 2018 7:30:16 AM com.datatorrent.stram.util.LoggerUtil 
getFileAppender
WARNING: Log information is unavailable. To enable log information 
log4j/logging should be configured with single FileAppender that has 
immediateFlush set to true and log level set to ERROR or greater.
Jun 17, 2018 7:30:16 AM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=14,name=split1,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream76,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:30:16 AM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=7,name=split4,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream48,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:30:16 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-26 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:30:16 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-27 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:30:16 AM com.datatorrent.stram.engine.StreamingContainer$2 
run
SEVERE: Operator set 
[OperatorDeployInfo[id=15,name=split0,type=INPUT,checkpoint={, 
0, 
0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream91,bufferServer=localhost
 stopped running due to an exception.
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:30:16 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-41 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at 
com.datatorrent.common.util.BaseOperator.shutdown(BaseOperator.java:96)
at 
org.apache.beam.runners.apex.translation.operators.ApexReadUnboundedInputOperator.endWindow(ApexReadUnboundedInputOperator.java:116)
at com.datatorrent.stram.engine.InputNode.run(InputNode.java:229)
at 
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)

Jun 17, 2018 7:30:16 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-39 msg: Stopped running due to an exception. 
com.datatorrent.api.Operator$ShutdownException
at

Build failed in Jenkins: beam_PerformanceTests_JDBC #732

2018-06-17 Thread Apache Jenkins Server

See 


--
[...truncated 121.86 KB...]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:748)

Jun 17, 2018 7:28:45 AM org.postgresql.Driver connect
SEVERE: Connection error: 
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.deleteTable(DatabaseTestHelper.java:57)
at org.apache.beam.sdk.io.jdbc.JdbcIOIT.tearDown(JdbcIOIT.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at

47 matches

Mail list logo