Build failed in Jenkins: beam_PerformanceTests_Python #892

[Apache Jenkins Server]

See 


Changes:

[robertwb] Disable verbose typecoder warnings.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8c5ebe4a4d158fa8c46640a0401376216a1d11c2 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8c5ebe4a4d158fa8c46640a0401376216a1d11c2
Commit message: "Merge pull request #4641 [BEAM-3644] Disable verbose typecoder 
warnings."
 > git rev-list 3df248b48d3be78f8b37470d427921874f1647ea # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3994245533533100568.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5623854011421385852.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1276338473924140777.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6867253349889358569.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3228263551531746592.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4521068121228572727.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

[beam] 01/01: Merge pull request #4641 [BEAM-3644] Disable verbose typecoder warnings.

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8c5ebe4a4d158fa8c46640a0401376216a1d11c2
Merge: 3df248b 3b8b753
Author: Robert Bradshaw 
AuthorDate: Thu Feb 8 21:27:15 2018 -0800

Merge pull request #4641 [BEAM-3644] Disable verbose typecoder warnings.

[BEAM-3644] Disable verbose typecoder warnings.

 sdks/python/apache_beam/coders/typecoders.py   | 8 
 sdks/python/apache_beam/typehints/typehints.py | 5 +
 2 files changed, 9 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (3df248b -> 8c5ebe4)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3df248b  Merge pull request #4600: [BEAM-410] Sort 
PriorityQueue with explicit comparator
 add 3b8b753  Disable verbose typecoder warnings.
 new 8c5ebe4  Merge pull request #4641 [BEAM-3644] Disable verbose 
typecoder warnings.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/coders/typecoders.py   | 8 
 sdks/python/apache_beam/typehints/typehints.py | 5 +
 2 files changed, 9 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #865

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4185

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4184

[Apache Jenkins Server]

See 


--
[...truncated 1.35 MB...]
test_log_calls_for_transient_failure (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_default_number_of_retries (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_explicit_decorator (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_explicit_initial_delay (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_explicit_number_of_retries (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_http_error_that_should_be_retried 
(apache_beam.utils.retry_test.RetryTest) ... ok
test_with_http_error_that_should_not_be_retried 
(apache_beam.utils.retry_test.RetryTest) ... ok
test_with_no_retry_decorator (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_real_clock (apache_beam.utils.retry_test.RetryTest) ... ok
test_arithmetic (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_of (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_precision (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_sort_order (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_str (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_arithmetic (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_of (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_precision (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_sort_order (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_str (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_utc_timestamp (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_equality (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_hash (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_pickle (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_timestamps (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_with_value (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_no_partial_writeouts (apache_beam.pipeline_test.DirectRunnerRetryTests) 
... ok
test_retry_fork_graph (apache_beam.pipeline_test.DirectRunnerRetryTests) ... ok
test_element (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_no_tag (apache_beam.pipeline_test.DoFnTest) ... ok
test_side_input_tagged (apache_beam.pipeline_test.DoFnTest) ... ok
test_timestamp_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_timestamp_param_map (apache_beam.pipeline_test.DoFnTest) ... ok
test_window_param (apache_beam.pipeline_test.DoFnTest) ... ok
test_attribute_setting (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_defaults (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_dir (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_flag_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_keyword_parsing (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_view_as (apache_beam.pipeline_test.PipelineOptionsTest) ... ok
test_aggregator_empty_input (apache_beam.pipeline_test.PipelineTest) ... ok
test_apply_custom_transform (apache_beam.pipeline_test.PipelineTest) ... ok
test_create (apache_beam.pipeline_test.PipelineTest) ... ok
test_create_singleton_pcollection (apache_beam.pipeline_test.PipelineTest) ... 
ok
test_fake_read (apache_beam.pipeline_test.PipelineTest) ... ok
test_flatmap_builtin (apache_beam.pipeline_test.PipelineTest) ... ok
test_memory_usage (apache_beam.pipeline_test.PipelineTest) ... ok
test_metrics_in_fake_source (apache_beam.pipeline_test.PipelineTest) ... ok
test_pipeline_as_context (apache_beam.pipeline_test.PipelineTest) ... 
:133:
 UserWarning: Using fallback coder for typehint: .
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
ok
test_ptransform_override_type_hints (apache_beam.pipeline_test.PipelineTest) 
... ok
test_ptransform_overrides (apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_cloned_custom_transform_instance 
(apache_beam.pipeline_test.PipelineTest) ... ok
test_reuse_custom_transform_instance (apache_beam.pipeline_test.PipelineTest) 
... 
:226:
 DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  cm.exception.message,
ok
test_transform_no_super_init (apache_beam.pipeline_test.PipelineTest) ... ok
test_visit_entire_graph (apache_beam.pipeline_test.PipelineTest) ... ok
test_parent_pointer (apache_beam.pipeline_test.RunnerApiTest) ... ok
test_pickling (apache_beam.pipeline_test.RunnerApiTest) ... ok
test_simple (apache_beam.pipeline_test.RunnerApiTest)
Tests serializing, deserializing, and running a simple pipeline. ... ok
test_assingleton_multi_element (apache_beam.pvalue_test.PValueTest) ... ok
test_pvalue_expected_arguments 

[beam] branch go-sdk updated (a328e44 -> 8cb9500)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a328e44  Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go 
SDK
 add 21bc8cf  [BEAM-3579] Fix textio.Write
 add 6e71095  CR: fix Go SDK textio.Write
 new 8cb9500  Merge pull request #4624

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/io/textio/textio.go | 55 +---
 1 file changed, 26 insertions(+), 29 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4624

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8cb95e79c591678870309982fb9cc728ded0
Merge: a328e44 6e71095
Author: Thomas Groh 
AuthorDate: Thu Feb 8 17:17:48 2018 -0800

Merge pull request #4624

[BEAM-3579] Fix textio.Write

 sdks/go/pkg/beam/io/textio/textio.go | 55 +---
 1 file changed, 26 insertions(+), 29 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Commented] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357782#comment-16357782
 ] 

Chamikara Jayalath commented on BEAM-3519:
--

Were previous releases affected by this issue as well or was there a recent 
change to the dependency structure that caused/exacerbated this ?

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3649:
--

Assignee: (was: Reuven Lax)

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357780#comment-16357780
 ] 

Ismaël Mejía commented on BEAM-3649:


Isn't this by any chance the same issue than BEAM-2790 ? If it is can you 
please help us confirm if this is working on current master (aka 
2.4.0-SNAPSHOT) ?

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357764#comment-16357764
 ] 

Luke Cwik commented on BEAM-3519:
-

Speaking with the gRPC folks, it seems like we should attempt to migrate to 
using [https://mvnrepository.com/artifact/io.grpc/grpc-netty-shaded/1.9.0]

This would require updating all GCP clients dependent on gRPC to this version.

 

Updating the gRPC/protobuf version will likely cause user pain unless we can 
shade them away completely as well.

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357759#comment-16357759
 ] 

Chamikara Jayalath commented on BEAM-3519:
--

Possibly. Does this only have to be fixed for Maven build or do we have to look 
at Gradle as well ?

Also, can the release proceed with the fixe to 
https://issues.apache.org/jira/browse/BEAM-3668 ?

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #891

[Apache Jenkins Server]

See 


Changes:

[robertwb] [BEAM-3625] Enable DoFn params in Map, Filter, etc.

[apilloud] [BEAM-410] Sort PriorityQueue with explicit 
comparator

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 3df248b48d3be78f8b37470d427921874f1647ea (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3df248b48d3be78f8b37470d427921874f1647ea
Commit message: "Merge pull request #4600: [BEAM-410] Sort 
PriorityQueue with explicit comparator"
 > git rev-list 6792846069763770bbfd92a5eb61e64968522967 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2682626538948213751.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1769671578288419310.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1090966839980201130.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7847252738010885446.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5355353597151300352.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4927249454973804205.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause 

[jira] [Commented] (BEAM-2591) Python shim for submitting to FlinkRunner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357748#comment-16357748
 ] 

Kenneth Knowles commented on BEAM-2591:
---

I think probably best to get feedback on idiomatic uses from [~robertwb], 
[~altay], [~jgrier], etc.

Basically I imagine a user will
1. test bounded smoke test with the Python direct runner
2. try the same on a local flink (or whatever is a one-liner to launch and 
teardown) and also some unbounded data at that point
3. go big by:
a. point the pipeline at a production endpoint where they don't know or 
care what tech is behind it
b. or launch their pipeline main alongside flink as a complete app via k8s 
or a bespoke setup
c. ???

This ticket had been filed for 2 with the idea that it would be reusable for 3 
but actually I think it should just be split into separate tasks and this focus 
on 2.

> Python shim for submitting to FlinkRunner
> -
>
> Key: BEAM-2591
> URL: https://issues.apache.org/jira/browse/BEAM-2591
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink, sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> Whatever the result of https://s.apache.org/beam-job-api, Python users will 
> need to be able to pass --runner=FlinkRunner and have it work.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #864

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-410] Sort PriorityQueue with explicit 
comparator

--
[...truncated 205.95 KB...]
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s28"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert:even/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:even/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s29"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
 

[jira] [Commented] (BEAM-2591) Python shim for submitting to FlinkRunner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357735#comment-16357735
 ] 

Kenneth Knowles commented on BEAM-2591:
---

Yea I don't think that this is quite the right ticket any more. I think we do 
want:

 - Python to be able to submit to a portable endpoint without knowing what 
runner it is, so maybe {{--runner-endpoint}}
 - Python to be able to spin up/down a Flink endpoint for easy local testing

> Python shim for submitting to FlinkRunner
> -
>
> Key: BEAM-2591
> URL: https://issues.apache.org/jira/browse/BEAM-2591
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink, sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> Whatever the result of https://s.apache.org/beam-job-api, Python users will 
> need to be able to pass --runner=FlinkRunner and have it work.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #22

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2591) Python shim for submitting to FlinkRunner

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357726#comment-16357726
 ] 

Ben Sidhom commented on BEAM-2591:
--

[~kenn]

Is this still the case? Do we want the Python SDK to spin up a Flink deployment 
and then submit jobs to it? If not, then we need the endpoint of an 
already-running Job API anyway. This is no different for Flink than any other 
runner.

The main distinction here is if we want the FlinkRunner to run locally, in 
which case it might be reasonable to have Flink create an in-process 
"deployment" and execute against it.

> Python shim for submitting to FlinkRunner
> -
>
> Key: BEAM-2591
> URL: https://issues.apache.org/jira/browse/BEAM-2591
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink, sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> Whatever the result of https://s.apache.org/beam-job-api, Python users will 
> need to be able to pass --runner=FlinkRunner and have it work.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3519:
--
Priority: Blocker  (was: Critical)

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3519:
--
Fix Version/s: 2.3.0

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Critical
> Fix For: 2.3.0
>
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3674) Port ElasticSearchIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357719#comment-16357719
 ] 

Kenneth Knowles edited comment on BEAM-3674 at 2/8/18 11:45 PM:


>From 
>https://issues.apache.org/jira/browse/BEAM-3159?focusedCommentId=16340765=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16340765

"how could you write a test such as this one (1) with only TestPipeline and 
DirectRunner (only external test capabilities over the pipeline)?

(1)  
https://github.com/apache/beam/blob/87670e6f525f3a9e51f6603f072410f86be48447/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java#L220;

I believe that test is probably pretty easy with TestStream. One thing that 
test does is that it checks assertions in the middle of processing. But it 
should be fine to test the overall results of the pipeline. I would say that it 
is also probably overspecified because it verifies exactly step by step what 
happens when the spec of max batch size should probably be an inequality not 
exact behavior.


was (Author: kenn):
>From 
>https://issues.apache.org/jira/browse/BEAM-3159?focusedCommentId=16340765=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16340765

"how could you write a test such as this one (1) with only TestPipeline and 
DirectRunner (only external test capabilities over the pipeline)?

(1)  
https://github.com/apache/beam/blob/87670e6f525f3a9e51f6603f072410f86be48447/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java#L220;

I believe that test is probably pretty easy with TestStream. One thing that 
test does is that it checks assertions in the middle of processing. But it 
should be fine to test the overall results of the pipeline.

> Port ElasticSearchIOTest off DoFnTester
> ---
>
> Key: BEAM-3674
> URL: https://issues.apache.org/jira/browse/BEAM-3674
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3674) Port ElasticSearchIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357719#comment-16357719
 ] 

Kenneth Knowles commented on BEAM-3674:
---

>From 
>https://issues.apache.org/jira/browse/BEAM-3159?focusedCommentId=16340765=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16340765

"how could you write a test such as this one (1) with only TestPipeline and 
DirectRunner (only external test capabilities over the pipeline)?

(1)  
https://github.com/apache/beam/blob/87670e6f525f3a9e51f6603f072410f86be48447/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java#L220;

I believe that test is probably pretty easy with TestStream. One thing that 
test does is that it checks assertions in the middle of processing. But it 
should be fine to test the overall results of the pipeline.

> Port ElasticSearchIOTest off DoFnTester
> ---
>
> Key: BEAM-3674
> URL: https://issues.apache.org/jira/browse/BEAM-3674
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3650) Deprecate and remove DoFnTester

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357715#comment-16357715
 ] 

Kenneth Knowles commented on BEAM-3650:
---

Thanks! I will answer specifics on BEAM-3674

> Deprecate and remove DoFnTester
> ---
>
> Key: BEAM-3650
> URL: https://issues.apache.org/jira/browse/BEAM-3650
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3159) DoFnTester should be deprecated in favor of TestPipeline

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3159.
---
Resolution: Fixed

> DoFnTester should be deprecated in favor of TestPipeline
> 
>
> Key: BEAM-3159
> URL: https://issues.apache.org/jira/browse/BEAM-3159
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.4.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Reasons:
> 1. The logical unit within a Beam pipeline is a transform. Either a small 
> transform like a ParDo or a larger composite transform. Unit tests should 
> focus on these units, rather than probing specific behaviors of the 
> user-defined functions.
> 2. The way that a runner interacts with a user-defined function is not 
> necessarily obvious. DoFnTester allows testing non-sensical cases that 
> wouldn't arise in practice, since it allows low-level interactions with the 
> actual UDFs.
> Instead, we should encourage the use of TestPipeline with the direct runner. 
> This allows testing a single transform (such as a ParDo running a UDF) in 
> context. It also makes it easier to test things like side-inputs and multiple 
> outputs, since you use the same techniques in the test as you would in a real 
> pipeline, rather than requiring a whole new API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3674) Port ElasticSearchIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3674:
--
Labels: beginner newbie starter  (was: )

> Port ElasticSearchIOTest off DoFnTester
> ---
>
> Key: BEAM-3674
> URL: https://issues.apache.org/jira/browse/BEAM-3674
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3674) Port ElasticSearchIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3674:
-

 Summary: Port ElasticSearchIOTest off DoFnTester
 Key: BEAM-3674
 URL: https://issues.apache.org/jira/browse/BEAM-3674
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-extensions
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3675) FlinkRunner: Logging server

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-3675:


 Summary: FlinkRunner: Logging server
 Key: BEAM-3675
 URL: https://issues.apache.org/jira/browse/BEAM-3675
 Project: Beam
  Issue Type: Sub-task
  Components: runner-flink
Reporter: Ben Sidhom
Assignee: Aljoscha Krettek


An implementation of BeamFnLogging that uses the default Flink logging 
mechanism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3650) Deprecate and remove DoFnTester

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357713#comment-16357713
 ] 

Ismaël Mejía commented on BEAM-3650:


ElasticsearchIO is missing in this list and the case is probably the same as 
SolrIO, but I am still curious of the unanswered question that Etienne asked on 
BEAM-3159

> Deprecate and remove DoFnTester
> ---
>
> Key: BEAM-3650
> URL: https://issues.apache.org/jira/browse/BEAM-3650
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-3673:


 Summary: FlinkRunner: Harness manager for connecting operators to 
SDK Harnesses
 Key: BEAM-3673
 URL: https://issues.apache.org/jira/browse/BEAM-3673
 Project: Beam
  Issue Type: Sub-task
  Components: runner-flink
Reporter: Ben Sidhom
Assignee: Aljoscha Krettek


SDK harnesses require a common set of gRPC services to operate. The role of the 
harness manager is to multiplex data streams and auxiliary data between SDK 
environments and operators within a given job.

Note that multiple operators may communicate with a single SDK environment to 
amortize container initialization cost. Environments are _not_ shared between 
different jobs.

The initial implementation will shell out to local docker, but the harness 
manager should eventually support working with externally-managed environments 
(e.g., created by Kubernetes).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

[Ben Sidhom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Sidhom updated BEAM-3673:
-
Description: 
SDK harnesses require a common set of gRPC services to operate. The role of the 
harness manager is to provide a uniform interface that multiplexes data streams 
and auxiliary data between SDK environments and operators within a given job.

Note that multiple operators may communicate with a single SDK environment to 
amortize container initialization cost. Environments are _not_ shared between 
different jobs.

The initial implementation will shell out to local docker, but the harness 
manager should eventually support working with externally-managed environments 
(e.g., created by Kubernetes).

  was:
SDK harnesses require a common set of gRPC services to operate. The role of the 
harness manager is to multiplex data streams and auxiliary data between SDK 
environments and operators within a given job.

Note that multiple operators may communicate with a single SDK environment to 
amortize container initialization cost. Environments are _not_ shared between 
different jobs.

The initial implementation will shell out to local docker, but the harness 
manager should eventually support working with externally-managed environments 
(e.g., created by Kubernetes).


> FlinkRunner: Harness manager for connecting operators to SDK Harnesses
> --
>
> Key: BEAM-3673
> URL: https://issues.apache.org/jira/browse/BEAM-3673
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Aljoscha Krettek
>Priority: Major
>
> SDK harnesses require a common set of gRPC services to operate. The role of 
> the harness manager is to provide a uniform interface that multiplexes data 
> streams and auxiliary data between SDK environments and operators within a 
> given job.
> Note that multiple operators may communicate with a single SDK environment to 
> amortize container initialization cost. Environments are _not_ shared between 
> different jobs.
> The initial implementation will shell out to local docker, but the harness 
> manager should eventually support working with externally-managed 
> environments (e.g., created by Kubernetes).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3519:
-

Assignee: Chamikara Jayalath

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Critical
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357708#comment-16357708
 ] 

Kenneth Knowles commented on BEAM-3519:
---

Can it be shaded?

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Priority: Critical
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357707#comment-16357707
 ] 

Ismaël Mejía commented on BEAM-3668:


This is not a problem of the spark runner, this is an issue of GCP IO leaking
netty into the classpath and producing a conflict with the spark runner. See
BEAM-3519. I just PRed a workaround for the archetype/quickstart, but I am
wondering if the unattended consequence is that the Spark runner make have some
issues with GCP IO for this release.

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: full_log.txt
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> 

[jira] [Assigned] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3668:
--

Assignee: Ismaël Mejía  (was: Jean-Baptiste Onofré)

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Ismaël Mejía
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: full_log.txt
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> 

[jira] [Created] (BEAM-3672) FlinkRunner: Implement an Artifact service using the Flink DistributedCache

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-3672:


 Summary: FlinkRunner: Implement an Artifact service using the 
Flink DistributedCache
 Key: BEAM-3672
 URL: https://issues.apache.org/jira/browse/BEAM-3672
 Project: Beam
  Issue Type: Sub-task
  Components: runner-flink
Reporter: Ben Sidhom
Assignee: Aljoscha Krettek


We need to have a DistributedCache-based artifact service to ship with the 
portable Flink runner. The DistributedCache is a perfect fit for Flink because 
it comes for free and is the mechanism that Flink already uses to distribute 
its own artifacts.

 

The final artifact service implementation should be pluggable, but using the 
DistributedCache allows the Flink runner to work without additional external 
dependencies (beyond perhaps Docker).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3671) Need Python 3.x interpreter in Jenkins machines

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-3671:
-

 Summary: Need Python 3.x interpreter in Jenkins machines
 Key: BEAM-3671
 URL: https://issues.apache.org/jira/browse/BEAM-3671
 Project: Beam
  Issue Type: New Feature
  Components: testing
Reporter: Ahmet Altay
Assignee: Alan Myrvold


This is required for being able to test with Py3. If possible, it would be best 
to have 3.5 and 3.6. If not we can settle on a single version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3668:
--
Fix Version/s: 2.3.0

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: full_log.txt
>
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 

[jira] [Created] (BEAM-3670) DirectRunner: TypeError: 'KeyedWorkItem' object is not iterable

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-3670:
-

 Summary: DirectRunner: TypeError: 'KeyedWorkItem' object is not 
iterable
 Key: BEAM-3670
 URL: https://issues.apache.org/jira/browse/BEAM-3670
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Charles Chen


I see the following error in a pre-commit test with no code changes:

[https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/2661/org.apache.beam$beam-sdks-python/console]

 
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/runners/direct/executor.py",
 line 305, in call
side_input_values)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/runners/direct/executor.py",
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/runners/direct/transform_evaluator.py",
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/transforms/core.py",
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3669) Linter error in statesampler_fast

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-3669:
-

 Summary: Linter error in statesampler_fast
 Key: BEAM-3669
 URL: https://issues.apache.org/jira/browse/BEAM-3669
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Pablo Estrada


This is a precommit failure, but I have seen it in 2 unrelated pre-commits. It 
is possible that this is an issue with the head:

Link to a job: 
https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/2661/org.apache.beam$beam-sdks-python/console

 
Running pylint for module apache_beam:
* Module apache_beam.runners.worker.statesampler_fast
E:  1, 0: compile() expected string without null bytes (syntax-error)
* Module apache_beam.coders.stream
E:  1, 0: compile() expected string without null bytes (syntax-error)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-410) ApproximateQuantiles$QuantileBuffer defines compareTo but not equals

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud resolved BEAM-410.
-
   Resolution: Fixed
Fix Version/s: 2.4.0

> ApproximateQuantiles$QuantileBuffer defines compareTo but not equals
> 
>
> Key: BEAM-410
> URL: https://issues.apache.org/jira/browse/BEAM-410
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Andrew Pilloud
>Priority: Minor
>  Labels: findbugs, newbie, starter
> Fix For: 2.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> [FindBugs 
> EQ_COMPARETO_USE_OBJECT_EQUALS|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L280]:
>  Class defines compareTo(...) and uses Object.equals()
> Applies to: 
> [ApproximateQuanitles$QuantileBuffer|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java#L627].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (f1909bc -> 3df248b)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f1909bc  Merge pull request #4616 from robertwb/map-params
 add c0a6de0  [BEAM-410] Sort PriorityQueue with explicit 
comparator
 new 3df248b  Merge pull request #4600: [BEAM-410] Sort 
PriorityQueue with explicit comparator

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../build-tools/src/main/resources/beam/findbugs-filter.xml|  6 --
 .../org/apache/beam/sdk/transforms/ApproximateQuantiles.java   | 10 +++---
 2 files changed, 3 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4600: [BEAM-410] Sort PriorityQueue with explicit comparator

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3df248b48d3be78f8b37470d427921874f1647ea
Merge: f1909bc c0a6de0
Author: Kenn Knowles 
AuthorDate: Thu Feb 8 15:01:48 2018 -0800

Merge pull request #4600: [BEAM-410] Sort PriorityQueue 
with explicit comparator

 .../build-tools/src/main/resources/beam/findbugs-filter.xml|  6 --
 .../org/apache/beam/sdk/transforms/ApproximateQuantiles.java   | 10 +++---
 2 files changed, 3 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Commented] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357623#comment-16357623
 ] 

Paweł Kaczmarczyk commented on BEAM-3605:
-

Ok, great [~aromanenko]

 

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #863

[Apache Jenkins Server]

See 


--
[...truncated 71.74 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.5.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, NoneType], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[NoneType, 
Tuple[Any, List[Any]]], Tuple[NoneType, Tuple[Any, NoneType]]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[Any, List[Any]], 
Tuple[Any, NoneType]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.5.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.5.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_metrics (apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[NoneType, 
Tuple[Any, List[Any]]], Tuple[NoneType, Tuple[Any, NoneType]]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4182

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3668:

Attachment: full_log.txt

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Attachments: full_log.txt
>
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> 

[jira] [Updated] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3668:

Summary: Apache Spark Java Quickstart fails 2.3.0 RC2  (was: Apache Spark 
Java Quickstart fails)

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4181

[Apache Jenkins Server]

See 


--
[...truncated 1.54 MB...]
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "SideInput-s15": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s15"
  }, 
  "SideInput-s16": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s16"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/FinalizeWrite.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": 
"ref_AppliedPTransform_write/Write/WriteImpl/FinalizeWrite_20", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-02-08_12_41_19-4541629610088221710]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-02-08_12_41_19-4541629610088221710?project=apache-beam-testing
root: INFO: Job 

[jira] [Created] (BEAM-3667) Failure in MongoDbIOTest.testReadWithCustomConnectionOptions

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3667:
-

 Summary: Failure in 
MongoDbIOTest.testReadWithCustomConnectionOptions
 Key: BEAM-3667
 URL: https://issues.apache.org/jira/browse/BEAM-3667
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Assignee: Reuven Lax


https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-mongodb/17675/console

There's an NPE and a text file contention error.

{code}
2018-02-08T18:40:34.453 [ERROR] 
testReadWithCustomConnectionOptions(org.apache.beam.sdk.io.mongodb.MongoDbIOTest)
  Time elapsed: 0.163 s  <<< ERROR!
java.io.IOException: Cannot run program 
"/tmp/extract-1753d5fc-31f4-4a30-bca0-58578c50e525extractmongod": error=26, 
Text file busy
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at 
de.flapdoodle.embed.process.runtime.ProcessControl.start(ProcessControl.java:205)
at 
de.flapdoodle.embed.process.runtime.AbstractProcess.(AbstractProcess.java:98)
at 
de.flapdoodle.embed.mongo.AbstractMongoProcess.(AbstractMongoProcess.java:53)
at de.flapdoodle.embed.mongo.MongodProcess.(MongodProcess.java:50)
at 
de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:44)
at 
de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:34)
at 
de.flapdoodle.embed.process.runtime.Executable.start(Executable.java:101)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIOTest.setup(MongoDbIOTest.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:324)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at
...
Caused by: java.io.IOException: error=26, Text file busy
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 47 more

2018-02-08T18:40:34.453 [ERROR] 
testReadWithCustomConnectionOptions(org.apache.beam.sdk.io.mongodb.MongoDbIOTest)
  Time elapsed: 0.163 s  <<< ERROR!
java.lang.NullPointerException
at 
org.apache.beam.sdk.io.mongodb.MongoDbIOTest.stop(MongoDbIOTest.java:139)
...
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3667) Failure in MongoDbIOTest.testReadWithCustomConnectionOptions

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3667:
-

Assignee: Jean-Baptiste Onofré  (was: Reuven Lax)

> Failure in MongoDbIOTest.testReadWithCustomConnectionOptions
> 
>
> Key: BEAM-3667
> URL: https://issues.apache.org/jira/browse/BEAM-3667
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-mongodb/17675/console
> There's an NPE and a text file contention error.
> {code}
> 2018-02-08T18:40:34.453 [ERROR] 
> testReadWithCustomConnectionOptions(org.apache.beam.sdk.io.mongodb.MongoDbIOTest)
>   Time elapsed: 0.163 s  <<< ERROR!
> java.io.IOException: Cannot run program 
> "/tmp/extract-1753d5fc-31f4-4a30-bca0-58578c50e525extractmongod": error=26, 
> Text file busy
>   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>   at 
> de.flapdoodle.embed.process.runtime.ProcessControl.start(ProcessControl.java:205)
>   at 
> de.flapdoodle.embed.process.runtime.AbstractProcess.(AbstractProcess.java:98)
>   at 
> de.flapdoodle.embed.mongo.AbstractMongoProcess.(AbstractMongoProcess.java:53)
>   at de.flapdoodle.embed.mongo.MongodProcess.(MongodProcess.java:50)
>   at 
> de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:44)
>   at 
> de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:34)
>   at 
> de.flapdoodle.embed.process.runtime.Executable.start(Executable.java:101)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIOTest.setup(MongoDbIOTest.java:115)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:324)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at
> ...
> Caused by: java.io.IOException: error=26, Text file busy
>   at java.lang.UNIXProcess.forkAndExec(Native Method)
>   at java.lang.UNIXProcess.(UNIXProcess.java:247)
>   at java.lang.ProcessImpl.start(ProcessImpl.java:134)
>   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
>   ... 47 more
> 2018-02-08T18:40:34.453 [ERROR] 
> testReadWithCustomConnectionOptions(org.apache.beam.sdk.io.mongodb.MongoDbIOTest)
>   Time elapsed: 0.163 s  <<< ERROR!
> java.lang.NullPointerException
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIOTest.stop(MongoDbIOTest.java:139)
> ...
> 

[jira] [Assigned] (BEAM-3038) Add support for Azure Data Lake Storage as a Apache Beam FileSystem

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3038:
--

Assignee: Jean-Baptiste Onofré  (was: Reuven Lax)

> Add support for Azure Data Lake Storage as a Apache Beam FileSystem
> ---
>
> Key: BEAM-3038
> URL: https://issues.apache.org/jira/browse/BEAM-3038
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Romit Girdhar
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: features
>
> This is for providing direct integration with Azure Data Lake Store as an 
> Apache Beam File system.
> There is already support for Azure Data Lake for using it as HDFS: 
> https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #890

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6792846069763770bbfd92a5eb61e64968522967 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6792846069763770bbfd92a5eb61e64968522967
Commit message: "Merge pull request #4204: [BEAM-3157] Generate BeamRecord 
types from Pojos"
 > git rev-list 6792846069763770bbfd92a5eb61e64968522967 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2163420454549967526.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins744456908022423087.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7321830917982415161.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8411628881629488443.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2766868182286632602.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3132807904252102636.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 

[beam] branch master updated (6792846 -> f1909bc)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6792846  Merge pull request #4204: [BEAM-3157] Generate BeamRecord 
types from Pojos
 add 4e333f7  [BEAM-3625] Enable DoFn params in Map, Filter, etc.
 new f1909bc  Merge pull request #4616 from robertwb/map-params

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline_test.py   | 6 ++
 sdks/python/apache_beam/transforms/core.py | 3 +++
 2 files changed, 9 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4616 from robertwb/map-params

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f1909bceb80c0446d8dd025c8dd532a6ca9375aa
Merge: 6792846 4e333f7
Author: Robert Bradshaw 
AuthorDate: Thu Feb 8 10:05:19 2018 -0800

Merge pull request #4616 from robertwb/map-params

[BEAM-3625] Enable DoFn params in Map, Filter, etc.

 sdks/python/apache_beam/pipeline_test.py   | 6 ++
 sdks/python/apache_beam/transforms/core.py | 3 +++
 2 files changed, 9 insertions(+)


-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Assigned] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3665:
-

Assignee: (was: Chamikara Jayalath)

> Port BigQueryIOWriteTest off DoFnTester
> ---
>
> Key: BEAM-3665
> URL: https://issues.apache.org/jira/browse/BEAM-3665
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3656) Port DatastoreV1Test off DoFnTester

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3656:
-

Assignee: (was: Chamikara Jayalath)

> Port DatastoreV1Test off DoFnTester
> ---
>
> Key: BEAM-3656
> URL: https://issues.apache.org/jira/browse/BEAM-3656
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3660) Port ReadSpannerSchemaTest off DoFnTester

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3660:
-

Assignee: (was: Chamikara Jayalath)

> Port ReadSpannerSchemaTest off DoFnTester
> -
>
> Key: BEAM-3660
> URL: https://issues.apache.org/jira/browse/BEAM-3660
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3658) Port SpannerIOReadTest off DoFnTester

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3658:
-

Assignee: (was: Chamikara Jayalath)

> Port SpannerIOReadTest off DoFnTester
> -
>
> Key: BEAM-3658
> URL: https://issues.apache.org/jira/browse/BEAM-3658
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3665:
-

 Summary: Port BigQueryIOWriteTest off DoFnTester
 Key: BEAM-3665
 URL: https://issues.apache.org/jira/browse/BEAM-3665
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-gcp
Reporter: Kenneth Knowles
Assignee: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3666) Port KafkaIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3666:
-

 Summary: Port KafkaIOTest off DoFnTester
 Key: BEAM-3666
 URL: https://issues.apache.org/jira/browse/BEAM-3666
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-extensions
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3662) Port MongoDbIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3662:
--
Component/s: (was: examples-java)
 sdk-java-extensions

> Port MongoDbIOTest off DoFnTester
> -
>
> Key: BEAM-3662
> URL: https://issues.apache.org/jira/browse/BEAM-3662
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3664) Port SolrIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3664:
-

 Summary: Port SolrIOTest off DoFnTester
 Key: BEAM-3664
 URL: https://issues.apache.org/jira/browse/BEAM-3664
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-extensions
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3663) Port SplittableParDoProcessFnTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3663:
-

 Summary: Port SplittableParDoProcessFnTest off DoFnTester
 Key: BEAM-3663
 URL: https://issues.apache.org/jira/browse/BEAM-3663
 Project: Beam
  Issue Type: Sub-task
  Components: runner-core
Reporter: Kenneth Knowles
Assignee: Eugene Kirpichov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3662) Port MongoDbIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3662:
-

 Summary: Port MongoDbIOTest off DoFnTester
 Key: BEAM-3662
 URL: https://issues.apache.org/jira/browse/BEAM-3662
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3661) Port TriggerExampleTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3661:
-

 Summary: Port TriggerExampleTest off DoFnTester
 Key: BEAM-3661
 URL: https://issues.apache.org/jira/browse/BEAM-3661
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3660) Port ReadSpannerSchemaTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3660:
-

 Summary: Port ReadSpannerSchemaTest off DoFnTester
 Key: BEAM-3660
 URL: https://issues.apache.org/jira/browse/BEAM-3660
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-gcp
Reporter: Kenneth Knowles
Assignee: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3658) Port SpannerIOReadTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3658:
-

 Summary: Port SpannerIOReadTest off DoFnTester
 Key: BEAM-3658
 URL: https://issues.apache.org/jira/browse/BEAM-3658
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-gcp
Reporter: Kenneth Knowles
Assignee: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3657) Port JoinExamplesTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3657:
-

 Summary: Port JoinExamplesTest off DoFnTester
 Key: BEAM-3657
 URL: https://issues.apache.org/jira/browse/BEAM-3657
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3659) Port UserScoreTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3659:
-

 Summary: Port UserScoreTest off DoFnTester
 Key: BEAM-3659
 URL: https://issues.apache.org/jira/browse/BEAM-3659
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3655) Port MaxPerKeyExamplesTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3655:
-

 Summary: Port MaxPerKeyExamplesTest off DoFnTester
 Key: BEAM-3655
 URL: https://issues.apache.org/jira/browse/BEAM-3655
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3656) Port DatastoreV1Test off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3656:
-

 Summary: Port DatastoreV1Test off DoFnTester
 Key: BEAM-3656
 URL: https://issues.apache.org/jira/browse/BEAM-3656
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-gcp
Reporter: Kenneth Knowles
Assignee: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3653) Port BatchViewOverridesTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3653:
-

 Summary: Port BatchViewOverridesTest off DoFnTester
 Key: BEAM-3653
 URL: https://issues.apache.org/jira/browse/BEAM-3653
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3654) Port FilterExamplesTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3654:
-

 Summary: Port FilterExamplesTest off DoFnTester
 Key: BEAM-3654
 URL: https://issues.apache.org/jira/browse/BEAM-3654
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3652) Port WriteWithShardingFactoryTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3652:
-

 Summary: Port WriteWithShardingFactoryTest off DoFnTester
 Key: BEAM-3652
 URL: https://issues.apache.org/jira/browse/BEAM-3652
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Kenneth Knowles
Assignee: Thomas Groh






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3651) Port BigQueryTornadoesTest off DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3651:
-

 Summary: Port BigQueryTornadoesTest off DoFnTester
 Key: BEAM-3651
 URL: https://issues.apache.org/jira/browse/BEAM-3651
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3650) Deprecate and remove DoFnTester

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3650:
-

 Summary: Deprecate and remove DoFnTester
 Key: BEAM-3650
 URL: https://issues.apache.org/jira/browse/BEAM-3650
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3159) DoFnTester should be deprecated in favor of TestPipeline

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3159:
--
Issue Type: Sub-task  (was: Bug)
Parent: BEAM-3650

> DoFnTester should be deprecated in favor of TestPipeline
> 
>
> Key: BEAM-3159
> URL: https://issues.apache.org/jira/browse/BEAM-3159
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.4.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Reasons:
> 1. The logical unit within a Beam pipeline is a transform. Either a small 
> transform like a ParDo or a larger composite transform. Unit tests should 
> focus on these units, rather than probing specific behaviors of the 
> user-defined functions.
> 2. The way that a runner interacts with a user-defined function is not 
> necessarily obvious. DoFnTester allows testing non-sensical cases that 
> wouldn't arise in practice, since it allows low-level interactions with the 
> actual UDFs.
> Instead, we should encourage the use of TestPipeline with the direct runner. 
> This allows testing a single transform (such as a ParDo running a UDF) in 
> context. It also makes it easier to test things like side-inputs and multiple 
> outputs, since you use the same techniques in the test as you would in a real 
> pipeline, rather than requiring a whole new API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-3159) DoFnTester should be deprecated in favor of TestPipeline

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-3159:
---

> DoFnTester should be deprecated in favor of TestPipeline
> 
>
> Key: BEAM-3159
> URL: https://issues.apache.org/jira/browse/BEAM-3159
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.4.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Reasons:
> 1. The logical unit within a Beam pipeline is a transform. Either a small 
> transform like a ParDo or a larger composite transform. Unit tests should 
> focus on these units, rather than probing specific behaviors of the 
> user-defined functions.
> 2. The way that a runner interacts with a user-defined function is not 
> necessarily obvious. DoFnTester allows testing non-sensical cases that 
> wouldn't arise in practice, since it allows low-level interactions with the 
> actual UDFs.
> Instead, we should encourage the use of TestPipeline with the direct runner. 
> This allows testing a single transform (such as a ParDo running a UDF) in 
> context. It also makes it easier to test things like side-inputs and multiple 
> outputs, since you use the same techniques in the test as you would in a real 
> pipeline, rather than requiring a whole new API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[Guillaume Balaine (JIRA)]

Guillaume Balaine created BEAM-3649:
---

 Summary: HadoopSeekableByteChannel breaks when backing InputStream 
doesn't supporte ByteBuffers
 Key: BEAM-3649
 URL: https://issues.apache.org/jira/browse/BEAM-3649
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 2.2.0, 2.1.0, 2.0.0
Reporter: Guillaume Balaine
Assignee: Reuven Lax


This happened last summer, when I wanted to use S3A as the backing HDFS access 
implementation. 

This is because while this method is called : 
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]

This class does not implement ByteBuffer readable 
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

I fixed it by manually incrementing the read position and copying the backing 
array instead of buffering.

[https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]

I know the s3 direct implementation is the preferred path, but this is 
possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3648) Support Splittable DoFn in Flink Batch Runner

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-3648:
--

 Summary: Support Splittable DoFn in Flink Batch Runner
 Key: BEAM-3648
 URL: https://issues.apache.org/jira/browse/BEAM-3648
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3637) HBaseIOTest methods do not clean up tables

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356999#comment-16356999
 ] 

Kenneth Knowles commented on BEAM-3637:
---

Yea, if it is a fake that will be dropped anyhow it might not matter to drop 
the tables one at a time. But maybe another way is to just make a pseudorandom 
suffix on the table name. The fanciest might be {{@Rule TemporaryHBaseTable}}. 
Since it is fixed and pretty stable I set the priority low. We could leave it 
for a starter ticket if someone wanted to just get into the codebase.

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356926#comment-16356926
 ] 

Jean-Baptiste Onofré commented on BEAM-302:
---

+1 !

[~sinisa_lyh] any chance we can discuss about this a new time, in regards of 
latest change in Beam ?

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[Jozef Vilcek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356923#comment-16356923
 ] 

Jozef Vilcek commented on BEAM-302:
---

BEAM-1920 is resolved. Looking forward to see next steps of integration Scio 
with Beam

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4949

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #4948

[Apache Jenkins Server]

See 


--
[...truncated 153.31 KB...]
2018-02-08T12:36:57.937 [INFO] Running 
org.apache.beam.runners.core.triggers.RepeatedlyStateMachineTest
2018-02-08T12:36:57.990 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.05 s - in 
org.apache.beam.runners.core.triggers.RepeatedlyStateMachineTest
2018-02-08T12:36:57.990 [INFO] Running 
org.apache.beam.runners.core.triggers.AfterWatermarkStateMachineTest
2018-02-08T12:36:58.103 [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.11 s - in 
org.apache.beam.runners.core.triggers.AfterWatermarkStateMachineTest
2018-02-08T12:36:58.105 [INFO] Running 
org.apache.beam.runners.core.triggers.AfterEachStateMachineTest
2018-02-08T12:36:58.120 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.012 s - in 
org.apache.beam.runners.core.triggers.AfterEachStateMachineTest
2018-02-08T12:36:58.121 [INFO] Running 
org.apache.beam.runners.core.triggers.FinishedTriggersBitSetTest
2018-02-08T12:36:58.126 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.002 s - in 
org.apache.beam.runners.core.triggers.FinishedTriggersBitSetTest
2018-02-08T12:36:58.133 [INFO] Running 
org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachineTest
2018-02-08T12:36:58.142 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.006 s - in 
org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachineTest
2018-02-08T12:36:58.142 [INFO] Running 
org.apache.beam.runners.core.triggers.DefaultTriggerStateMachineTest
2018-02-08T12:36:58.166 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.021 s - in 
org.apache.beam.runners.core.triggers.DefaultTriggerStateMachineTest
2018-02-08T12:36:58.171 [INFO] Running 
org.apache.beam.runners.core.triggers.AfterAllStateMachineTest
2018-02-08T12:36:58.195 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.021 s - in 
org.apache.beam.runners.core.triggers.AfterAllStateMachineTest
2018-02-08T12:36:58.195 [INFO] Running 
org.apache.beam.runners.core.triggers.ReshuffleTriggerStateMachineTest
2018-02-08T12:36:58.198 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in 
org.apache.beam.runners.core.triggers.ReshuffleTriggerStateMachineTest
2018-02-08T12:36:58.199 [INFO] Running 
org.apache.beam.runners.core.triggers.AfterProcessingTimeStateMachineTest
2018-02-08T12:36:58.212 [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.001 s - in 
org.apache.beam.runners.core.triggers.AfterProcessingTimeStateMachineTest
2018-02-08T12:36:58.233 [INFO] Running 
org.apache.beam.runners.core.triggers.AfterPaneStateMachineTest
2018-02-08T12:36:58.236 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in 
org.apache.beam.runners.core.triggers.AfterPaneStateMachineTest
2018-02-08T12:36:58.238 [INFO] Running 
org.apache.beam.runners.core.triggers.AfterSynchronizedProcessingTimeStateMachineTest
2018-02-08T12:36:58.241 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0 s - in 
org.apache.beam.runners.core.triggers.AfterSynchronizedProcessingTimeStateMachineTest
2018-02-08T12:36:58.852 [INFO] 
2018-02-08T12:36:58.852 [INFO] Results:
2018-02-08T12:36:58.852 [INFO] 
2018-02-08T12:36:58.852 [INFO] Tests run: 239, Failures: 0, Errors: 0, Skipped: 0
2018-02-08T12:36:58.852 [INFO] 
[JENKINS] Recording test results
2018-02-08T12:36:59.430 [INFO] 
2018-02-08T12:36:59.430 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-runners-core-java ---
2018-02-08T12:36:59.543 [INFO] 
2018-02-08T12:36:59.543 [INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ 
beam-runners-core-java ---
2018-02-08T12:36:59.545 [INFO] Loading execution data file 

2018-02-08T12:36:59.680 [INFO] Analyzed bundle 'Apache Beam :: Runners :: Core 
Java' with 173 classes
2018-02-08T12:37:00.977 [INFO] 
2018-02-08T12:37:00.977 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-runners-core-java ---
2018-02-08T12:37:01.055 [INFO] Building jar: 

2018-02-08T12:37:01.241 [INFO] 
2018-02-08T12:37:01.241 [INFO] --- maven-site-plugin:3.7:attach-descriptor 
(attach-descriptor) @ beam-runners-core-java ---
2018-02-08T12:37:01.242 [INFO] Skipping because packaging 'jar' is not pom.
2018-02-08T12:37:01.456 [INFO] 
2018-02-08T12:37:01.457 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-runners-core-java ---
2018-02-08T12:37:01.494 [INFO] Building jar: 

Build failed in Jenkins: beam_PerformanceTests_Python #889

[Apache Jenkins Server]

See 


Changes:

[kedin] [Schema Generation] Generate BeamRecordTypes based on pojos.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6792846069763770bbfd92a5eb61e64968522967 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6792846069763770bbfd92a5eb61e64968522967
Commit message: "Merge pull request #4204: [BEAM-3157] Generate BeamRecord 
types from Pojos"
 > git rev-list f2c87bd385da2399938d6ff414176ea953c3c776 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4492582335843149834.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5170207305639875550.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins302109044703221.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1102699554046592868.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3933460881392256366.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7140448655422899048.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #116

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #119

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3647) Default Coder/Reading Coder From File

[Kishan Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kishan Kumar updated BEAM-3647:
---
Description: 
*Requirement*-: Need to Run Template With Same Logics on Different Tables 
Data.(Example is Given Below)

 

*Need*: Default Coder is Required So According to Data It Make All Fields as 
String and Read Data else Thier must be Dynamic Options to Read Coder From GCS 
as JSON FILE and Parse Data on Basis of That (But We can Pass Location Using 
ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider.

 

 

*Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, SUB_PRICE)

And 2 Table is (NAME, ROLL, SUB, TEST_MARKS)

 

On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can 
Read Coder at Run Time The Same Template Can Be Used For Different Tables at 
Run Time.

 

Such Situations Make Our Work Easy and Make Our job Easy.

 

  was:
*Requirement*-: Need to Run Template With Same Logics on Different Tables 
Data.(Example is Given Below)

 

*Need*: Default Coder is Required So According to Data It Make All Fields as 
String and Read Data else Thier must be Dynamic Options to Read Coder From GCS 
or SomeWhere Else so At Runtime Using ValueProvider or Something Else We Can 
Change Coder for Data Whose Query are Same For Common Fields But Data is 
Different 

 

 

*Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, SUB_PRICE)

And 2 Table is (NAME, ROLL, SUB, TEST_MARKS)

 

On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can 
Read Coder at Run Time The Same Template Can Be Used For Different Tables at 
Run Time.

 

Such Situations Make Our Work Easy and Make Our job Easy.

 


> Default Coder/Reading Coder From File 
> --
>
> Key: BEAM-3647
> URL: https://issues.apache.org/jira/browse/BEAM-3647
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: newbie
>
> *Requirement*-: Need to Run Template With Same Logics on Different Tables 
> Data.(Example is Given Below)
>  
> *Need*: Default Coder is Required So According to Data It Make All Fields as 
> String and Read Data else Thier must be Dynamic Options to Read Coder From 
> GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location 
> Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider.
>  
>  
> *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, 
> SUB_PRICE)
> And 2 Table is (NAME, ROLL, SUB, TEST_MARKS)
>  
> On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can 
> Read Coder at Run Time The Same Template Can Be Used For Different Tables at 
> Run Time.
>  
> Such Situations Make Our Work Easy and Make Our job Easy.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3647) Default Coder/Reading Coder From File

[Kishan Kumar (JIRA)]

Kishan Kumar created BEAM-3647:
--

 Summary: Default Coder/Reading Coder From File 
 Key: BEAM-3647
 URL: https://issues.apache.org/jira/browse/BEAM-3647
 Project: Beam
  Issue Type: New Feature
  Components: beam-model, dsl-sql
Affects Versions: 2.2.0
Reporter: Kishan Kumar
Assignee: Kenneth Knowles


*Requirement*-: Need to Run Template With Same Logics on Different Tables 
Data.(Example is Given Below)

 

*Need*: Default Coder is Required So According to Data It Make All Fields as 
String and Read Data else Thier must be Dynamic Options to Read Coder From GCS 
or SomeWhere Else so At Runtime Using ValueProvider or Something Else We Can 
Change Coder for Data Whose Query are Same For Common Fields But Data is 
Different 

 

 

*Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, SUB_PRICE)

And 2 Table is (NAME, ROLL, SUB, TEST_MARKS)

 

On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can 
Read Coder at Run Time The Same Template Can Be Used For Different Tables at 
Run Time.

 

Such Situations Make Our Work Easy and Make Our job Easy.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3637) HBaseIOTest methods do not clean up tables

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356710#comment-16356710
 ] 

Ismaël Mejía commented on BEAM-3637:


Ok just reassigned to Alexey given that he knows well HBase that should be an 
easy one. Only worry here is that deleting the tables will make the tests 
longer, but we'll see.

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3637) HBaseIOTest methods do not clean up tables

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3637:
--

Assignee: Alexey Romanenko  (was: Jean-Baptiste Onofré)

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-673) Data locality for Read.Bounded

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-673:
--
Issue Type: Improvement  (was: Bug)

> Data locality for Read.Bounded
> --
>
> Key: BEAM-673
> URL: https://issues.apache.org/jira/browse/BEAM-673
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Ismaël Mejía
>Priority: Major
>
> In some distributed filesystems, such as HDFS, we should be able to hint to 
> Spark the preferred locations of splits.
> Here is an example of how Spark does that for Hadoop RDDs:
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3592) Spark-runner profile is broken on Nexmark after move to Spark 2.x

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3592.

   Resolution: Fixed
Fix Version/s: 2.3.0

> Spark-runner profile is broken on Nexmark after move to Spark 2.x
> -
>
> Key: BEAM-3592
> URL: https://issues.apache.org/jira/browse/BEAM-3592
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.3.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 2.3.0
>
>
> I found this issue by testing release 2.3.0 with Nexmark. It breaks because 
> of a versioning problem of the netty dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)