Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4892

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5877

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3398

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #107

[Apache Jenkins Server]

See 


--
[...truncated 15.51 KB...]
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 

 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5916002223588525032.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/107/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-06 06:00:35,281 4b1cd6db MainThread INFO Verbose logging to: 

2018-02-06 06:00:35,282 4b1cd6db MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-06 06:00:35,282 4b1cd6db MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/107/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.tfrecordioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-06 06:00:35,569 4b1cd6db MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-06 06:00:35,569 4b1cd6db MainThread INFO 

Build failed in Jenkins: beam_PerformanceTests_Python #880

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8e2092ba28ad31517d65b3085840e6e9a46c83bf (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e2092ba28ad31517d65b3085840e6e9a46c83bf
Commit message: "Merge pull request #4529 from [BEAM-3566] Replace apply_* 
hooks in DirectRunner with PTransformOverrides"
 > git rev-list 8e2092ba28ad31517d65b3085840e6e9a46c83bf # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8050750355974562083.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4583841911990687607.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8261732458401309699.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2058054755165199310.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/aa/64/ea9f0f022bb9660b8a3c549754bf9105e05284ec408e93a65644522c1ecc/setuptools-38.5.0-py2.py3-none-any.whl#md5=f9582f22c550e37a0aa30a0dd2f72661
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins682193667744957802.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6282038092927760416.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))

Build failed in Jenkins: beam_PostCommit_Python_Verify #4153

[Apache Jenkins Server]

See 


--
[...truncated 1.52 MB...]
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #835

[Apache Jenkins Server]

See 


--
[...truncated 130.35 KB...]
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 807, in save_code
pickler.save_reduce(CodeType, args, obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 807, in save_code
pickler.save_reduce(CodeType, args, obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 489, in save_string
self.memoize(obj)
  File "/usr/lib/python2.7/pickle.py", line 246, in memoize
self.write(self.put(memo_len))
  File "/usr/lib/python2.7/pickle.py", line 250, in put
def put(self, i, pack=struct.pack):
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 364, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 377, in run
return self.runner.run_pipeline(self)
  File 
"
 line 283, in run_pipeline
proto_pipeline = pipeline.to_runner_api()
  File 
"
 line 569, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 

[jira] [Created] (BEAM-3621) HCatalog failing frequently on fetch of org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3621:
-

 Summary: HCatalog failing frequently on fetch of 
org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde
 Key: BEAM-3621
 URL: https://issues.apache.org/jira/browse/BEAM-3621
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Assignee: Reuven Lax


The download seems to fail. There are three aspects to this:

 - The Gradle build should have a local persistent cache so it isn't fetching. 
But as I understand it Gradle has a policy of still pinging the repo even when 
cached.
 - The Gradle build is probably re-fetching due to {{--rerun-tasks}} so we need 
to stop doing that.
 - The artifact is not in "Central" repository but in "Spring Plugins". That is 
probably why all the failures are on this artifact and no other artifacts cause 
failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3228:
-

Assignee: Ismaël Mejía  (was: Kenneth Knowles)

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Ismaël Mejía
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> 

[jira] [Updated] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3228:
--
Priority: Critical  (was: Major)

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
>   at 

[jira] [Assigned] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3228:
-

Assignee: Kenneth Knowles

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> 

[jira] [Updated] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3228:
--
Labels: flake  (was: )

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Priority: Major
>  Labels: flake
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
>   at 

[jira] [Commented] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353283#comment-16353283
 ] 

Kenneth Knowles commented on BEAM-3228:
---

I haven't seen this one in a bit.

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> 

[jira] [Resolved] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3228.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> 

[jira] [Assigned] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3598:
-

Assignee: Kenneth Knowles  (was: Reuven Lax)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3605:
-

Assignee: Kenneth Knowles  (was: Paweł Kaczmarczyk)

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard 

[jira] [Assigned] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3599:
-

Assignee: Kenneth Knowles  (was: Reuven Lax)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3317:
-

Assignee: Kenneth Knowles  (was: Reuven Lax)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3317:
--
Labels: flake sickbay  (was: sickbay)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Major
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3317:
--
Priority: Critical  (was: Major)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3317:
--
Summary: KinesisReaderTest is Flaky due to overadvanced watermarks  (was: 
KenisisReaderTest is Flaky due to overadvanced watermarks)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Major
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3598:
--
Priority: Critical  (was: Major)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3598:
--
Labels: flake  (was: )

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3599:
--
Priority: Critical  (was: Major)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3599:
--
Labels: flake  (was: )

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Major
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5876

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3397

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3620) Deprecate older Kafka clients

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-3620:
--

 Summary: Deprecate older Kafka clients
 Key: BEAM-3620
 URL: https://issues.apache.org/jira/browse/BEAM-3620
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Affects Versions: 2.3.0
Reporter: Raghu Angadi
Assignee: Raghu Angadi
 Fix For: 2.4.0


Kafka versions 0.10.1 and above have better support for timestamps. We intend 
to deprecate older versions as a first step towards dropping the support 
entirely.

The older versions (0.9.x and 0.10.0) at run time still work, but KafkaIO 
itself does not support these at build time. It lets us remove some of the 
reflection based code used to support older versions at build time. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3526) Support for checkpointMark finalize in KafkaIO

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-3526:
---
Fix Version/s: 2.4.0

> Support for checkpointMark finalize in KafkaIO
> --
>
> Key: BEAM-3526
> URL: https://issues.apache.org/jira/browse/BEAM-3526
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>
> Users have been asking for option to commit offsets back to Kafka inside 
> `CheckpointMark.finalizeCheckpoint()'. This option is some what better than 
> setting 'AUTO_COMMIT' in Kafka consumer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3526) Support for checkpointMark finalize in KafkaIO

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi closed BEAM-3526.
--
Resolution: Fixed

> Support for checkpointMark finalize in KafkaIO
> --
>
> Key: BEAM-3526
> URL: https://issues.apache.org/jira/browse/BEAM-3526
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>
> Users have been asking for option to commit offsets back to Kafka inside 
> `CheckpointMark.finalizeCheckpoint()'. This option is some what better than 
> setting 'AUTO_COMMIT' in Kafka consumer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4152

[Apache Jenkins Server]

See 


Changes:

[ccy] [BEAM-3566] Replace apply_* hooks in DirectRunner with

[ccy] Address reviewer comments

--
[...truncated 1.52 MB...]
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Giving up after 4 attempts.
- >> end captured logging << -

==
ERROR: test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest)
--
Traceback (most recent call last):
  File 
"
 line 148, in test_gbk_execution
p.run()
  File 
"
 line 104, in run
state = result.wait_until_finish()
  File 
"
 line 318, in wait_until_finish
self._executor.await_completion()
  File 
"
 line 358, in await_completion
self._executor.await_completion()
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4889

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #879

[Apache Jenkins Server]

See 


Changes:

[Pablo] Moving User metrics to be in the PTransform proto for Fn API.

[ccy] [BEAM-3566] Replace apply_* hooks in DirectRunner with

[ccy] Address reviewer comments

[jiangkai] Covariance Functions

[robertwb] [BEAM-3183] Allow a callable as input to runner.run().

[sidhom] Fix gradle java sdk image build

[kenn] Add MoreFutures utility

[kenn] Switch runners/java-fn-execution from Guava futures to Java 8 futures

[kenn] Switch DataflowRunner from Guava futures to Java 8 futures

[kenn] Switch gcp-core from Guava futures to Java 8 futures

[kenn] Switch runners/core-construction-java from Guava futures to Java 8

[kenn] Switch AWS IO from Guava futures to Java 8 futures

[kenn] Switch BigTableIO from Guava futures to Java 8 futures

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8e2092ba28ad31517d65b3085840e6e9a46c83bf (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e2092ba28ad31517d65b3085840e6e9a46c83bf
Commit message: "Merge pull request #4529 from [BEAM-3566] Replace apply_* 
hooks in DirectRunner with PTransformOverrides"
 > git rev-list 2aca6ce2b69d76b9c1f8127d8647f9b1c7589219 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8695652388871773062.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8584536492190967477.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6593059882232593754.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1460524177545159586.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6518771498913675996.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8705279625347698268.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) 

[jira] [Updated] (BEAM-3146) Fn Execution APIs should not depend on the Java SDK

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-3146:
--
Description: 
Runners should be able to interact with the Fn API with complete ignorance of 
the underlying SDK, and as such should not obtain a dependency on the Java SDK.

This dependency edge should be banned. This will also require moving shared 
model abstractions which require code execution to a module on which both fn 
execution APIs and the Java SDK can depend on.

NOTE: the current state of this change requires either factoring of Coder, 
BoundedWindow, and associated runner-relevant code to a common location. 
Currently, these dependency edges exist to reduce work required to encode and 
decode these elements and minimize duplication between the runner and SDK for 
model-level concepts.

  was:
Runners should be able to interact with the Fn API with complete ignorance of 
the underlying SDK, and as such should not obtain a dependency on the Java SDK.

This dependency edge should be banned. This will also require moving shared 
model abstractions which require code execution to a module on which both fn 
execution APIs and the Java SDK can depend on.


> Fn Execution APIs should not depend on the Java SDK
> ---
>
> Key: BEAM-3146
> URL: https://issues.apache.org/jira/browse/BEAM-3146
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Runners should be able to interact with the Fn API with complete ignorance of 
> the underlying SDK, and as such should not obtain a dependency on the Java 
> SDK.
> This dependency edge should be banned. This will also require moving shared 
> model abstractions which require code execution to a module on which both fn 
> execution APIs and the Java SDK can depend on.
> NOTE: the current state of this change requires either factoring of Coder, 
> BoundedWindow, and associated runner-relevant code to a common location. 
> Currently, these dependency edges exist to reduce work required to encode and 
> decode these elements and minimize duplication between the runner and SDK for 
> model-level concepts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3146) Fn Execution APIs should not depend on the Java SDK

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3146:
-

Assignee: (was: Thomas Groh)

> Fn Execution APIs should not depend on the Java SDK
> ---
>
> Key: BEAM-3146
> URL: https://issues.apache.org/jira/browse/BEAM-3146
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Major
>
> Runners should be able to interact with the Fn API with complete ignorance of 
> the underlying SDK, and as such should not obtain a dependency on the Java 
> SDK.
> This dependency edge should be banned. This will also require moving shared 
> model abstractions which require code execution to a module on which both fn 
> execution APIs and the Java SDK can depend on.
> NOTE: the current state of this change requires either factoring of Coder, 
> BoundedWindow, and associated runner-relevant code to a common location. 
> Currently, these dependency edges exist to reduce work required to encode and 
> decode these elements and minimize duplication between the runner and SDK for 
> model-level concepts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2769) Java SDK support for submitting a Portable Pipeline

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-2769:
-

Assignee: (was: Thomas Groh)

> Java SDK support for submitting a Portable Pipeline
> ---
>
> Key: BEAM-2769
> URL: https://issues.apache.org/jira/browse/BEAM-2769
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Priority: Major
>  Labels: portability
>
> The Java codebase should provide a way to submit a Job to a Job Service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2769) Java SDK support for submitting a Portable Pipeline

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-2769:
--
Summary: Java SDK support for submitting a Portable Pipeline  (was: Java 
SDK support for portable pipelines)

> Java SDK support for submitting a Portable Pipeline
> ---
>
> Key: BEAM-2769
> URL: https://issues.apache.org/jira/browse/BEAM-2769
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>
> The Java codebase should provide a way to submit a Job to a Job Service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2718) Add bundle retry logic to the DirectRunner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

María GH updated BEAM-2718:
---
Fix Version/s: (was: Not applicable)
   2.3.0

> Add bundle retry logic to the DirectRunner
> --
>
> Key: BEAM-2718
> URL: https://issues.apache.org/jira/browse/BEAM-2718
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: Not applicable
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 2.3.0
>
>
> When processing of a bundle fails, the bundle should be retried 3 times (for 
> a total of 4 attempts to process it).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-849) Redesign PipelineResult API

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353111#comment-16353111
 ] 

Thomas Groh commented on BEAM-849:
--

This should be strongly informed by and should in turn inform the structure of 
the Beam Job API, especially the APIs available for interacting with an 
existing job.

> Redesign PipelineResult API
> ---
>
> Key: BEAM-849
> URL: https://issues.apache.org/jira/browse/BEAM-849
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Thomas Groh
>Priority: Major
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses 
> waitUntilFinish() and cancel(). 
> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2430) Java FnApiDoFnRunner to share across runners

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2430.
---
   Resolution: Done
Fix Version/s: 2.3.0

I claim that a {{FnApiDoFnRunner}} is the wrong abstraction. However, a similar 
concept exists in the SdkHarnessClient, BundleProcessor, and ActiveBundle 
abstractions introduced in [https://github.com/apache/beam/pull/4472,] which is 
what clients who wish to use the portability framework should use when sending 
data to a remote Sdk Harness.

> Java FnApiDoFnRunner to share across runners
> 
>
> Key: BEAM-2430
> URL: https://issues.apache.org/jira/browse/BEAM-2430
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.3.0
>
>
> As the portability framework comes into focus, let's fill out the support 
> code for making it easy to onboard a new runner.
> There is some amount of using the Fn API that has to do only with the fact 
> that a runner is implemented in Java, and is not specific to that runner. 
> This should be part of the runners-core library, and designed so that a 
> runner can set it up however it likes, and just pass elements without having 
> to explicitly manage things like requests, responses, protos, and coders.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3619) Add tests for Python DirectRunner implementation of PubSub I/O

[Charles Chen (JIRA)]

Charles Chen created BEAM-3619:
--

 Summary: Add tests for Python DirectRunner implementation of 
PubSub I/O
 Key: BEAM-3619
 URL: https://issues.apache.org/jira/browse/BEAM-3619
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


We should add behavior tests and mocks to the Python DirectRunner 
implementation of PubSub I/O.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3408) PackageUtil should retry check for whether file is already staged

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3408:
-

Assignee: (was: Thomas Groh)

> PackageUtil should retry check for whether file is already staged
> -
>
> Key: BEAM-3408
> URL: https://issues.apache.org/jira/browse/BEAM-3408
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Priority: Major
>
> The code to stage files retries the upload, but the existence (and size) 
> check is attempted only once:
> https://github.com/apache/beam/blob/dc3a664e3b4b4f63500954aae25dce0cb5f8b314/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L193
> This causes post-commit tests to fail sometimes:
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4632/testReport/junit/org.apache.beam.sdk.io/BoundedReadFromUnboundedSourceTest/testDedup/
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4636/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyMultimapSideInput/
> We should retry this check as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3318) ConcurrentModificationException in populating Display Data

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3318:
-

Assignee: (was: Thomas Groh)

> ConcurrentModificationException in populating Display Data
> --
>
> Key: BEAM-3318
> URL: https://issues.apache.org/jira/browse/BEAM-3318
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, sdk-java-core
>Reporter: Thomas Groh
>Priority: Major
>
> Example: 
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/16088/testReport/junit/org.apache.beam.examples/WindowedWordCountIT/testWindowedWordCountInBatchDynamicSharding/
> (Link is to a successful execution, but there is a stacktrace present within 
> the build)
> 2017-12-07T20:12:36.484 [ERROR] 
> testWindowedWordCountInBatchDynamicSharding(org.apache.beam.examples.WindowedWordCountIT)
>   Time elapsed: 5.833 s  <<< ERROR!
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder$PopulateDisplayDataException:
>  Error while populating display data for component 
> 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData':
>  null
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:802)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.delegate(DisplayData.java:768)
>   at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:145)
>   at com.sun.proxy.$Proxy30.populateDisplayData(Unknown Source)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:794)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.access$100(DisplayData.java:733)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData.from(DisplayData.java:81)
>   at 
> org.apache.beam.runners.direct.DisplayDataValidator.evaluateDisplayData(DisplayDataValidator.java:47)
>   at 
> org.apache.beam.runners.direct.DisplayDataValidator.validateOptions(DisplayDataValidator.java:39)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:180)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:63)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:214)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:187)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchDynamicSharding(WindowedWordCountIT.java:94)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
>   at 
> 

[jira] [Commented] (BEAM-3529) Side Input callbacks should fire after a window has expired in addition to when the trigger would fire

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353096#comment-16353096
 ] 

Thomas Groh commented on BEAM-3529:
---

Yes; note that the simple test implementation uses {{TestStream}}, so there's 
limited coverage by default, but it is certainly worth writing.

> Side Input callbacks should fire after a window has expired in addition to 
> when the trigger would fire
> --
>
> Key: BEAM-3529
> URL: https://issues.apache.org/jira/browse/BEAM-3529
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Otherwise a streaming pipeline where elements trying to access a window that 
> has expired will not progress, even though we can be certain that there will 
> never be inputs in that window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3528) Populate Environment in Side Input SDKFunctionSpecs in ParDoTranslation

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3528:
-

Assignee: (was: Thomas Groh)

> Populate Environment in Side Input SDKFunctionSpecs in ParDoTranslation
> ---
>
> Key: BEAM-3528
> URL: https://issues.apache.org/jira/browse/BEAM-3528
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3564) Handle an Unpopulated "Environment" field in Environments

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-3564.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Handle an Unpopulated "Environment" field in Environments
> -
>
> Key: BEAM-3564
> URL: https://issues.apache.org/jira/browse/BEAM-3564
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: Not applicable
>
>
> The field may be present and unset (for example, Window Into the Global 
> Windows need not set an environment, but is permitted to). This should 
> succeed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8e2092ba28ad31517d65b3085840e6e9a46c83bf
Merge: 8edc18e e1a96eb
Author: Robert Bradshaw 
AuthorDate: Mon Feb 5 15:32:09 2018 -0800

Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in 
DirectRunner with PTransformOverrides

[BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides

 sdks/python/apache_beam/io/gcp/pubsub_test.py  |  58 -
 sdks/python/apache_beam/pipeline.py|  34 +--
 sdks/python/apache_beam/pipeline_test.py   |   9 +-
 .../runners/dataflow/dataflow_runner_test.py   |   4 +-
 .../runners/dataflow/ptransform_overrides.py   |   6 +-
 .../apache_beam/runners/direct/direct_runner.py| 234 -
 .../runners/direct/helper_transforms.py|   8 +-
 .../runners/direct/sdf_direct_runner.py|   9 +-
 sdks/python/apache_beam/runners/sdf_common.py  |  15 +-
 sdks/python/apache_beam/transforms/combiners.py|  39 ++--
 10 files changed, 253 insertions(+), 163 deletions(-)

diff --cc sdks/python/apache_beam/pipeline.py
index 4c48428,c59a29a..f164bf5
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@@ -61,8 -61,8 +61,9 @@@ from apache_beam.options.pipeline_optio
  from apache_beam.options.pipeline_options import StandardOptions
  from apache_beam.options.pipeline_options import TypeOptions
  from apache_beam.options.pipeline_options_validator import 
PipelineOptionsValidator
 +from apache_beam.portability import common_urns
  from apache_beam.pvalue import PCollection
+ from apache_beam.pvalue import PDone
  from apache_beam.runners import PipelineRunner
  from apache_beam.runners import create_runner
  from apache_beam.transforms import ptransform

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (8edc18e -> 8e2092b)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8edc18e  Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures
 add 16e5fef  [BEAM-3566] Replace apply_* hooks in DirectRunner with 
PTransformOverrides
 add e1a96eb  Address reviewer comments
 new 8e2092b  Merge pull request #4529 from [BEAM-3566] Replace apply_* 
hooks in DirectRunner with PTransformOverrides

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/gcp/pubsub_test.py  |  58 -
 sdks/python/apache_beam/pipeline.py|  34 +--
 sdks/python/apache_beam/pipeline_test.py   |   9 +-
 .../runners/dataflow/dataflow_runner_test.py   |   4 +-
 .../runners/dataflow/ptransform_overrides.py   |   6 +-
 .../apache_beam/runners/direct/direct_runner.py| 234 -
 .../runners/direct/helper_transforms.py|   8 +-
 .../runners/direct/sdf_direct_runner.py|   9 +-
 sdks/python/apache_beam/runners/sdf_common.py  |  15 +-
 sdks/python/apache_beam/transforms/combiners.py|  39 ++--
 10 files changed, 253 insertions(+), 163 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4888

[Apache Jenkins Server]

See 

[beam] branch master updated (b812fa0 -> 8edc18e)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b812fa0  Merge pull request #4544: [BEAM-3563] Adding User metric 
proto to PTransform metrics proto.
 add b5dd1c1  Add MoreFutures utility
 add e71baf0  Switch runners/java-fn-execution from Guava futures to Java 8 
futures
 add c757854  Switch DataflowRunner from Guava futures to Java 8 futures
 add 6bb6530  Switch gcp-core from Guava futures to Java 8 futures
 add f64a633  Switch runners/core-construction-java from Guava futures to 
Java 8 futures
 add 0f19c32  Switch AWS IO from Guava futures to Java 8 futures
 add b1fd123  Switch BigTableIO from Guava futures to Java 8 futures
 new 8edc18e  Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/ArtifactServiceStager.java   |  32 +--
 .../beam/runners/dataflow/util/PackageUtil.java|  60 ++---
 .../fnexecution/control/FnApiControlClient.java|  35 ++-
 .../fnexecution/control/SdkHarnessClient.java  |  34 ++-
 .../control/FnApiControlClientPoolServiceTest.java |   9 +-
 .../control/FnApiControlClientTest.java|  30 +--
 .../fnexecution/control/SdkHarnessClientTest.java  |  17 +-
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 251 +
 .../apache/beam/sdk/util/ThrowingRunnable.java}|   8 +-
 .../util/{Weighted.java => ThrowingSupplier.java}  |  14 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java |  21 +-
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java|  10 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   |  32 +--
 .../beam/sdk/io/gcp/bigtable/BigtableService.java  |   4 +-
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |  24 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   |  14 +-
 .../io/gcp/bigtable/BigtableServiceImplTest.java   |   9 +-
 17 files changed, 427 insertions(+), 177 deletions(-)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/util/MoreFutures.java
 copy 
sdks/java/{extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/package-info.java
 => core/src/main/java/org/apache/beam/sdk/util/ThrowingRunnable.java} (84%)
 copy sdks/java/core/src/main/java/org/apache/beam/sdk/util/{Weighted.java => 
ThrowingSupplier.java} (81%)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8edc18e8a91c0b65f124ac33d4c3f67c9678fc98
Merge: b812fa0 b1fd123
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 14:32:31 2018 -0800

Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures

 .../core/construction/ArtifactServiceStager.java   |  32 +--
 .../beam/runners/dataflow/util/PackageUtil.java|  60 ++---
 .../fnexecution/control/FnApiControlClient.java|  35 ++-
 .../fnexecution/control/SdkHarnessClient.java  |  34 ++-
 .../control/FnApiControlClientPoolServiceTest.java |   9 +-
 .../control/FnApiControlClientTest.java|  30 +--
 .../fnexecution/control/SdkHarnessClientTest.java  |  17 +-
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 251 +
 .../org/apache/beam/sdk/util/ThrowingRunnable.java |  24 ++
 .../org/apache/beam/sdk/util/ThrowingSupplier.java |  26 +++
 .../java/org/apache/beam/sdk/util/GcsUtil.java |  21 +-
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java|  10 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   |  32 +--
 .../beam/sdk/io/gcp/bigtable/BigtableService.java  |   4 +-
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |  24 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   |  14 +-
 .../io/gcp/bigtable/BigtableServiceImplTest.java   |   9 +-
 17 files changed, 465 insertions(+), 167 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4544: [BEAM-3563] Adding User metric proto to PTransform metrics proto.

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b812fa061f0ca09f343ca2111444e585520594b5
Merge: 7904112 2de70ff
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 14:22:53 2018 -0800

Merge pull request #4544: [BEAM-3563] Adding User metric proto to 
PTransform metrics proto.

 model/fn-execution/src/main/proto/beam_fn_api.proto |  9 +++--
 sdks/python/apache_beam/metrics/execution.py| 15 ++-
 sdks/python/apache_beam/metrics/metricbase.py   | 10 ++
 .../runners/portability/fn_api_runner.py| 21 -
 .../apache_beam/runners/worker/bundle_processor.py  |  5 +
 .../python/apache_beam/runners/worker/operations.py |  3 ++-
 6 files changed, 30 insertions(+), 33 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (7904112 -> b812fa0)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7904112  Merge pull request #4590 [BEAM-3183] Allow a callable as 
input to runner.run().
 add 2de70ff  Moving User metrics to be in the PTransform proto for Fn API.
 new b812fa0  Merge pull request #4544: [BEAM-3563] Adding User metric 
proto to PTransform metrics proto.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 model/fn-execution/src/main/proto/beam_fn_api.proto |  9 +++--
 sdks/python/apache_beam/metrics/execution.py| 15 ++-
 sdks/python/apache_beam/metrics/metricbase.py   | 10 ++
 .../runners/portability/fn_api_runner.py| 21 -
 .../apache_beam/runners/worker/bundle_processor.py  |  5 +
 .../python/apache_beam/runners/worker/operations.py |  3 ++-
 6 files changed, 30 insertions(+), 33 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Assigned] (BEAM-3320) DataflowRunner for python streaming uses portable containers

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3320:
-

Assignee: Alan Myrvold  (was: Ahmet Altay)

> DataflowRunner for python streaming uses portable containers
> 
>
> Key: BEAM-3320
> URL: https://issues.apache.org/jira/browse/BEAM-3320
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: portability
>
> Python streaming uses Dataflow built containers to run on Dataflow. It should 
> start using the portable containers to take advantage of the changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3543) Fn API metrics in Java SDK Harness

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3543:
-

Assignee: (was: Kenneth Knowles)

> Fn API metrics in Java SDK Harness
> --
>
> Key: BEAM-3543
> URL: https://issues.apache.org/jira/browse/BEAM-3543
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3543) Fn API metrics in Java SDK Harness

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3543:
--
Labels: portability  (was: )

> Fn API metrics in Java SDK Harness
> --
>
> Key: BEAM-3543
> URL: https://issues.apache.org/jira/browse/BEAM-3543
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3608) Pre-shade Guava for things we want to keep using

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3608:
--
Labels: portability  (was: )

> Pre-shade Guava for things we want to keep using
> 
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> Instead of shading as part of our build, we can shade before build so that it 
> is apparent when reading code, and in IDEs, that a particular class resides 
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs 
> unless they are annotated {{@Internal}}, and it makes sharing between our own 
> modules never get broken by shading again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3601) Switch to Java 8 futures

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3601:
--
Labels: portability  (was: )

> Switch to Java 8 futures
> 
>
> Key: BEAM-3601
> URL: https://issues.apache.org/jira/browse/BEAM-3601
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4887

[Apache Jenkins Server]

See 

[beam] branch master updated (cb38392 -> 7904112)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cb38392  Match Maven Java SDK harness jar name format
 add 4d06e82  [BEAM-3183] Allow a callable as input to runner.run().
 new 7904112  Merge pull request #4590 [BEAM-3183] Allow a callable as 
input to runner.run().

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/runner.py  |  9 +++--
 sdks/python/apache_beam/runners/runner_test.py | 14 ++
 2 files changed, 21 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4590 [BEAM-3183] Allow a callable as input to runner.run().

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 79041126519ad64c4f357fb8ddcc7ae67a7b2203
Merge: cb38392 4d06e82
Author: Robert Bradshaw 
AuthorDate: Mon Feb 5 12:15:35 2018 -0800

Merge pull request #4590 [BEAM-3183] Allow a callable as input to 
runner.run().

 sdks/python/apache_beam/runners/runner.py  |  9 +++--
 sdks/python/apache_beam/runners/runner_test.py | 14 ++
 2 files changed, 21 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Commented] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352829#comment-16352829
 ] 

Kenneth Knowles commented on BEAM-3617:
---

Good idea to check this. I added it to the spreadsheet and signed you up JB. I 
only added direct runner since other runners should opt in.

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.3.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (d13c1fe -> cb38392)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d13c1fe  Merge pull request #4466: [BEAM-3476] [SQL] covariance 
aggregation functions
 add 17e41c3  Fix gradle java sdk image build
 new cb38392  Match Maven Java SDK harness jar name format

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/container/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Match Maven Java SDK harness jar name format

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit cb383921f8c45cf88c123f893b881ae44ccc75bf
Merge: d13c1fe 17e41c3
Author: Lukasz Cwik 
AuthorDate: Mon Feb 5 11:36:44 2018 -0800

Match Maven Java SDK harness jar name format

 sdks/java/container/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Commented] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352808#comment-16352808
 ] 

Kenneth Knowles commented on BEAM-3414:
---

Those instructions are not quite right. The JStorm runner is a work in progress 
on a development branch, and not released with the nightly build either. You 
will need to check out that branch and install it locally. I should say that 
the JStorm runner has not been updated in a very long time.

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Aljoscha Krettek
>Priority: Major
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3617) Performance degradation on the direct runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352800#comment-16352800
 ] 

Jean-Baptiste Onofré commented on BEAM-3617:


I just ran nexmark ({{mvn exec:java 
-Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Dexec.args="--suite=SMOKE 
--streaming=true --manageResources=false --monitorJobs=true"}}):

* On Beam 2.2.0

{code}
Performance:
  Conf  Runtime(sec)(Baseline)  Events(/sec)(Baseline)   Results
(Baseline)
     8.1 12335.0  10
  0001   4.0 25106.7   92000
  0002   2.2 46082.9 351
  0003   4.6 21767.5 444
  0004   1.4  7092.2  40
  0005   5.9 16869.1  12
  0006   1.0 10111.2 401
  0007 153.2   653.0   1
  0008   3.3 30413.66000
  0009   0.9 1.1 298
  0010   5.2 19241.9   1
  0011   4.3 23153.51919
  0012   3.3 30712.51919
==
{code}

* On Beam 2.3.0 (release branch):

{code}
Performance:
  Conf  Runtime(sec)(Baseline)  Events(/sec)(Baseline)   Results
(Baseline)
    10.5  9554.7  10
  0001   7.2 13848.5   92000
  0002   3.9 25654.2 351
  0003   5.9 17059.0 444
  0004   1.7  6013.2  40
  0005   8.4 11899.1  12
  0006   1.4  7077.1 401
  00071019.098.1   1
  0008   5.0 19888.66000
  0009   1.3  7905.1 298
  0010   6.2 16186.5   1
  0011   9.0 11088.91919
  0012   6.4 15535.21919
==
{code}

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.3.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (2aca6ce -> d13c1fe)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2aca6ce  BEAM-3618 - Remove extraneous "return" statement
 add 94fd067  Covariance Functions
 new d13c1fe  Merge pull request #4466: [BEAM-3476] [SQL] covariance 
aggregation functions

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/extensions/sql/pom.xml   |   3 +
 .../impl/transform/BeamAggregationTransforms.java  |  90 ++--
 .../impl/transform/agg/CovarianceAccumulator.java  | 151 +
 .../sql/impl/transform/agg/CovarianceFn.java   | 120 
 .../agg => utils}/BigDecimalConverter.java |   2 +-
 .../sql/BeamSqlDslAggregationCovarianceTest.java   | 116 
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  50 ---
 .../agg => utils}/BigDecimalConverterTest.java |   2 +-
 8 files changed, 470 insertions(+), 64 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceAccumulator.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
 rename 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{transform/agg
 => utils}/BigDecimalConverter.java (97%)
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
 rename 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/{transform/agg
 => utils}/BigDecimalConverterTest.java (96%)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4466: [BEAM-3476] [SQL] covariance aggregation functions

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d13c1feeaeeec4af079f2da523849c218cc50882
Merge: 2aca6ce 94fd067
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 11:10:36 2018 -0800

Merge pull request #4466: [BEAM-3476] [SQL] covariance aggregation functions

 sdks/java/extensions/sql/pom.xml   |   3 +
 .../impl/transform/BeamAggregationTransforms.java  |  90 ++--
 .../impl/transform/agg/CovarianceAccumulator.java  | 151 +
 .../sql/impl/transform/agg/CovarianceFn.java   | 120 
 .../agg => utils}/BigDecimalConverter.java |   2 +-
 .../sql/BeamSqlDslAggregationCovarianceTest.java   | 116 
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  50 ---
 .../agg => utils}/BigDecimalConverterTest.java |   2 +-
 8 files changed, 470 insertions(+), 64 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Assigned] (BEAM-3182) [Nexmark][SQL] Implement supported queries

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud reassigned BEAM-3182:


Assignee: Andrew Pilloud  (was: Anton Kedin)

> [Nexmark][SQL] Implement supported queries
> --
>
> Key: BEAM-3182
> URL: https://issues.apache.org/jira/browse/BEAM-3182
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Andrew Pilloud
>Priority: Major
>
> Implement all queries which can be run with current SQL features and Nexmark 
> infrastructure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #878

[Apache Jenkins Server]

See 


Changes:

[lcwik] Adjust gradle build dirs and hints to help IntelliJ (#4583)

[coheigea] BEAM-3618 - Remove extraneous "return" statement

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 2aca6ce2b69d76b9c1f8127d8647f9b1c7589219 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2aca6ce2b69d76b9c1f8127d8647f9b1c7589219
Commit message: "BEAM-3618 - Remove extraneous "return" statement"
 > git rev-list 93818abf901dd996aa30445457153811b13ece43 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9062689376651431394.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7049756771144757052.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7059146757956175519.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4007753776212593603.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins598886307047372180.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2497797300101069930.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4886

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #237

[Apache Jenkins Server]

See 


Changes:

[lcwik] Adjust gradle build dirs and hints to help IntelliJ (#4583)

[coheigea] BEAM-3618 - Remove extraneous "return" statement

--
[...truncated 29.32 KB...]
[INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded 
jar.
[INFO] Excluding org.apache.commons:commons-compress:jar:1.14 from the shaded 
jar.
[INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
[INFO] Excluding joda-time:joda-time:jar:2.4 from the shaded jar.
[INFO] Including com.google.guava:guava:jar:20.0 in the shaded jar.
[INFO] Excluding com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-dbcp2:jar:2.1.1 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-pool2:jar:2.4.2 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.4.0-SNAPSHOT
 from the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev355-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5867

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3618) Remove extraneous "return" statement

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3618.
-
Resolution: Fixed

> Remove extraneous "return" statement
> 
>
> Key: BEAM-3618
> URL: https://issues.apache.org/jira/browse/BEAM-3618
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Colm O hEigeartaigh
>Assignee: Colm O hEigeartaigh
>Priority: Trivial
> Fix For: 2.4.0
>
>
> There is an extraneous "return" statement in one of the tests that could be 
> removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: BEAM-3618 - Remove extraneous "return" statement

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2aca6ce2b69d76b9c1f8127d8647f9b1c7589219
Merge: af7920d 7c19700
Author: Lukasz Cwik 
AuthorDate: Mon Feb 5 09:31:07 2018 -0800

BEAM-3618 - Remove extraneous "return" statement

 .../java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java | 1 -
 1 file changed, 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (af7920d -> 2aca6ce)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from af7920d  Adjust gradle build dirs and hints to help IntelliJ (#4583)
 add 7c19700  BEAM-3618 - Remove extraneous "return" statement
 new 2aca6ce  BEAM-3618 - Remove extraneous "return" statement

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java | 1 -
 1 file changed, 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Created] (BEAM-3618) Remove extraneous "return" statement

[Colm O hEigeartaigh (JIRA)]

Colm O hEigeartaigh created BEAM-3618:
-

 Summary: Remove extraneous "return" statement
 Key: BEAM-3618
 URL: https://issues.apache.org/jira/browse/BEAM-3618
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Colm O hEigeartaigh
Assignee: Colm O hEigeartaigh
 Fix For: 2.4.0


There is an extraneous "return" statement in one of the tests that could be 
removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3602) Add source set for generated Java gRPC code

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3602.
-
   Resolution: Fixed
Fix Version/s: 2.4.0

> Add source set for generated Java gRPC code
> ---
>
> Key: BEAM-3602
> URL: https://issues.apache.org/jira/browse/BEAM-3602
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Ben Sidhom
>Assignee: Kenneth Knowles
>Priority: Trivial
> Fix For: 2.4.0
>
>
> Generated code is not currently exposed to IDEs (either in source or .class 
> form). This leads to spurious compile errors in editors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3602) Add source set for generated Java gRPC code

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352643#comment-16352643
 ] 

Ben Sidhom commented on BEAM-3602:
--

This should be fixed by https://github.com/apache/beam/pull/4583.

> Add source set for generated Java gRPC code
> ---
>
> Key: BEAM-3602
> URL: https://issues.apache.org/jira/browse/BEAM-3602
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Ben Sidhom
>Assignee: Kenneth Knowles
>Priority: Trivial
>
> Generated code is not currently exposed to IDEs (either in source or .class 
> form). This leads to spurious compile errors in editors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3602) Add source set for generated Java gRPC code

[Ben Sidhom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Sidhom reassigned BEAM-3602:


Assignee: Kenneth Knowles  (was: Ben Sidhom)

> Add source set for generated Java gRPC code
> ---
>
> Key: BEAM-3602
> URL: https://issues.apache.org/jira/browse/BEAM-3602
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Ben Sidhom
>Assignee: Kenneth Knowles
>Priority: Trivial
>
> Generated code is not currently exposed to IDEs (either in source or .class 
> form). This leads to spurious compile errors in editors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Adjust gradle build dirs and hints to help IntelliJ (#4583)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new af7920d  Adjust gradle build dirs and hints to help IntelliJ (#4583)
af7920d is described below

commit af7920dce12e1a3c018b90f7c305f7531b6df0d6
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 08:34:12 2018 -0800

Adjust gradle build dirs and hints to help IntelliJ (#4583)
---
 build_rules.gradle| 48 +--
 sdks/java/extensions/sql/build.gradle | 20 +--
 2 files changed, 64 insertions(+), 4 deletions(-)

diff --git a/build_rules.gradle b/build_rules.gradle
index fa691af..c819776 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -67,6 +67,9 @@ repositories {
   maven { url "https://repository.apache.org/content/repositories/releases; }
 }
 
+// Apply a plugin which enables configuring projects imported into Intellij.
+apply plugin: "idea"
+
 // Provide code coverage
 // TODO: Should this only apply to Java projects?
 apply plugin: "jacoco"
@@ -363,6 +366,18 @@ ext.applyJavaNature = {
   force library.java.values()
 }
   }
+
+  def aptGeneratedMain = "${project.buildDir}/generated/source/apt/main"
+  def aptGeneratedTest = "${project.buildDir}/generated/source/apt/test"
+  idea {
+module {
+  sourceDirs += file(aptGeneratedMain)
+  generatedSourceDirs += file(aptGeneratedMain)
+
+  testSourceDirs += file(aptGeneratedTest)
+  generatedSourceDirs += file(aptGeneratedTest)
+}
+  }
 }
 
 
/*/
@@ -400,6 +415,17 @@ ext.applyGoNature = {
   tasks.getByPath(project.path + ":resolveTestDependencies").mustRunAfter 
tasks.getByPath(previous + ":installDependencies")
 }
   }
+
+  idea {
+module {
+  // The gogradle plugin downloads all dependencies into the source tree 
here,
+  // which is a path baked into golang
+  excludeDirs += file("${project.path}/vendor")
+
+  // gogradle's private working directory
+  excludeDirs += file("${project.path}/.gogradle")
+}
+  }
 }
 
 
/*/
@@ -443,6 +469,26 @@ ext.applyGrpcNature = {
   }
 }
   }
+
+  def generatedProtoMainJavaDir = 
"${project.buildDir}/generated/source/proto/main/java"
+  def generatedProtoTestJavaDir = 
"${project.buildDir}/generated/source/proto/test/java"
+  def generatedGrpcMainJavaDir = 
"${project.buildDir}/generated/source/proto/main/grpc"
+  def generatedGrpcTestJavaDir = 
"${project.buildDir}/generated/source/proto/test/grpc"
+  idea {
+module {
+  sourceDirs += file(generatedProtoMainJavaDir)
+  generatedSourceDirs += file(generatedProtoMainJavaDir)
+
+  testSourceDirs += file(generatedProtoTestJavaDir)
+  generatedSourceDirs += file(generatedProtoTestJavaDir)
+
+  sourceDirs += file(generatedProtoMainJavaDir)
+  generatedSourceDirs += file(generatedGrpcMainJavaDir)
+
+  testSourceDirs += file(generatedGrpcTestJavaDir)
+  generatedSourceDirs += file(generatedGrpcTestJavaDir)
+}
+  }
 }
 
 
/*/
@@ -453,5 +499,3 @@ ext.applyAvroNature = {
   println "applyAvroNature with " + (it ? "$it" : "default configuration") + " 
for project $project.name"
   apply plugin: "com.commercehub.gradle.plugin.avro"
 }
-
-
diff --git a/sdks/java/extensions/sql/build.gradle 
b/sdks/java/extensions/sql/build.gradle
index bcca0cf..b7ba79a 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -80,7 +80,7 @@ task copyFmppTemplatesFromCalciteCore(type: Copy) {
 }
 
 // Generate the FMPP sources from the FMPP templates.
-def generateFmppOutputDir = "${project.buildDir}/generated-fmpp"
+def generateFmppOutputDir = "${project.buildDir}/generated/fmpp"
 task generateFmppSources {
   dependsOn configurations.fmppTask
   dependsOn copyFmppTemplatesFromSrc
@@ -91,9 +91,14 @@ task generateFmppSources {
   }
 }
 
+// Match the output directory for generated code with the package, to be more 
tool-friendly
+def generateFmppJavaccRoot = "${generateFmppOutputDir}/javacc"
+def generatedJavaccSourceDir = "${project.buildDir}/generated/javacc"
+def generatedJavaccPackageDir = 
"${generatedJavaccSourceDir}/org/apache/beam/sdk/extensions/sql/impl/parser/impl"
 compileJavacc {
   dependsOn generateFmppSources
-  inputDirectory = file(generateFmppOutputDir)
+  inputDirectory = file(generateFmppJavaccRoot)
+  outputDirectory = file(generatedJavaccPackageDir)
   arguments = [grammar_encoding: "UTF-8", static: "false", lookahead: "2"]
 }
 
@@ -116,3 +121,14 @@ shadowJar {
   // module.
   relocate "org.codehaus", 

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352612#comment-16352612
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


It's not a problem in Beam codebase itself, so not a blocker for the release. 
More a website issue to add a note on the runners (spark, flink, others as they 
all work the same way).

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3587:
---
Fix Version/s: (was: 2.3.0)

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3587:
---
Component/s: (was: runner-flink)
 website

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3587:
---
Priority: Minor  (was: Blocker)

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #236

[Apache Jenkins Server]

See 


--
[...truncated 52.95 KB...]
Error from server (AlreadyExists): error when creating 
":
 replicationcontrollers "postgres" already exists

2018-02-05 16:10:29,349 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 16:10:48,322 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-02-05 16:10:48,497 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

 Got return code (1).
STDOUT: 
STDERR: Error from server (Invalid): error when creating 
":
 Service "postgres" is invalid: spec.ports[0].nodePort: Invalid value: 31234: 
provided port is already allocated
Error from server (AlreadyExists): error when creating 
":
 replicationcontrollers "postgres" already exists

2018-02-05 16:10:48,498 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 16:11:18,273 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-02-05 16:11:18,541 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

 Got return code (1).
STDOUT: 
STDERR: Error from server (Invalid): error when creating 
":
 Service "postgres" is invalid: spec.ports[0].nodePort: Invalid value: 31234: 
provided port is already allocated
Error from server (AlreadyExists): error when creating 
":
 replicationcontrollers "postgres" already exists

2018-02-05 16:11:18,541 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 16:11:46,415 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-02-05 16:11:46,582 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

 Got return code (1).
STDOUT: 
STDERR: Error from server (Invalid): error when creating 
":
 Service "postgres" is invalid: spec.ports[0].nodePort: Invalid value: 31234: 
provided port is already allocated
Error from server (AlreadyExists): error when creating 
":
 replicationcontrollers "postgres" already exists

2018-02-05 16:11:46,582 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 16:12:15,198 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

2018-02-05 16:12:15,368 0f96c487 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config create -f 

 Got return code (1).
STDOUT: 
STDERR: Error from server (Invalid): error when creating 

[jira] [Comment Edited] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352587#comment-16352587
 ] 

Jean-Baptiste Onofré edited comment on BEAM-3587 at 2/5/18 4:10 PM:


As we have:

{code}

org.apache.maven.plugins
maven-shade-plugin
${maven-shade-plugin.version}


false


*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA






package

shade



true
shaded







{code}

for a Maven build, the same has to be applied to Gradle:

{code}
shadowJar {
transform(AppendingTransformer) {
resource = 'reference.conf'
}
relocate 'com.google.protobuf', 'shaded.protobuf'
mergeServiceFiles()
}
{code}

I'm adding a note in the documentation.


was (Author: jbonofre):
As we have:

{code}

org.apache.maven.plugins
maven-shade-plugin
${maven-shade-plugin.version}


false


*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA






package

shade



true
shaded







{code}

for a Maven build, the same has to be applied to Gradle.

I'm adding a note in the documentation.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3587:
---
Attachment: screen1.png

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352590#comment-16352590
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


I created a PR on the user example to fix the issue: 
https://github.com/pelletier/beam-flink-example/pull/1

I also took screenshots that it runs fine on Flink cluster.

I'm adding a note in Beam documentation as some other users might have the same 
issue.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3587:
---
Attachment: screen2.png

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: screen1.png, screen2.png
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352587#comment-16352587
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


As we have:

{code}

org.apache.maven.plugins
maven-shade-plugin
${maven-shade-plugin.version}


false


*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA






package

shade



true
shaded







{code}

for a Maven build, the same has to be applied to Gradle.

I'm adding a note in the documentation.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #235

[Apache Jenkins Server]

See 


--
[...truncated 85.13 KB...]
2018-02-05 15:47:50,986 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 15:48:06,720 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-02-05 15:48:06,812 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: error: the path 
"
 does not exist

2018-02-05 15:48:06,812 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 15:48:31,687 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-02-05 15:48:31,779 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: error: the path 
"
 does not exist

2018-02-05 15:48:31,779 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 15:48:58,236 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-02-05 15:48:58,326 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: error: the path 
"
 does not exist

2018-02-05 15:48:58,326 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 15:49:23,148 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-02-05 15:49:23,244 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: error: the path 
"
 does not exist

2018-02-05 15:49:23,245 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 15:49:47,616 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-02-05 15:49:47,711 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Ran kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: error: the path 
"
 does not exist

2018-02-05 15:49:47,712 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2018-02-05 15:50:13,667 14bbe231 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl --kubeconfig=/home/jenkins/.kube/config delete -f 

2018-02-05 

[jira] [Created] (BEAM-3617) Performance degradation on the direct runner

[JIRA]

Jean-Baptiste Onofré created BEAM-3617:
--

 Summary: Performance degradation on the direct runner
 Key: BEAM-3617
 URL: https://issues.apache.org/jira/browse/BEAM-3617
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Affects Versions: 2.3.0
Reporter: Jean-Baptiste Onofré
Assignee: Thomas Groh
 Fix For: 2.3.0


Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
shows a performance degradation:

{code}

 Beam 2.2.0   Beam 2.3.0
  Query  Runtime(sec) Runtime(sec)

     6.410.6
  0001   5.110.2
  0002   3.0 5.8
  0003   3.8 6.2
  0004   0.9 1.4
  0005   5.811.4
  0006   0.8 1.4
  0007 193.8  1249.1
  0008   3.9 6.9
  0009   0.9 1.3
  0010   6.4 8.2
  0011   5.0 9.4
  0012   4.7 9.1
{code}

We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3587:
--

Assignee: Jean-Baptiste Onofré  (was: Ben Sidhom)

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352564#comment-16352564
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


I cloned the test repo: https://github.com/pelletier/beam-flink-example

Clearly the problem is not in the artifacts, but in the shadowJar. I'm testing 
the fix and will provide a PR to user. If it's what I think, I will close this 
Jira as it's not a Beam issue.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #234

[Apache Jenkins Server]

See 


--
GitHub pull request #4585 of commit 4d2fed623eddef9f3a8bb1981e1911bdcb2001b3, 
no merge conflicts.
Setting status of 4d2fed623eddef9f3a8bb1981e1911bdcb2001b3 to PENDING with url 
https://builds.apache.org/job/beam_PerformanceTests_JDBC/234/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Java JdbcIO Performance Test
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4585/*:refs/remotes/origin/pr/4585/*
 > git rev-parse refs/remotes/origin/pr/4585/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4585/merge^{commit} # timeout=10
Checking out Revision 93c244a4c3f5ef4dc392b2edfb977f68811f4e56 
(refs/remotes/origin/pr/4585/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 93c244a4c3f5ef4dc392b2edfb977f68811f4e56
Commit message: "Merge 4d2fed623eddef9f3a8bb1981e1911bdcb2001b3 into 
93818abf901dd996aa30445457153811b13ece43"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins538140067291400.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
Fetching cluster endpoint and auth data.
DEBUG: unable to load default kubeconfig: [Errno 2] No such file or directory: 
'/home/jenkins/.kube/config'; recreating /home/jenkins/.kube/config
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: Checking for updates...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4589194026611297974.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4356914166225145917.sh
+ rm -rf .env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8814813696663637891.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5246839393785401843.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5115933430353196731.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4143112721671497997.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4885

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352390#comment-16352390
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


Using 2.3.0-SNAPSHOT artifacts provided by Maven with the following pipeline 
works without problem with the Flink runner:

{code}
public static final void main(String args[]) throws Exception {
PipelineOptions options = 
PipelineOptionsFactory.fromArgs(args).create();
Pipeline pipeline = Pipeline.create(options);

pipeline
.apply(TextIO.read().from("/home/jbonofre/artists.csv"))
.apply(ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext processContext) {
String element = processContext.element();
String[] split = element.split(",");
processContext.output(split[1]);
}
}))
.apply(Count.perElement())
.apply(MapElements.via(new SimpleFunction, 
String>() {
public String apply(KV element) {
return "{\"" + element.getKey() + "\": \"" + 
element.getValue() + "\"}";
}
}))
.apply(TextIO.write().to("/home/jbonofre/label.json"));

pipeline.run();
}
{code}

I'm now testing with artifacts built by Gradle.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5866

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #233

[Apache Jenkins Server]

See 


--
[...truncated 282 B...]
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4585/*:refs/remotes/origin/pr/4585/*
 > git rev-parse refs/remotes/origin/pr/4585/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4585/merge^{commit} # timeout=10
Checking out Revision cdd4ca35a9b4a8b415dab2c0df3b1f6f44c7492f 
(refs/remotes/origin/pr/4585/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f cdd4ca35a9b4a8b415dab2c0df3b1f6f44c7492f
Commit message: "Merge f521bcb07aa1097bc1cda7300a6e4a446e068a1c into 
93818abf901dd996aa30445457153811b13ece43"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1810647867324642778.sh
+ gcloud container clusters list
NAME   ZONE   MASTER_VERSION  MASTER_IP   MACHINE_TYPE   
NODE_VERSION  NUM_NODES  STATUS
io-datastores  us-central1-a  1.7.11-gke.1104.154.102.21  n1-standard-1  
1.7.11-gke.1  1  RUNNING
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7476215323738956507.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
WARNING: Accessing a Container Engine cluster requires the kubernetes 
commandline
client [kubectl]. To install, run
  $ gcloud components install kubectl

Fetching cluster endpoint and auth data.
DEBUG: unable to load default kubeconfig: [Errno 2] No such file or directory: 
'/home/jenkins/.kube/config'; recreating /home/jenkins/.kube/config
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7724032151790843545.sh
+ find / -regex '.*kube.*config.*'
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/postgres/pkb-config-local.yml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/config-files/hdfs-envs