Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4661

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5593

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3835

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #758

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e215b3817e986036c953b2b54b84fb9d67f24d1f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e215b3817e986036c953b2b54b84fb9d67f24d1f
Commit message: "Merge pull request #4354 from mariapython/import_fix"
 > git rev-list e215b3817e986036c953b2b54b84fb9d67f24d1f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6015760566870401813.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3613754396853367439.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1854996572982999473.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement 

Build failed in Jenkins: beam_PerformanceTests_Spark #1202

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e215b3817e986036c953b2b54b84fb9d67f24d1f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e215b3817e986036c953b2b54b84fb9d67f24d1f
Commit message: "Merge pull request #4354 from mariapython/import_fix"
 > git rev-list e215b3817e986036c953b2b54b84fb9d67f24d1f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3068414211553972742.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8894247663837835339.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4162097379892354237.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3892

[Apache Jenkins Server]

See 


--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3891

[Apache Jenkins Server]

See 


Changes:

[ehudm] Split up hdfs and sdk into separate containers.

--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #590

[Apache Jenkins Server]

See 


--
[...truncated 585.29 KB...]
"num_workers": 1, 
"pipelineUrl": 
"gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/pipeline.pb",
 
"pipeline_type_check": true, 
"profile_cpu": false, 
"profile_memory": false, 
"project": "apache-beam-testing", 
"region": "us-central1", 
"requirements_file": "postcommit_requirements.txt", 
"runner": "TestDataflowRunner", 
"runtime_type_check": false, 
"save_main_session": false, 
"sdk_location": "dist/apache-beam-2.3.0.dev0.tar.gz", 
"staging_location": 
"gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597",
 
"streaming": false, 
"style": "scrambled", 
"temp_location": 
"gs://temp-storage-for-end-to-end-tests/temp-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597",
 
"type_check_strictness": "DEFAULT_TO_ANY"
  }
}, 
"tempStoragePrefix": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/temp-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597",
 
"userAgent": {
  "name": "Apache Beam SDK for Python", 
  "version": "2.3.0.dev"
}, 
"version": {
  "job_type": "PYTHON_BATCH", 
  "major": "6"
}, 
"workerPools": [
  {
"autoscalingSettings": {}, 
"kind": "harness", 
"numWorkers": 1, 
"packages": [
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/requirements.txt",
 
"name": "requirements.txt"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/setuptools-38.3.0.zip",
 
"name": "setuptools-38.3.0.zip"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/six-1.11.0.tar.gz",
 
"name": "six-1.11.0.tar.gz"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/setuptools-38.4.0.zip",
 
"name": "setuptools-38.4.0.zip"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/setuptools-38.2.5.zip",
 
"name": "setuptools-38.2.5.zip"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/mock-2.0.0.tar.gz",
 
"name": "mock-2.0.0.tar.gz"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/pbr-3.1.1.tar.gz",
 
"name": "pbr-3.1.1.tar.gz"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/funcsigs-1.0.2.tar.gz",
 
"name": "funcsigs-1.0.2.tar.gz"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/PyHamcrest-1.9.0.tar.gz",
 
"name": "PyHamcrest-1.9.0.tar.gz"
  }, 
  {
"location": 
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0106033652-260308.1515209812.260597/dataflow_python_sdk.tar",
 
"name": "dataflow_python_sdk.tar"
  }
], 
"taskrunnerSettings": {
  "parallelWorkerSettings": {
"baseUrl": "https://dataflow.googleapis.com;, 
"servicePath": "https://dataflow.googleapis.com;
  }
}, 
"workerHarnessContainerImage": 
"dataflow.gcr.io/v1beta3/python:beam-master-20171214"
  }
]
  }, 
  "name": "beamapp-jenkins-0106033652-260308", 
  "steps": [
{
  "kind": "ParallelRead", 
  "name": "s1", 
  "properties": {
"custom_source_step_input": {
  "metadata": {
"estimated_size_bytes": {
  "@type": "http://schema.org/Integer;, 
  "value": 100
}
  }, 
  "spec": {

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4660

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3890

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-3412) Update BigTable client version to 1.0

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3412:


Assignee: Solomon Duskis  (was: Chamikara Jayalath)

> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Solomon Duskis
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314312#comment-16314312
 ] 

Chamikara Jayalath commented on BEAM-3412:
--

Solomon, is using already shaded dependency bigtable-hbase-1.x-shaded [1] an 
option ?

[1] 
https://mvnrepository.com/artifact/com.google.cloud.bigtable/bigtable-hbase-1.x-shaded/1.0.0

> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314303#comment-16314303
 ] 

ASF GitHub Bot commented on BEAM-3411:
--

angoenka opened a new pull request #4355: [BEAM-3411] Fixing KeyError when on 
progress update request
URL: https://github.com/apache/beam/pull/4355
 
 
   Fixing bug where we were throwing uncaught exception when processing 
progress bundle request for unregistered instruction_id
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Test apache_beam.examples.wordcount_it_test.WordCountIT times out
> -
>
> Key: BEAM-3411
> URL: https://issues.apache.org/jira/browse/BEAM-3411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>
> Failed run: 
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console
> Log snippet:
> test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
> ... ERROR
> ==
> ERROR: test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 812, in run
> test(orig)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 45, in __call__
> return self.run(*arg, **kwarg)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 133, in run
> self.runTest(result)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 151, in runTest
> test(result)
>   File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
> return self.run(*args, **kwds)
>   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
> testMethod()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
>  line 77, in test_wordcount_fnapi_it
> on_success_matcher=PipelineStateMatcher()))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
>  line 130, in run
> result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 956, in wait_until_finish
> time.sleep(5.0)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 276, in signalhandler
> raise TimedOutException()
> TimedOutException: 'test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)'
> --
> Ran 3 tests in 901.290s
> FAILED (errors=1)
> Build step 'Execute shell' marked build as failure



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #5591

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3422) The jar files uploaded to maven do not work with Java 9

[Chris Heisterkamp (JIRA)]

Chris Heisterkamp created BEAM-3422:
---

 Summary: The jar files uploaded to maven do not work with Java 9
 Key: BEAM-3422
 URL: https://issues.apache.org/jira/browse/BEAM-3422
 Project: Beam
  Issue Type: Bug
  Components: build-system
Affects Versions: 2.2.0
 Environment: OSX Darwin 17.3.0
Reporter: Chris Heisterkamp
Assignee: Davor Bonaci


The apache beam jars uploaded to maven have invalid date modified times in the 
jar files and cannot be loaded by JDK 9 because of 
https://bugs.openjdk.java.net/browse/JDK-8184940.  I've submitted a new bug to 
http://bugreport.java.com asking that the fix be back ported to the next Java 9 
release.

In the meantime, can you repackage the jars with valid modified times? And 
uploaded fixed jars to Maven?

To reproduce you just need to put one of the jars on the classpath for javac.  
So far I've found the following jars on maven central are affected.
beam-runners-google-cloud-dataflow-java-2.2.0.jar
beam-sdks-common-runner-api-2.2.0.jar
beam-sdks-java-core-2.2.0.jar
beam-sdks-java-extensions-google-cloud-platform-core-2.2.0.jar
beam-sdks-java-extensions-protobuf-2.2.0.jar
beam-sdks-java-io-google-cloud-platform-2.2.0.jar

e.g.

```
> curl -O 
> http://central.maven.org/maven2/org/apache/beam/beam-sdks-java-core/2.2.0/beam-sdks-java-core-2.2.0.jar

> javac -cp beam-sdks-java-core-2.2.0.jar HelloWorld.java
An exception has occurred in the compiler (9.0.1). Please file a bug against 
the Java compiler via the Java bug reporting page (http://bugreport.java.com) 
after checking the Bug Database (http://bugs.java.com) for duplicates. Include 
your program and the following diagnostic in your report. Thank you.
java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 
12): 0
at 
java.base/java.time.temporal.ValueRange.checkValidValue(ValueRange.java:311)
at 
java.base/java.time.temporal.ChronoField.checkValidValue(ChronoField.java:714)
at java.base/java.time.LocalDate.of(LocalDate.java:269)
at java.base/java.time.LocalDateTime.of(LocalDateTime.java:336)
at jdk.zipfs/jdk.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:109)
at 
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:1950)
at 
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1937)
at 
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getEntry(ZipFileSystem.java:1324)
at 
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.newInputStream(ZipFileSystem.java:550)
at 
jdk.zipfs/jdk.nio.zipfs.JarFileSystem.isMultiReleaseJar(JarFileSystem.java:91)
at jdk.zipfs/jdk.nio.zipfs.JarFileSystem.(JarFileSystem.java:67)
at 
jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:134)
at 
jdk.compiler/com.sun.tools.javac.file.JavacFileManager$ArchiveContainer.(JavacFileManager.java:517)
at 
jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getContainer(JavacFileManager.java:319)
at 
jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:715)
at 
jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:722)
at 
jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:648)
at 
jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:529)
at 
jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:293)
at 
jdk.compiler/com.sun.tools.javac.code.Symtab.lambda$addRootPackageFor$7(Symtab.java:758)
at 
jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:633)
at 
jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:354)
at 
jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
at 
jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:285)
at 
jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:300)
at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:570)
at jdk.compiler/com.sun.tools.javac.comp.Enter.main(Enter.java:554)
at 
jdk.compiler/com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:1052)
at 
jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:923)
at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:302)
at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:162)
at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:57)
at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:43)

> java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
```

Unpacking and re-creating the jar with JDK 8 fixes the problem on 

[jira] [Comment Edited] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314279#comment-16314279
 ] 

Ankur Goenka edited comment on BEAM-3411 at 1/6/18 1:48 AM:


We missed catching a case where progress report for an unregistered work item 
is requested from the worker which resulted in an uncaught exceptions.
Erroneous Code block: 

{code:java}
worker = self._instruction_id_vs_worker[request.instruction_id]
{code}

Underlying error stack trace:
{code:java}
Python sdk harness failed: 
Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 134, in main
worker_count=_get_worker_count(sdk_pipeline_options)).run()
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 97, in run
work_request)
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 162, in _request_process_bundle_progress
worker = self._instruction_id_vs_worker[request.instruction_id]
KeyError: u'-39'
{code}



was (Author: angoenka):
We missed catching a case where progress report for an unregistered work item 
is requested from the worker which resulted in an uncaught exceptions.
Erroneous Code block: 

{code:python}
worker = self._instruction_id_vs_worker[request.instruction_id]
{code}

Underlying error stack trace:
{code:python}
Python sdk harness failed: 
Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 134, in main
worker_count=_get_worker_count(sdk_pipeline_options)).run()
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 97, in run
work_request)
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 162, in _request_process_bundle_progress
worker = self._instruction_id_vs_worker[request.instruction_id]
KeyError: u'-39'
{code}


> Test apache_beam.examples.wordcount_it_test.WordCountIT times out
> -
>
> Key: BEAM-3411
> URL: https://issues.apache.org/jira/browse/BEAM-3411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>
> Failed run: 
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console
> Log snippet:
> test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
> ... ERROR
> ==
> ERROR: test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 812, in run
> test(orig)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 45, in __call__
> return self.run(*arg, **kwarg)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 133, in run
> self.runTest(result)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 151, in runTest
> test(result)
>   File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
> return self.run(*args, **kwds)
>   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
> testMethod()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
>  line 77, in test_wordcount_fnapi_it
> on_success_matcher=PipelineStateMatcher()))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
>  line 130, in run
> result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 956, in wait_until_finish
> time.sleep(5.0)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 276, in signalhandler
> raise TimedOutException()
> TimedOutException: 'test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)'
> --
> Ran 3 tests in 901.290s
> FAILED (errors=1)

[jira] [Comment Edited] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314279#comment-16314279
 ] 

Ankur Goenka edited comment on BEAM-3411 at 1/6/18 1:47 AM:


We missed catching a case where progress report for an unregistered work item 
is requested from the worker which resulted in an uncaught exceptions.
Erroneous Code block: 

{code:python}
worker = self._instruction_id_vs_worker[request.instruction_id]
{code}

Underlying error stack trace:
{code:python}
Python sdk harness failed: 
Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 134, in main
worker_count=_get_worker_count(sdk_pipeline_options)).run()
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 97, in run
work_request)
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 162, in _request_process_bundle_progress
worker = self._instruction_id_vs_worker[request.instruction_id]
KeyError: u'-39'
{code}



was (Author: angoenka):
We missed catching a case where progress report for an unregistered work item 
is requested from the worker which resulted in an uncaught exceptions:
Underlying error stack trace:
{code:java}
Python sdk harness failed: 
Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 134, in main
worker_count=_get_worker_count(sdk_pipeline_options)).run()
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 97, in run
work_request)
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 162, in _request_process_bundle_progress
worker = self._instruction_id_vs_worker[request.instruction_id]
KeyError: u'-39'
{code}


> Test apache_beam.examples.wordcount_it_test.WordCountIT times out
> -
>
> Key: BEAM-3411
> URL: https://issues.apache.org/jira/browse/BEAM-3411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>
> Failed run: 
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console
> Log snippet:
> test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
> ... ERROR
> ==
> ERROR: test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 812, in run
> test(orig)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 45, in __call__
> return self.run(*arg, **kwarg)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 133, in run
> self.runTest(result)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 151, in runTest
> test(result)
>   File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
> return self.run(*args, **kwds)
>   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
> testMethod()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
>  line 77, in test_wordcount_fnapi_it
> on_success_matcher=PipelineStateMatcher()))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
>  line 130, in run
> result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 956, in wait_until_finish
> time.sleep(5.0)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 276, in signalhandler
> raise TimedOutException()
> TimedOutException: 'test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)'
> --
> Ran 3 tests in 901.290s
> FAILED (errors=1)
> Build step 'Execute shell' marked build as failure



--
This message was sent by Atlassian JIRA

[jira] [Commented] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314279#comment-16314279
 ] 

Ankur Goenka commented on BEAM-3411:


We missed catching a case where progress report for an unregistered work item 
is requested from the worker which resulted in an uncaught exceptions:
Underlying error stack trace:
{code:java}
Python sdk harness failed: 
Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 134, in main
worker_count=_get_worker_count(sdk_pipeline_options)).run()
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 97, in run
work_request)
  File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 162, in _request_process_bundle_progress
worker = self._instruction_id_vs_worker[request.instruction_id]
KeyError: u'-39'
{code}


> Test apache_beam.examples.wordcount_it_test.WordCountIT times out
> -
>
> Key: BEAM-3411
> URL: https://issues.apache.org/jira/browse/BEAM-3411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>
> Failed run: 
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console
> Log snippet:
> test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
> ... ERROR
> ==
> ERROR: test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 812, in run
> test(orig)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 45, in __call__
> return self.run(*arg, **kwarg)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 133, in run
> self.runTest(result)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 151, in runTest
> test(result)
>   File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
> return self.run(*args, **kwds)
>   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
> testMethod()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
>  line 77, in test_wordcount_fnapi_it
> on_success_matcher=PipelineStateMatcher()))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
>  line 130, in run
> result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 956, in wait_until_finish
> time.sleep(5.0)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 276, in signalhandler
> raise TimedOutException()
> TimedOutException: 'test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)'
> --
> Ran 3 tests in 901.290s
> FAILED (errors=1)
> Build step 'Execute shell' marked build as failure



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3833

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #3136

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3421) Have the pipeline supply beam-sdks-java-harness instead of embedding it within the beam-sdks-java-container

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3421:
---

 Summary: Have the pipeline supply beam-sdks-java-harness instead 
of embedding it within the beam-sdks-java-container
 Key: BEAM-3421
 URL: https://issues.apache.org/jira/browse/BEAM-3421
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-harness
Reporter: Luke Cwik
Assignee: Luke Cwik
Priority: Minor


Have the Java SDK harness container require the user to ship all the 
dependencies including "beam-sdks-java-harness".

This allows for a bare container where all java code is supplied by the 
pipeline being run allowing for testing at HEAD.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3888

[Apache Jenkins Server]

See 


Changes:

[mariagh] Clean mock import

--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying 

Jenkins build is back to normal : beam_PerformanceTests_FileBasedIO_IT #64

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #757

[Apache Jenkins Server]

See 


Changes:

[mariagh] Clean mock import

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e215b3817e986036c953b2b54b84fb9d67f24d1f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e215b3817e986036c953b2b54b84fb9d67f24d1f
Commit message: "Merge pull request #4354 from mariapython/import_fix"
 > git rev-list 4f947133c55b8c35a66274dfeb587b168ad3c659 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2586455570639362400.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3301375833801930251.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5235706618181653496.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r 

[jira] [Commented] (BEAM-3397) beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because of too long commandline

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314200#comment-16314200
 ] 

Alan Myrvold commented on BEAM-3397:


That pull request didn't work, so closed without merge

> beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because 
> of too long commandline
> -
>
> Key: BEAM-3397
> URL: https://issues.apache.org/jira/browse/BEAM-3397
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, testing
>Reporter: Luke Cwik
> Fix For: 2.3.0
>
>
> The Dataflow worker fails to launch the process because the command line is 
> too long. The worker logs are riddled with a very long command line followed 
> by this error message:
> `java failed with fork/exec /usr/local/bin/java: argument list too long`
> Example failures from run from master @ 
> 49981f15009de434af9c09c3c9f50003c7417b88
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PreCommit_Java_MavenInstall/16428/
> Job ids (project: apache-beam-testing):
> 2017-12-28_01_32_55-16084854344397840285
> 2017-12-28_01_32_55-213040034028880428
> 2017-12-28_01_32_55-16742595557951289122



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3397) beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because of too long commandline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314179#comment-16314179
 ] 

ASF GitHub Bot commented on BEAM-3397:
--

alanmyrvold closed pull request #4345: [BEAM-3397] Fix the failure in the 
dataflow integration test by removing the spark and flink pipeline options
URL: https://github.com/apache/beam/pull/4345
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ce4527d6096..1f4f9baf25e 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -267,6 +267,16 @@
 WordCountIT.java
 WindowedWordCountIT.java
   
+  
+ 
+ 
org.apache.beam:beam-runners-spark
+ 
org.apache.beam:beam-runners-flink_2.11
+  
   all
   4
   


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because 
> of too long commandline
> -
>
> Key: BEAM-3397
> URL: https://issues.apache.org/jira/browse/BEAM-3397
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, testing
>Reporter: Luke Cwik
> Fix For: 2.3.0
>
>
> The Dataflow worker fails to launch the process because the command line is 
> too long. The worker logs are riddled with a very long command line followed 
> by this error message:
> `java failed with fork/exec /usr/local/bin/java: argument list too long`
> Example failures from run from master @ 
> 49981f15009de434af9c09c3c9f50003c7417b88
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PreCommit_Java_MavenInstall/16428/
> Job ids (project: apache-beam-testing):
> 2017-12-28_01_32_55-16084854344397840285
> 2017-12-28_01_32_55-213040034028880428
> 2017-12-28_01_32_55-16742595557951289122



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3887

[Apache Jenkins Server]

See 

[beam] branch master updated (4f94713 -> e215b38)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4f94713  Merge pull request #4324 from aaltay/gcsi
 add eb892e9  Clean mock import
 new e215b38  Merge pull request #4354 from mariapython/import_fix

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/01: Merge pull request #4354 from mariapython/import_fix

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e215b3817e986036c953b2b54b84fb9d67f24d1f
Merge: 4f94713 eb892e9
Author: Ahmet Altay 
AuthorDate: Fri Jan 5 15:59:07 2018 -0800

Merge pull request #4354 from mariapython/import_fix

Clean mock import

 sdks/python/apache_beam/pipeline_test.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3831

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1817) Python environment is different on some nodes

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314147#comment-16314147
 ] 

Udi Meiri commented on BEAM-1817:
-

PR 2308 resulted in limiting postcommit runs to exactly 1 machine. 
(https://issues.apache.org/jira/projects/BEAM/issues/BEAM-3395)

What exactly is different in the environment?


> Python environment is different on some nodes
> -
>
> Key: BEAM-1817
> URL: https://issues.apache.org/jira/browse/BEAM-1817
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Sourabh Bajaj
>Assignee: Jason Kuster
>
> We've pined the python post commits to specific nodes as pip in some of the 
> new nodes seems to be installed differently from the ones before.
> https://github.com/apache/beam/pull/2308/files
> Please remove the changes in above PR once this is resolved.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-3412) Update BigTable client version to 1.0

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313766#comment-16313766
 ] 

Chamikara Jayalath edited comment on BEAM-3412 at 1/5/18 11:26 PM:
---

Bumping up Bigtable client version requires bumping up GRPC dependency which 
require bumping up netty/jetty dependencies of Beam. Historically bumping up 
these dependencies has resulted in issues. So we'll have to shade away these 
dependencies.

cc: [~lcwik] [~sduskis]


was (Author: chamikara):
Bumping up Bigtable client version requires bumping up GRPC dependency which 
require bumping up netty/jetty dependencies of Beam. Historically bumping up 
these dependencies has resulted in issues. So we'll have to shard away these 
dependencies.

cc: [~lcwik] [~sduskis]

> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3420) TimerData#compareTo should respect Timer IDs

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3420:
-

 Summary: TimerData#compareTo should respect Timer IDs
 Key: BEAM-3420
 URL: https://issues.apache.org/jira/browse/BEAM-3420
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Batkhuyag Batsaikhan


https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java#L209

Two timers that have distinct IDs should not compare as equal, even if they're 
in the same namespace, for the same time in the same domain. They should appear 
in an arbitrary, but consistent, order.

We should have more tests surrounding these code paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3419) Enable iterable side input for beam runners.

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3419:

Issue Type: Sub-task  (was: New Feature)
Parent: BEAM-2863

> Enable iterable side input for beam runners.
> 
>
> Key: BEAM-3419
> URL: https://issues.apache.org/jira/browse/BEAM-3419
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Robert Bradshaw
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3419) Enable iterable side input for beam runners.

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3419:
---

Assignee: (was: Kenneth Knowles)

> Enable iterable side input for beam runners.
> 
>
> Key: BEAM-3419
> URL: https://issues.apache.org/jira/browse/BEAM-3419
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Robert Bradshaw
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314008#comment-16314008
 ] 

Alan Myrvold commented on BEAM-3406:


Created the table and loaded the two records in the attached file.
Used the superset schema from 
https://github.com/GoogleCloudPlatform/PerfKitExplorer/blob/master/data/samples_mart/results_table_schema.json

bq load --source_format=NEWLINE_DELIMITED_JSON 
apache-beam-testing:beam_performance.filebasedioit_pkb_results textiot.txt 
schema.txt

This painpoint of needing to create the table seems to be 
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/229

> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
> Attachments: schema, textioit
>
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3886

[Apache Jenkins Server]

See 


--
[...truncated 1.14 MB...]
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "pair_with_one"
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "group.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": 
"%0AJ%22H%0A%1Dref_Coder_GlobalWindowCoder_1%12%27%0A%25%0A%23%0A%21urn%3Abeam%3Acoders%3Aglobal_window%3A0.1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "group"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "count_ones"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "count.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "count"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s6", 
  "properties": {
"display_data": [
  {
"key": "fn", 

[jira] [Assigned] (BEAM-3368) Go model pipeline doesn't support External transform

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-3368:
---

Assignee: Bill Neubauer  (was: Henning Rohde)

> Go model pipeline doesn't support External transform
> 
>
> Key: BEAM-3368
> URL: https://issues.apache.org/jira/browse/BEAM-3368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The model pipeline translation process doesn't support the external 
> transform, causing the translation process to panic when it encounters it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3885

[Apache Jenkins Server]

See 


Changes:

[altay] Remove Queue based 60 seconds timeout for GCS io.

--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying 

[jira] [Commented] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313770#comment-16313770
 ] 

Chamikara Jayalath commented on BEAM-3406:
--

Alan will look into creating the table. Thanks Alan.

> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
> Attachments: schema, textioit
>
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3419) Enable iterable side input for beam runners.

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-3419:
-

 Summary: Enable iterable side input for beam runners.
 Key: BEAM-3419
 URL: https://issues.apache.org/jira/browse/BEAM-3419
 Project: Beam
  Issue Type: New Feature
  Components: runner-core
Reporter: Robert Bradshaw
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313766#comment-16313766
 ] 

Chamikara Jayalath commented on BEAM-3412:
--

Bumping up Bigtable client version requires bumping up GRPC dependency which 
require bumping up netty/jetty dependencies of Beam. Historically bumping up 
these dependencies has resulted in issues. So we'll have to shard away these 
dependencies.

cc: [~lcwik] [~sduskis]

> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313731#comment-16313731
 ] 

Ahmet Altay commented on BEAM-3106:
---

[~m...@maxroos.com] if it helps you can upgrade Beam's bigquery version as a 
temporary relief until we have a permanent fix to this issue.

> Consider not pinning all python dependencies, or moving them to 
> requirements.txt
> 
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Affects Versions: 2.1.0
> Environment: python
>Reporter: Maximilian Roos
>
> Currently all python dependencies are [pinned or 
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well 
> tested dependencies, having them specified in `setup.py` forces them to an 
> exact state on each install of Beam. This makes using Beam in any environment 
> with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have 
> libraries that won't work with an older version (but Beam _does_ work with an 
> newer version). We have to do a bunch of gymnastics to get the correct 
> versions installed because of this. Unfortunately, airflow repeats this 
> practice and conflicts on a number of dependencies, adding further 
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for 
> example no libraries in numerical python do this. Here's a [discussion on 
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2870) BQ Partitioned Table Write Fails When Destination has Partition Decorator

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2870.
--
Resolution: Fixed
  Assignee: Eugene Kirpichov  (was: Reuven Lax)

This was fixed for batch by https://github.com/apache/beam/pull/4177

> BQ Partitioned Table Write Fails When Destination has Partition Decorator
> -
>
> Key: BEAM-2870
> URL: https://issues.apache.org/jira/browse/BEAM-2870
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.3.0
> Environment: Dataflow Runner, Streaming, 10 x (n1-highmem-8 & 500gb 
> SDD)
>Reporter: Steven Jon Anderson
>Assignee: Eugene Kirpichov
>  Labels: bigquery, dataflow, google, google-cloud-bigquery, 
> google-dataflow
> Fix For: 2.3.0
>
>
> Dataflow Job ID: 
> https://console.cloud.google.com/dataflow/job/2017-09-08_23_03_14-14637186041605198816
> Tagging [~reuvenlax] as I believe he built the time partitioning integration 
> that was merged into master.
> *Background*
> Our production pipeline ingests millions of events per day and routes events 
> into our clients' numerous tables. To keep costs down, all of our tables are 
> partitioned. However, this requires that we create the tables before we allow 
> events to process as creating partitioned tables isn't supported in 2.1.0. 
> We've been looking forward to [~reuvenlax]'s partition table write feature 
> ([#3663|https://github.com/apache/beam/pull/3663]) to get merged into master 
> for some time now as it'll allow us to launch our client platforms much, much 
> faster. Today we got around to testing the 2.2.0 nightly and discovered this 
> bug.
> *Issue*
> Our pipeline writes to a table with a decorator. When attempting to write to 
> an existing partitioned table with a decorator, the write succeeds. When 
> using a partitioned table destination that doesn't exist without a decorator, 
> the write succeeds. *However, when writing to a partitioned table that 
> doesn't exist with a decorator, the write fails*. 
> *Example Implementation*
> {code:java}
> BigQueryIO.writeTableRows()
>   .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
>   .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
>   .withFailedInsertRetryPolicy(InsertRetryPolicy.alwaysRetry())
>   .to(new DynamicDestinations() {
> @Override
> public String getDestination(ValueInSingleWindow element) {
>   return "PROJECT_ID:DATASET_ID.TABLE_ID$20170902";
> }
> @Override
> public TableDestination getTable(String destination) {
>   TimePartitioning DAY_PARTITION = new TimePartitioning().setType("DAY");
>   return new TableDestination(destination, null, DAY_PARTITION);
> }
> @Override
> public TableSchema getSchema(String destination) {
>   return TABLE_SCHEMA;
> }
>   })
> {code}
> *Relevant Logs & Errors in StackDriver*
> {code:none}
> 23:06:26.790 
> Trying to create BigQuery table: PROJECT_ID:DATASET_ID.TABLE_ID$20170902
> 23:06:26.873 
> Invalid table ID \"TABLE_ID$20170902\". Table IDs must be alphanumeric (plus 
> underscores) and must be at most 1024 characters long. Also, Table decorators 
> cannot be used.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4658

[Apache Jenkins Server]

See 


Changes:

[altay] Remove Queue based 60 seconds timeout for GCS io.

--
Started by GitHub push by aaltay
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4f947133c55b8c35a66274dfeb587b168ad3c659 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4f947133c55b8c35a66274dfeb587b168ad3c659
Commit message: "Merge pull request #4324 from aaltay/gcsi"
 > git rev-list d2690fae0cd3c16b18973ed68f514a2f2319af2b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Modules changed, recalculating dependency graph
Established TCP socket on 42248
maven33-agent.jar already up to date
maven33-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[src] $ /home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS 
-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -cp 
/home/jenkins/jenkins-slave/maven33-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven33Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven33-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 42248
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/google-cloud-dataflow-java -DforkCount=0 
-DvalidatesRunnerPipelineOptions=[ "--runner=TestDataflowRunner", 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-validates-runner-tests/" ]
2018-01-05T19:23:11.169 [INFO] Error stacktraces are turned on.
2018-01-05T19:23:11.377 [INFO] Scanning for projects...
2018-01-05T19:23:18.340 [WARNING] 
2018-01-05T19:23:18.353 [WARNING] Some problems were encountered while building 
the effective model for 
org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
2018-01-05T19:23:18.354 [WARNING] The expression ${parent.version} is 
deprecated. Please use ${project.parent.version} instead.
2018-01-05T19:23:18.354 [WARNING] 
2018-01-05T19:23:18.354 [WARNING] It is highly recommended to fix these 
problems because they threaten the stability of your build.
2018-01-05T19:23:18.354 [WARNING] 
2018-01-05T19:23:18.355 [WARNING] For this reason, future Maven versions might 
no longer support building such malformed projects.
2018-01-05T19:23:18.355 [WARNING] 
2018-01-05T19:23:18.415 [INFO] 

2018-01-05T19:23:18.416 [INFO] Detecting the operating system and CPU 
architecture
2018-01-05T19:23:18.416 [INFO] 

2018-01-05T19:23:18.417 [INFO] os.detected.name: linux
2018-01-05T19:23:18.417 [INFO] os.detected.arch: x86_64
2018-01-05T19:23:18.417 [INFO] os.detected.version: 4.4
2018-01-05T19:23:18.417 [INFO] os.detected.version.major: 4
2018-01-05T19:23:18.418 [INFO] os.detected.version.minor: 4
2018-01-05T19:23:18.440 [INFO] os.detected.release: ubuntu
2018-01-05T19:23:18.440 [INFO] os.detected.release.version: 14.04
2018-01-05T19:23:18.440 [INFO] os.detected.release.like.ubuntu: true
2018-01-05T19:23:18.440 [INFO] os.detected.release.like.debian: true
2018-01-05T19:23:18.440 [INFO] os.detected.classifier: linux-x86_64
2018-01-05T19:23:58.901 [INFO] 

2018-01-05T19:23:58.901 [INFO] Reactor Build Order:
2018-01-05T19:23:58.901 [INFO] 
2018-01-05T19:23:58.901 

[jira] [Updated] (BEAM-3418) Python Fnapi - Multiprocess worker

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka updated BEAM-3418:
---
Issue Type: Improvement  (was: Bug)

> Python Fnapi - Multiprocess worker
> --
>
> Key: BEAM-3418
> URL: https://issues.apache.org/jira/browse/BEAM-3418
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>
> Support multiple python SDK process on a VM to fully utilize a machine.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3418) Python Fnapi - Multiprocess worker

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-3418:
--

 Summary: Python Fnapi - Multiprocess worker
 Key: BEAM-3418
 URL: https://issues.apache.org/jira/browse/BEAM-3418
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-harness
Reporter: Ankur Goenka
Assignee: Ankur Goenka


Support multiple python SDK process on a VM to fully utilize a machine.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3125) Portable flattens in Java SDK Harness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313704#comment-16313704
 ] 

ASF GitHub Bot commented on BEAM-3125:
--

youngoli opened a new pull request #4353: [BEAM-3125] 
URL: https://github.com/apache/beam/pull/4353
 
 
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   
   This change adds a runner to the Java SDK Harness for flattens so that if it 
can handle receiving pTransforms with the URN for flattens. Currently flattens 
are done implicitly by having DoFns with multiple inputs. This change will not 
remove that functionality, but will also allow explicit flattens.
   
   R: @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Portable flattens in Java SDK Harness
> -
>
> Key: BEAM-3125
> URL: https://issues.apache.org/jira/browse/BEAM-3125
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>  Labels: portability
>
> Add flattens to the graphs executed in the Java SDK Harness. This means 
> creating the flatten nodes and wiring it up to everything.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4657

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4630

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #63

[Apache Jenkins Server]

See 


Changes:

[altay] Remove Queue based 60 seconds timeout for GCS io.

[robertwb] Plumb user metrics through Fn API.

--
[...truncated 32.82 KB...]
2018-01-05 18:50:29,684 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:50:49,402 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:50:51,236 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
2018-01-05 18:50:51,236 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:51:12,555 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:51:14,538 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
2018-01-05 18:51:14,539 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:51:33,461 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:51:35,363 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
2018-01-05 18:51:35,364 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:52:01,658 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:52:03,780 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
2018-01-05 18:52:03,780 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:52:32,774 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:52:34,695 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
2018-01-05 18:52:34,696 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:52:56,710 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:52:58,447 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
2018-01-05 18:52:58,447 e68a329c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 18:53:15,303 e68a329c MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e68a329c/perfkit-bq-pubRX_afq.json
2018-01-05 18:53:17,032 e68a329c MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 

Build failed in Jenkins: beam_PerformanceTests_Python #756

[Apache Jenkins Server]

See 


Changes:

[altay] Remove Queue based 60 seconds timeout for GCS io.

[robertwb] Plumb user metrics through Fn API.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4f947133c55b8c35a66274dfeb587b168ad3c659 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4f947133c55b8c35a66274dfeb587b168ad3c659
Commit message: "Merge pull request #4324 from aaltay/gcsi"
 > git rev-list 63e269604a437f7110e15a8fcd27257864c9100d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1467476839865352388.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4097186873150407175.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4455636442073340339.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5589

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-3368) Go model pipeline doesn't support External transform

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-3368:
---

Assignee: Henning Rohde  (was: Bill Neubauer)

> Go model pipeline doesn't support External transform
> 
>
> Key: BEAM-3368
> URL: https://issues.apache.org/jira/browse/BEAM-3368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Henning Rohde
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The model pipeline translation process doesn't support the external 
> transform, causing the translation process to panic when it encounters it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PerformanceTests_Spark #1200

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3182) [Nexmark][SQL] Implement supported queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313618#comment-16313618
 ] 

ASF GitHub Bot commented on BEAM-3182:
--

akedin opened a new pull request #4352: [BEAM-3182][Nexmark][SQL] Implement sql 
query 3
URL: https://github.com/apache/beam/pull/4352
 
 
   [BEAM-3182](https://issues.apache.org/jira/browse/BEAM-3182)
   
   [Java version of the same 
query](https://github.com/apache/beam/blob/64ff21f35ee2946d38645fb0a51678628e49e62a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java#L74)
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> [Nexmark][SQL] Implement supported queries
> --
>
> Key: BEAM-3182
> URL: https://issues.apache.org/jira/browse/BEAM-3182
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Implement all queries which can be run with current SQL features and Nexmark 
> infrastructure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

[Maximilian Roos (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313574#comment-16313574
 ] 

Maximilian Roos commented on BEAM-3106:
---

Thanks for your earlier responses Ahmet. 

To give a concrete case, as I find those can be helpful beyond the abstract: 
Currently Beam tags google-cloud-bigquery to 0.25.0, from [June 
26|https://github.com/GoogleCloudPlatform/google-cloud-python/releases/tag/bigquery-0.25.0].
 The most up-to-date is 0.29.0. 

We have a library that depends on >=0.28.0, which we can't use at the same time 
as Beam. And we have to set up two separate build paths - one to test with Beam 
and another to test with the existing library.

Cheers, Max

> Consider not pinning all python dependencies, or moving them to 
> requirements.txt
> 
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Affects Versions: 2.1.0
> Environment: python
>Reporter: Maximilian Roos
>
> Currently all python dependencies are [pinned or 
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well 
> tested dependencies, having them specified in `setup.py` forces them to an 
> exact state on each install of Beam. This makes using Beam in any environment 
> with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have 
> libraries that won't work with an older version (but Beam _does_ work with an 
> newer version). We have to do a bunch of gymnastics to get the correct 
> versions installed because of this. Unfortunately, airflow repeats this 
> practice and conflicts on a number of dependencies, adding further 
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for 
> example no libraries in numerical python do this. Here's a [discussion on 
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3884

[Apache Jenkins Server]

See 


Changes:

[robertwb] Plumb user metrics through Fn API.

--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying 

[jira] [Commented] (BEAM-3182) [Nexmark][SQL] Implement supported queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313554#comment-16313554
 ] 

ASF GitHub Bot commented on BEAM-3182:
--

akedin opened a new pull request #4351: [BEAM-3182][Nexmark][SQL] Implement sql 
query 2
URL: https://github.com/apache/beam/pull/4351
 
 
   [BEAM-3182](https://issues.apache.org/jira/browse/BEAM-3182)
   
   [Java version of the same 
query](https://github.com/apache/beam/blob/64ff21f35ee2946d38645fb0a51678628e49e62a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query2.java#L46)
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> [Nexmark][SQL] Implement supported queries
> --
>
> Key: BEAM-3182
> URL: https://issues.apache.org/jira/browse/BEAM-3182
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Implement all queries which can be run with current SQL features and Nexmark 
> infrastructure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #586

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2549) gcsio should set timeouts for http requests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313526#comment-16313526
 ] 

ASF GitHub Bot commented on BEAM-2549:
--

aaltay closed pull request #4324: [BEAM-2549] Remove Queue based 60 seconds 
timeout for GCS io.
URL: https://github.com/apache/beam/pull/4324
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py 
b/sdks/python/apache_beam/io/gcp/gcsio.py
index 68ca0265601..d5657e7ca3e 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio.py
@@ -26,7 +26,6 @@
 import logging
 import multiprocessing
 import os
-import Queue
 import re
 import threading
 import time
@@ -427,14 +426,12 @@ def __init__(self,
client,
path,
mode='r',
-   buffer_size=DEFAULT_READ_BUFFER_SIZE,
-   segment_timeout=DEFAULT_READ_SEGMENT_TIMEOUT_SECONDS):
+   buffer_size=DEFAULT_READ_BUFFER_SIZE):
 self.client = client
 self.path = path
 self.bucket, self.name = parse_gcs_path(path)
 self.mode = mode
 self.buffer_size = buffer_size
-self.segment_timeout = segment_timeout
 
 # Get object state.
 self.get_request = (storage.StorageObjectsGetRequest(
@@ -575,47 +572,18 @@ def _fetch_next_if_buffer_exhausted(self):
 self.buffer_start_position + len(self.buffer) <= self.position):
   bytes_to_request = min(self._remaining(), self.buffer_size)
   self.buffer_start_position = self.position
-  retry_count = 0
-  while retry_count <= 10:
-queue = Queue.Queue()
-t = threading.Thread(target=self._fetch_to_queue,
- args=(queue, self._get_segment,
-   (self.position, bytes_to_request)))
-t.daemon = True
-t.start()
-try:
-  result, exn, tb = queue.get(timeout=self.segment_timeout)
-except Queue.Empty:
-  logging.warning(
-  ('Timed out fetching %d bytes from position %d of %s after %f '
-   'seconds; retrying...'), bytes_to_request, self.position,
-  self.path, self.segment_timeout)
-  retry_count += 1
-  # Reinitialize download objects.
-  self.download_stream = cStringIO.StringIO()
-  self.downloader = transfer.Download(
-  self.download_stream, auto_transfer=False,
-  chunksize=self.buffer_size)
-  self.client.objects.Get(self.get_request, download=self.downloader)
-  continue
-if exn:
-  logging.error(
-  ('Exception while fetching %d bytes from position %d of %s: '
-   '%s\n%s'),
-  bytes_to_request, self.position, self.path, exn, tb)
-  raise exn
-self.buffer = result
-return
-  raise GcsIOError(
-  'Reached retry limit for _fetch_next_if_buffer_exhausted.')
+  try:
+result = self._get_segment(self.position, bytes_to_request)
+  except Exception as e:  # pylint: disable=broad-except
+tb = traceback.format_exc()
+logging.error(
+('Exception while fetching %d bytes from position %d of %s: '
+ '%s\n%s'),
+bytes_to_request, self.position, self.path, e, tb)
+raise
 
-  def _fetch_to_queue(self, queue, func, args):
-try:
-  value = func(*args)
-  queue.put((value, None, None))
-except Exception as e:  # pylint: disable=broad-except
-  tb = traceback.format_exc()
-  queue.put((None, e, tb))
+  self.buffer = result
+  return
 
   def _remaining(self):
 return self.size - self.position
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_test.py 
b/sdks/python/apache_beam/io/gcp/gcsio_test.py
index 06a82272900..6994c523032 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_test.py
@@ -22,7 +22,6 @@
 import os
 import random
 import threading
-import time
 import unittest
 
 import httplib2
@@ -436,43 +435,6 @@ def test_full_file_read(self):
 f.seek(0)
 self.assertEqual(f.read(), random_file.contents)
 
-  def test_flaky_file_read(self):
-file_name = 'gs://gcsio-test/flaky_file'
-file_size = 5 * 1024 * 1024 + 100
-random_file = self._insert_random_file(self.client, file_name, file_size)
-f = self.gcs.open(file_name)
-random.seed(0)
-f.buffer_size = 1024 * 1024
-f.segment_timeout = 0.01
-self.assertEqual(f.mode, 'r')
-f._real_get_segment = f._get_segment
-
-def flaky_get_segment(start, size):
-  if random.randint(0, 3) == 1:
-time.sleep(600)
-  return 

[beam] branch master updated (d2690fa -> 4f94713)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d2690fa  Merge pull request #4344 from robertwb/user-metrics
 add d7aaf9b  Remove Queue based 60 seconds timeout for GCS io.
 new 4f94713  Merge pull request #4324 from aaltay/gcsi

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/gcp/gcsio.py  | 56 ++--
 sdks/python/apache_beam/io/gcp/gcsio_test.py | 38 ---
 2 files changed, 12 insertions(+), 82 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/01: Merge pull request #4324 from aaltay/gcsi

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4f947133c55b8c35a66274dfeb587b168ad3c659
Merge: d2690fa d7aaf9b
Author: Ahmet Altay 
AuthorDate: Fri Jan 5 09:45:30 2018 -0800

Merge pull request #4324 from aaltay/gcsi

[BEAM-2549] Remove Queue based 60 seconds timeout for GCS io.

 sdks/python/apache_beam/io/gcp/gcsio.py  | 56 ++--
 sdks/python/apache_beam/io/gcp/gcsio_test.py | 38 ---
 2 files changed, 12 insertions(+), 82 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Updated] (BEAM-3417) Fix Calcite assertions

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3417:
--
Description: 
Currently we disable assertions in test for every project which depends on Beam 
SQL / Calcite. Otherwise it fails assertions when Calcite validates relational 
representation of the query. E.g. in projects which depend on Beam SQL / 
Calcite we have to specify 
{code:java|title=build.gradle}
test {
 jvmArgs "-da" 
}
{code}

We need to either update our relational conversion logic or come up with some 
other solution so that we don't have to disable assertions globally. If it's an 
incorrect assertion in Calcite then we need to fix it there.

  was:
Currently we disable assertions in test for every project which depends on Beam 
SQL / Calcite. Otherwise it fails assertions when Calcite validates relational 
representation of the query. E.g. in `build.gradle` you have to specify 
```java
test {
 jvmArgs "-da" 
}
```
We need to either update our relational conversion logic or come up with some 
other solution so that we don't have to disable assertions globally.


> Fix Calcite assertions
> --
>
> Key: BEAM-3417
> URL: https://issues.apache.org/jira/browse/BEAM-3417
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Xu Mingmin
>
> Currently we disable assertions in test for every project which depends on 
> Beam SQL / Calcite. Otherwise it fails assertions when Calcite validates 
> relational representation of the query. E.g. in projects which depend on Beam 
> SQL / Calcite we have to specify 
> {code:java|title=build.gradle}
> test {
>  jvmArgs "-da" 
> }
> {code}
> We need to either update our relational conversion logic or come up with some 
> other solution so that we don't have to disable assertions globally. If it's 
> an incorrect assertion in Calcite then we need to fix it there.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3417) Fix Calcite assertions

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-3417:
-

Assignee: (was: Xu Mingmin)

> Fix Calcite assertions
> --
>
> Key: BEAM-3417
> URL: https://issues.apache.org/jira/browse/BEAM-3417
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Anton Kedin
>
> Currently we disable assertions in test for every project which depends on 
> Beam SQL / Calcite. Otherwise it fails assertions when Calcite validates 
> relational representation of the query. E.g. in projects which depend on Beam 
> SQL / Calcite we have to specify 
> {code:java|title=build.gradle}
> test {
>  jvmArgs "-da" 
> }
> {code}
> We need to either update our relational conversion logic or come up with some 
> other solution so that we don't have to disable assertions globally. If it's 
> an incorrect assertion in Calcite then we need to fix it there.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3417) Fix Calcite assertions

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3417:
-

 Summary: Fix Calcite assertions
 Key: BEAM-3417
 URL: https://issues.apache.org/jira/browse/BEAM-3417
 Project: Beam
  Issue Type: Task
  Components: dsl-sql
Reporter: Anton Kedin
Assignee: Xu Mingmin


Currently we disable assertions in test for every project which depends on Beam 
SQL / Calcite. Otherwise it fails assertions when Calcite validates relational 
representation of the query. E.g. in `build.gradle` you have to specify 
```java
test {
 jvmArgs "-da" 
}
```
We need to either update our relational conversion logic or come up with some 
other solution so that we don't have to disable assertions globally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313491#comment-16313491
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

robertwb closed pull request #4344: [BEAM-1866] Plumb user metrics through Fn 
API.
URL: https://github.com/apache/beam/pull/4344
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto 
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index ca23c619f33..36ed4242d6c 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -280,11 +280,44 @@ message Metrics {
 
   // User defined metrics
   message User {
-// TODO: Define it.
+
+// A key for identifying a metric at the most granular level.
+message MetricKey {
+  // The step, if any, this metric is associated with.
+  string step = 1;
+
+  // (Required): The namespace of this metric.
+  string namespace = 2;
+
+  // (Required): The name of this metric.
+  string name = 3;
+}
+
+// Data associated with a counter metric.
+message CounterData {
+  int64 value = 1;
+}
+
+// Data associated with a distribution metric.
+message DistributionData {
+  int64 count = 1;
+  int64 sum = 2;
+  int64 min = 3;
+  int64 max = 4;
+}
+
+// (Required) The identifier for this metric.
+MetricKey key = 1;
+
+// (Required) The data for this metric.
+oneof data {
+  CounterData counter_data = 1001;
+  DistributionData distribution_data = 1002;
+}
   }
 
   map ptransforms = 1;
-  map user = 2;
+  repeated User user = 2;
 }
 
 message ProcessBundleProgressResponse {
diff --git a/sdks/python/apache_beam/metrics/cells.py 
b/sdks/python/apache_beam/metrics/cells.py
index ba840f752b1..2b5a4e4094d 100644
--- a/sdks/python/apache_beam/metrics/cells.py
+++ b/sdks/python/apache_beam/metrics/cells.py
@@ -28,6 +28,7 @@
 
 from apache_beam.metrics.metricbase import Counter
 from apache_beam.metrics.metricbase import Distribution
+from apache_beam.portability.api import beam_fn_api_pb2
 
 __all__ = ['DistributionResult']
 
@@ -286,10 +287,18 @@ def combine(self, other):
 new_min,
 new_max)
 
-  @classmethod
-  def singleton(cls, value):
+  @staticmethod
+  def singleton(value):
 return DistributionData(value, 1, value, value)
 
+  def to_runner_api(self):
+return beam_fn_api_pb2.Metrics.User.DistributionData(
+count=self.count, sum=self.sum, min=self.min, max=self.max)
+
+  @staticmethod
+  def from_runner_api(proto):
+return DistributionData(proto.sum, proto.count, proto.min, proto.max)
+
 
 class MetricAggregator(object):
   """For internal use only; no backwards-compatibility guarantees.
diff --git a/sdks/python/apache_beam/metrics/execution.py 
b/sdks/python/apache_beam/metrics/execution.py
index 1704b98d46a..9e547a983de 100644
--- a/sdks/python/apache_beam/metrics/execution.py
+++ b/sdks/python/apache_beam/metrics/execution.py
@@ -34,6 +34,8 @@
 
 from apache_beam.metrics.cells import CounterCell
 from apache_beam.metrics.cells import DistributionCell
+from apache_beam.metrics.metricbase import MetricName
+from apache_beam.portability.api import beam_fn_api_pb2
 
 
 class MetricKey(object):
@@ -63,6 +65,14 @@ def __str__(self):
   def __hash__(self):
 return hash((self.step, self.metric))
 
+  def to_runner_api(self):
+return beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step, namespace=self.metric.namespace, name=self.metric.name)
+
+  @staticmethod
+  def from_runner_api(proto):
+return MetricKey(proto.step, MetricName(proto.namespace, proto.name))
+
 
 class MetricResult(object):
   """Keeps track of the status of a metric within a single bundle.
@@ -192,6 +202,20 @@ def get_cumulative(self):
 """
 return self._get_updates()
 
+  def to_runner_api(self):
+return (
+[beam_fn_api_pb2.Metrics.User(
+key=beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step_name, namespace=k.namespace, name=k.name),
+counter_data=beam_fn_api_pb2.Metrics.User.CounterData(
+value=v.get_cumulative()))
+ for k, v in self.counters.items()] +
+[beam_fn_api_pb2.Metrics.User(
+key=beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step_name, namespace=k.namespace, name=k.name),
+distribution_data=v.get_cumulative().to_runner_api())
+ for k, v in self.distributions.items()])
+
 
 class ScopedMetricsContainer(object):
 
diff --git 

[beam] 01/01: Merge pull request #4344 from robertwb/user-metrics

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d2690fae0cd3c16b18973ed68f514a2f2319af2b
Merge: 63e2696 a59b767
Author: Robert Bradshaw 
AuthorDate: Fri Jan 5 09:30:25 2018 -0800

Merge pull request #4344 from robertwb/user-metrics

[BEAM-1866] Plumb user metrics through Fn API.

 .../fn-execution/src/main/proto/beam_fn_api.proto  |  37 ++-
 sdks/python/apache_beam/metrics/cells.py   |  13 +-
 sdks/python/apache_beam/metrics/execution.py   |  24 ++
 .../runners/portability/fn_api_runner.py   | 365 ++---
 .../runners/portability/fn_api_runner_test.py  |  31 +-
 .../apache_beam/runners/worker/bundle_processor.py |  11 +-
 .../apache_beam/runners/worker/operations.py   |   8 +-
 .../apache_beam/runners/worker/sdk_worker.py   |   6 +-
 8 files changed, 357 insertions(+), 138 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] branch master updated (63e2696 -> d2690fa)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 63e2696  Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix
 add a59b767  Plumb user metrics through Fn API.
 new d2690fa  Merge pull request #4344 from robertwb/user-metrics

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../fn-execution/src/main/proto/beam_fn_api.proto  |  37 ++-
 sdks/python/apache_beam/metrics/cells.py   |  13 +-
 sdks/python/apache_beam/metrics/execution.py   |  24 ++
 .../runners/portability/fn_api_runner.py   | 365 ++---
 .../runners/portability/fn_api_runner_test.py  |  31 +-
 .../apache_beam/runners/worker/bundle_processor.py |  11 +-
 .../apache_beam/runners/worker/operations.py   |   8 +-
 .../apache_beam/runners/worker/sdk_worker.py   |   6 +-
 8 files changed, 357 insertions(+), 138 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Created] (BEAM-3416) File is not properly close in VcfSource when exception is thrown

[Asha Rostamianfar (JIRA)]

Asha Rostamianfar created BEAM-3416:
---

 Summary: File is not properly close in VcfSource when exception is 
thrown
 Key: BEAM-3416
 URL: https://issues.apache.org/jira/browse/BEAM-3416
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Asha Rostamianfar
Assignee: Asha Rostamianfar
Priority: Minor


Files are not properly closed when an exception is thrown in VcfSource 
(apache_beam/io/vcfio.py).
This is because the file is opened within TextSource (using a 'with' 
statement), but the exception in VcfSource is thrown outside of that scope.

This only causes an issue in Windows unit tests where files cannot be deleted 
if they're used by another process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3416) File is not properly close in VcfSource when exception is thrown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313390#comment-16313390
 ] 

ASF GitHub Bot commented on BEAM-3416:
--

arostamianfar opened a new pull request #4349: [BEAM-3416] Fixes unclosed files 
in VcfSource when exception is thrown.
URL: https://github.com/apache/beam/pull/4349
 
 
   Throw the exception inside TextSource, which ensures that the file is 
properly closed inside the 'with' statement inside TextSource.
   
   @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> File is not properly close in VcfSource when exception is thrown
> 
>
> Key: BEAM-3416
> URL: https://issues.apache.org/jira/browse/BEAM-3416
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Asha Rostamianfar
>Assignee: Asha Rostamianfar
>Priority: Minor
>
> Files are not properly closed when an exception is thrown in VcfSource 
> (apache_beam/io/vcfio.py).
> This is because the file is opened within TextSource (using a 'with' 
> statement), but the exception in VcfSource is thrown outside of that scope.
> This only causes an issue in Windows unit tests where files cannot be deleted 
> if they're used by another process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #4627

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-3370) Add ability to stage directories with compiled classes to Flink

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3370:
--

Assignee: (was: Aljoscha Krettek)

> Add ability to stage directories with compiled classes to Flink
> ---
>
> Key: BEAM-3370
> URL: https://issues.apache.org/jira/browse/BEAM-3370
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Łukasz Gajowy
>Priority: Minor
>
> Currently, when _filesToStage_ contain a path to directory with resources, 
> flink runner throws a {{"java.io.FileNotFoundException:  (Is 
> a directory)"}}. A way to include directory resources would be helpful. 
> This "blocker" occurs while trying to run IOITs on flink runner, which 
> basically makes it impossible/very inconvenient to run. When the tests are 
> run via "mvn verify" command, a "test-classes" *directory* gets detected by 
> detectClasspathResourcesToStage() method which in turn causes the above 
> error. 
> One way to solve this issue is to package the directories to jars with unique 
> names and update the paths accordingly before staging the files on flink. 
> Something similar is already done in the Dataflow runner 
> ([GcsStager|https://github.com/apache/beam/blob/cd186a531aaff0b21cf009b034e1a41f7e7b64af/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java#L74]),
>  more specifically in 
> [PackageUtil|https://github.com/apache/beam/blob/cd186a531aaff0b21cf009b034e1a41f7e7b64af/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L280]
>  class. We are able to run the tests on dataflow thanks to that. 
> As I checked in a [small experiment of 
> mine|https://github.com/lgajowy/beam/commits/spark-and-flink-run-tests], 
> providing analogous change makes it possible to run the tests on a Flink 
> cluster. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3359) Unable to change "flinkMaster" from "[auto]" in TestFlinkRunner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3359:
--

Assignee: (was: Aljoscha Krettek)

> Unable to change "flinkMaster" from "[auto]" in TestFlinkRunner
> ---
>
> Key: BEAM-3359
> URL: https://issues.apache.org/jira/browse/BEAM-3359
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Łukasz Gajowy
>Priority: Minor
>
> In TestFlinkRunner's constructor there is a line like this:
> {{options.setFlinkMaster("\[auto\]");}}
> which basically ignores any "flinkMaster" provided earlier (eg. using command 
> line) leading to  errors that are hard to find (for example wondering: "i 
> provided good url in pipeline options... why is it not connecting to my 
> cluster?). 
> Setting a {{@Default.String("\[auto\]")}} in FlinkPipelineOptions could be 
> one solution I guess. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3379) PAssert not successful when running IOITs with Flink runner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313338#comment-16313338
 ] 

Aljoscha Krettek commented on BEAM-3379:


Could you check whether this PR fixes it? 
https://github.com/apache/beam/pull/4348

I'm assuming the File IOs use SDF and this PR fixes problems around that that 
have to do with processing-time timers being dropped.

> PAssert not successful when running IOITs with Flink runner
> ---
>
> Key: BEAM-3379
> URL: https://issues.apache.org/jira/browse/BEAM-3379
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Łukasz Gajowy
>Assignee: Aljoscha Krettek
>
> When running IOIT tests on Flink either with command:
> {{mvn clean verify \-pl sdks/java/io/file-based-io-tests/ \-Pio-it 
> \-DintegrationTestPipelineOptions='\["\-\-runner=FlinkRunner", 
> "\-\-flinkMaster=localhost:6123", "\-\-numberOfRecords=10", 
> "\-\-filenamePrefix=TEST", 
> "\-\-filesToStage=/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT-shaded.jar,/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT-tests.jar"\]'
>  -Dit.test=TextIOIT -Pflink-runner}}
> or without the filesToStage argument but using modifications described in 
> BEAM-3370, we get the following error, even though the assertion is succesful 
> (checked manually):
> java.lang.AssertionError:
> Expected 1 successful assertions, but found 0.
> Expected: is <1L>
>  but: was <0L>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at 
> org.apache.beam.sdk.testing.TestPipeline.verifyPAssertsSucceeded(TestPipeline.java:541)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:359)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:340)
>   at 
> org.apache.beam.sdk.io.text.TextIOIT.writeThenReadAll(TextIOIT.java:109)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:329)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313329#comment-16313329
 ] 

Aljoscha Krettek commented on BEAM-3414:


I openen a PR for a related bug and I think this issue should also be fixed by 
the change: https://github.com/apache/beam/pull/4348

The reason for the issue here is that processing-time timers don't block 
shutdown of a pipeline.

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Kenneth Knowles
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3883

[Apache Jenkins Server]

See 


--
[...truncated 1.04 MB...]
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #585

[Apache Jenkins Server]

See 


--
[...truncated 73.36 KB...]
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, NoneType], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[NoneType, 
Tuple[Any, List[Any]]], Tuple[NoneType, Tuple[Any, NoneType]]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[Any, List[Any]], 
Tuple[Any, NoneType]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[NoneType, 
Tuple[Any, List[Any]]], Tuple[NoneType, Tuple[Any, NoneType]]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
:133:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[Any, List[Any]], 
Tuple[Any, NoneType]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.4.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.4.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz

[jira] [Commented] (BEAM-3415) JUnit5 support

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313251#comment-16313251
 ] 

Jean-Baptiste Onofré commented on BEAM-3415:


+1 to start a PoC around that.

> JUnit5 support
> --
>
> Key: BEAM-3415
> URL: https://issues.apache.org/jira/browse/BEAM-3415
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3415) JUnit5 support

[Romain Manni-Bucau (JIRA)]

Romain Manni-Bucau created BEAM-3415:


 Summary: JUnit5 support
 Key: BEAM-3415
 URL: https://issues.apache.org/jira/browse/BEAM-3415
 Project: Beam
  Issue Type: New Feature
  Components: testing
Reporter: Romain Manni-Bucau
Assignee: Romain Manni-Bucau






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #4625

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313138#comment-16313138
 ] 

ASF GitHub Bot commented on BEAM-2140:
--

aljoscha opened a new pull request #4348: [BEAM-2140] Fix SplittableDoFn 
ValidatesRunner tests in Flink Runner
URL: https://github.com/apache/beam/pull/4348
 
 
   I managed to sneak a change into Flink 1.4.0 that allows blocking on pending 
process-time timers. Now we can finally fix those tests for the Flink Runner.  
   
   Most of these changes are pretty straightforward, the only controversial 
change is using the output WM instead of the input WM for determining lateness 
in `StateFulDoFnRunner`. This is there to make sure that timers (which are 
elements to the `ProcessFn`) are not dropped when the input watermark goes to 
`+Inf`. I tried changing `FlinkTimerInternals.currentInputWatermarkTime()` to 
be held back by the watermark hold but that doesn't work because windows and 
stuff will then never fire.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4656

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2377) Cross compile flink runner to scala 2.11

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313077#comment-16313077
 ] 

ASF GitHub Bot commented on BEAM-2377:
--

aljoscha closed pull request #3881: [BEAM-2377] Allow cross compilation 
(2.10,2.11) for flink runner
URL: https://github.com/apache/beam/pull/3881
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ade4cac7a49..817af333690 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -95,7 +95,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
   runtime
   
 
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 585d7b8ce2d..f27f6dfbe2a 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -95,7 +95,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
   runtime
   
 
diff --git a/pom.xml b/pom.xml
index 236645cb989..f112c6409a8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -154,6 +154,7 @@
 1.1.4
 0.10.1.0
 1.4
+2.11
 
 1.5.0.Final
 2.0
@@ -363,6 +364,19 @@
 
   
 
+
+
+  flink-scala-2.10
+  
+
+  flink-scala-2.10
+
+  
+  
+2.10
+  
+
+
   
 
   
@@ -606,7 +620,7 @@
 
   
 org.apache.beam
-beam-runners-flink_2.10
+beam-runners-flink_${flink.scala.version}
 ${project.version}
   
 
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 0ef19312b84..5c680c86861 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -26,7 +26,7 @@
 ../pom.xml
   
 
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
   Apache Beam :: Runners :: Flink
   jar
 
@@ -165,7 +165,7 @@
 
 
   org.apache.flink
-  flink-clients_2.10
+  flink-clients_${flink.scala.version}
   ${flink.version}
 
 
@@ -189,13 +189,13 @@
 
 
   org.apache.flink
-  flink-runtime_2.10
+  flink-runtime_${flink.scala.version}
   ${flink.version}
 
 
 
   org.apache.flink
-  flink-streaming-java_2.10
+  flink-streaming-java_${flink.scala.version}
   ${flink.version}
 
 
@@ -210,7 +210,7 @@
 
 
   org.apache.flink
-  flink-runtime_2.10
+  flink-runtime_${flink.scala.version}
   ${flink.version}
   test-jar
   test
@@ -336,7 +336,7 @@
 
 
   org.apache.flink
-  flink-streaming-java_2.10
+  flink-streaming-java_${flink.scala.version}
   ${flink.version}
   test
   test-jar
@@ -344,7 +344,7 @@
 
 
   org.apache.flink
-  flink-test-utils_2.10
+  flink-test-utils_${flink.scala.version}
   ${flink.version}
   test
   
diff --git a/sdks/java/javadoc/pom.xml b/sdks/java/javadoc/pom.xml
index 35f0b864e51..1d90046838c 100644
--- a/sdks/java/javadoc/pom.xml
+++ b/sdks/java/javadoc/pom.xml
@@ -64,7 +64,7 @@
 
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
 
 
 
diff --git 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index ffdb0660466..9257989ddf7 100644
--- 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -215,7 +215,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_@flink.scala.version@
   ${beam.version}
   runtime
 
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index b8b9c9f0fa4..59bd5c211a4 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -214,7 +214,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_@flink.scala.version@
   ${beam.version}
   runtime
 
diff --git a/sdks/java/nexmark/pom.xml b/sdks/java/nexmark/pom.xml
index 67628a12831..d7e41759ef3 100644
--- a/sdks/java/nexmark/pom.xml
+++ b/sdks/java/nexmark/pom.xml
@@ -67,7 +67,7 @@
   
 
   

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #62

[Apache Jenkins Server]

See 


--
[...truncated 54.73 KB...]
2018-01-05 12:23:45,839 118e34ba MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 12:23:45,840 118e34ba MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 12:24:02,111 118e34ba MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json
2018-01-05 12:24:02,705 118e34ba MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 12:24:02,705 118e34ba MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 12:24:20,061 118e34ba MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json
2018-01-05 12:24:20,578 118e34ba MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 12:24:20,579 118e34ba MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 12:24:38,188 118e34ba MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json
2018-01-05 12:24:38,742 118e34ba MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 12:24:38,742 118e34ba MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 12:24:55,130 118e34ba MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json
2018-01-05 12:24:55,824 118e34ba MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/118e34ba/perfkit-bq-pubvYKcz4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for 

Build failed in Jenkins: beam_PerformanceTests_Python #755

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 63e269604a437f7110e15a8fcd27257864c9100d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 63e269604a437f7110e15a8fcd27257864c9100d
Commit message: "Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix"
 > git rev-list 63e269604a437f7110e15a8fcd27257864c9100d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4671337982530349083.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4450922901595761156.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2164954642409653164.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))

Build failed in Jenkins: beam_PerformanceTests_Spark #1199

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 63e269604a437f7110e15a8fcd27257864c9100d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 63e269604a437f7110e15a8fcd27257864c9100d
Commit message: "Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix"
 > git rev-list 63e269604a437f7110e15a8fcd27257864c9100d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5848700061896592629.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1834753808728598760.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins76073147736329815.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already 

[jira] [Assigned] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3409:
--

Assignee: Jean-Baptiste Onofré  (was: Thomas Groh)

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312950#comment-16312950
 ] 

Łukasz Gajowy commented on BEAM-3406:
-

I've found an example schema in the Perfkit repo, which is a superset of what 
we need: 
https://github.com/GoogleCloudPlatform/PerfKitExplorer/blob/master/data/samples_mart/results_table_schema.json

The minimum schema we need is in the issue attachment. It solves the issue as I 
checked locally. I also added a sample result that gets published to bq in case 
it's needed.

[~jasonkuster] [~alanmyrvold] that would be great if you could help. :)

> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
> Attachments: schema, textioit
>
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Łukasz Gajowy updated BEAM-3406:

Attachment: schema
textioit

Schema and sample test results for Bigquery

> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
> Attachments: schema, textioit
>
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312910#comment-16312910
 ] 

Łukasz Gajowy commented on BEAM-981:


[~iemejia] Thanks! I have other stuff on my plate now so anyone can feel free 
to take it.

I think its best to have directories transformed to jars as proposed in 3371. 
This way we would get rid of the error (and others of this kind) and allow 
IOITs on spark so that would be great if its done. Is using the --filesToStage 
to stage already packaged jars instead of detecting them not an option in your 
case?

[~jbonofre] Thanks for taking the 3371 issue. I really look forward for the 
solution! Feel free to contact me if something is unclear. :) 

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Łukasz Gajowy
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
> 

[jira] [Assigned] (BEAM-3043) Set user-specified Transform names on Flink operations

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3043:
--

Assignee: (was: Aljoscha Krettek)

> Set user-specified Transform names on Flink operations
> --
>
> Key: BEAM-3043
> URL: https://issues.apache.org/jira/browse/BEAM-3043
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>
> Currently, we don't always set a name on the generated operations or we set 
> the wrong name. For example, in the batch translation we set the result of 
> {{PTransform.getName()}} as the name, which is only the name of the 
> {{PTransform}} itself, not the name that the user specified when creating a 
> Pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2873) Detect number of shards for file sink in Flink Streaming Runner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-2873:
---
Summary: Detect number of shards for file sink in Flink Streaming Runner  
(was: Detect number of sharsd for file sink in Flink Streaming Runner)

> Detect number of shards for file sink in Flink Streaming Runner
> ---
>
> Key: BEAM-2873
> URL: https://issues.apache.org/jira/browse/BEAM-2873
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>
> [~reuvenlax] mentioned that this is done for the Dataflow Runner and the 
> default behaviour on Flink can be somewhat surprising for users.
> ML entry: https://www.mail-archive.com/dev@beam.apache.org/msg02665.html:
> This is how the file sink has always worked in Beam. If no sharding is 
> specified, then this means runner-determined sharding, and by default that is 
> one file per bundle. If Flink has small bundles, then I suggest using the 
> withNumShards method to explicitly pick the number of output shards.
> The Flink runner can detect that runner-determined sharding has been chosen, 
> and override it with a specific number of shards. For example, the Dataflow 
> streaming runner (which as you mentioned also has small bundles) detects this 
> case and sets the number of out files shards based on the number of workers 
> in the worker pool 
> [Here|https://github.com/apache/beam/blob/9e6530adb00669b7cf0f01cb8b128be0a21fd721/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L354]
>  is the code that does this; it should be quite simple to do something 
> similar for Flink, and then there will be no need for users to explicitly 
> call withNumShards themselves.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2873) Detect number of sharsd for file sink in Flink Streaming Runner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-2873:
--

Assignee: (was: Aljoscha Krettek)

> Detect number of sharsd for file sink in Flink Streaming Runner
> ---
>
> Key: BEAM-2873
> URL: https://issues.apache.org/jira/browse/BEAM-2873
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>
> [~reuvenlax] mentioned that this is done for the Dataflow Runner and the 
> default behaviour on Flink can be somewhat surprising for users.
> ML entry: https://www.mail-archive.com/dev@beam.apache.org/msg02665.html:
> This is how the file sink has always worked in Beam. If no sharding is 
> specified, then this means runner-determined sharding, and by default that is 
> one file per bundle. If Flink has small bundles, then I suggest using the 
> withNumShards method to explicitly pick the number of output shards.
> The Flink runner can detect that runner-determined sharding has been chosen, 
> and override it with a specific number of shards. For example, the Dataflow 
> streaming runner (which as you mentioned also has small bundles) detects this 
> case and sets the number of out files shards based on the number of workers 
> in the worker pool 
> [Here|https://github.com/apache/beam/blob/9e6530adb00669b7cf0f01cb8b128be0a21fd721/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L354]
>  is the code that does this; it should be quite simple to do something 
> similar for Flink, and then there will be no need for users to explicitly 
> call withNumShards themselves.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3882

[Apache Jenkins Server]

See 


--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #584

[Apache Jenkins Server]

See