[jira] [Created] (BEAM-3752) PAssert doesn't use the correct Coder

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3752:


 Summary: PAssert doesn't use the correct Coder
 Key: BEAM-3752
 URL: https://issues.apache.org/jira/browse/BEAM-3752
 Project: Beam
  Issue Type: Bug
  Components: testing
Affects Versions: 2.3.0
Reporter: Xu Mingmin


Run into issue with below test, that it can pass although I'm sure it should 
fail. It seems {{PAssert}} doesn't use the right Coder to compare 
expected/actual values.

See test case here 
[https://github.com/XuMingmin/beam/blob/5d2bc2aa7c344b8b9adf4ceeac2d7450d6ddec1e/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java#L93-L104]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3751) enforce data validation in org.apache.beam.sdk.values.Row

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3751:


 Summary: enforce data validation in org.apache.beam.sdk.values.Row
 Key: BEAM-3751
 URL: https://issues.apache.org/jira/browse/BEAM-3751
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql, sdk-java-core
Affects Versions: 2.3.0
Reporter: Xu Mingmin


It doesn't verify data when creating a {{Row}} record, below lines have 
type-mismatch but it can pass without error.
{code:java}
RowType subStrRowType = RowSqlType.builder()
.withIntegerField("f_int")
.withBigIntField("f_bigint")
.build();
Row subStrRow = Row.withRowType(subStrRowType).addValues(2, "s").build();
{code}

CC: [~kedin] [~kenn] [~xumingming]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1406

[Apache Jenkins Server]

See 


Changes:

[holden] First pass at fixing all of E999 (invalid parsing) errors in Py3 found

[holden] Fix some raise_from to reraise.

[holden] vcfio somehow has some sort issues. It's not overly important and

[yifanzou] [BEAM-3339] add python RC validation automation

[XuMingmin] [BEAM-591]: Update KafkaIO JavaDoc to reflect new timestamp API. 
(#4749)

[tgroh] Add To/From Proto Round Trip for ExecutableStage

[tgroh] Make GreedyStageFuser a Factory Class

--
[...truncated 94.91 KB...]
'apache-beam-testing:bqjob_r589695653316af0b_0161d5ea4062_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-27 06:19:16,434 46aa4317 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-27 06:19:31,850 46aa4317 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-27 06:19:34,176 46aa4317 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.31s,  CPU:0.37s,  MaxMemory:25184kb 
STDOUT: Upload complete.
Waiting on bqjob_r56208efd51985858_0161d5ea8614_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r56208efd51985858_0161d5ea8614_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r56208efd51985858_0161d5ea8614_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-27 06:19:34,176 46aa4317 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-27 06:19:50,558 46aa4317 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-27 06:19:58,537 46aa4317 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:07.96s,  CPU:0.38s,  MaxMemory:25368kb 
STDOUT: Upload complete.
Waiting on bqjob_r6928ba15d34ce2c9_0161d5ead159_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r6928ba15d34ce2c9_0161d5ead159_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r6928ba15d34ce2c9_0161d5ead159_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-27 06:19:58,538 46aa4317 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-27 06:20:26,058 46aa4317 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-27 06:20:28,358 46aa4317 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

Build failed in Jenkins: beam_PerformanceTests_Python #962

[Apache Jenkins Server]

See 


Changes:

[holden] First pass at fixing all of E999 (invalid parsing) errors in Py3 found

[holden] Fix some raise_from to reraise.

[holden] vcfio somehow has some sort issues. It's not overly important and

[yifanzou] [BEAM-3339] add python RC validation automation

[XuMingmin] [BEAM-591]: Update KafkaIO JavaDoc to reflect new timestamp API. 
(#4749)

[tgroh] Add To/From Proto Round Trip for ExecutableStage

[tgroh] Make GreedyStageFuser a Factory Class

--
[...truncated 1.05 KB...]
Commit message: "Merge pull request #4695: Add To/From Proto Round Trip for 
ExecutableStage"
 > git rev-list --no-walk dd1f0473e93acd144f314c1241a5ca482514c0db # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3070542493726921866.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7281724796329739395.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5031311074825737767.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5587247832453572414.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4857304463611768484.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9025026920863702944.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #190

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

[Willy Lulciuc (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378055#comment-16378055
 ] 

Willy Lulciuc commented on BEAM-3664:
-

[~kenn] Great suggestions. I'll try a few approaches. Feel free to assign this 
ticket to me.

[~iemejia] Happy to update ElasticsearchIO tests as well after I figure out an 
approach.

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6071

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4319

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add To/From Proto Round Trip for ExecutableStage

[tgroh] Make GreedyStageFuser a Factory Class

--
[...truncated 1.02 MB...]
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #4290

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1000

[Apache Jenkins Server]

See 


Changes:

[holden] First pass at fixing all of E999 (invalid parsing) errors in Py3 found

[holden] Fix some raise_from to reraise.

[holden] vcfio somehow has some sort issues. It's not overly important and

[yifanzou] [BEAM-3339] add python RC validation automation

[XuMingmin] [BEAM-591]: Update KafkaIO JavaDoc to reflect new timestamp API. 
(#4749)

--
[...truncated 129.80 KB...]
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 285, in run_pipeline
return_context=True)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4318

[Apache Jenkins Server]

See 


Changes:

[holden] First pass at fixing all of E999 (invalid parsing) errors in Py3 found

[holden] Fix some raise_from to reraise.

[holden] vcfio somehow has some sort issues. It's not overly important and

[yifanzou] [BEAM-3339] add python RC validation automation

[XuMingmin] [BEAM-591]: Update KafkaIO JavaDoc to reflect new timestamp API. 
(#4749)

--
[...truncated 1.02 MB...]
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #6070

[Apache Jenkins Server]

See 

[beam] branch master updated (798e907 -> 8a39c80)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 798e907  Merge pull request #4716 from 
yifanzou/yifan_python_rc_validation
 add 8dfafcc  Add To/From Proto Round Trip for ExecutableStage
 add 51293f7  Make GreedyStageFuser a Factory Class
 new 8a39c80  Merge pull request #4695: Add To/From Proto Round Trip for 
ExecutableStage

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/core/construction/Environments.java|  22 ++
 .../core/construction/graph/ExecutableStage.java   |  84 ++--
 .../core/construction/graph/FusedPipeline.java |  38 
 .../graph/GreedilyFusedExecutableStage.java| 233 -
 .../graph/GreedyPCollectionFusers.java |  38 +++-
 .../construction/graph/GreedyPipelineFuser.java|   4 +-
 .../core/construction/graph/GreedyStageFuser.java  | 178 
 .../graph/ImmutableExecutableStage.java|  52 +
 .../core/construction/graph/QueryablePipeline.java |   2 +-
 .../construction/graph/ExecutableStageTest.java| 156 ++
 ...bleStageTest.java => GreedyStageFuserTest.java} |  34 +--
 11 files changed, 566 insertions(+), 275 deletions(-)
 delete mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedilyFusedExecutableStage.java
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
 create mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageTest.java
 rename 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/{GreedilyFusedExecutableStageTest.java
 => GreedyStageFuserTest.java} (97%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4289

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #6069

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #4716 from yifanzou/yifan_python_rc_validation

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 798e907e35073c313db4503475445f356143563a
Merge: d29f7b8 d4a3102
Author: Ahmet Altay 
AuthorDate: Mon Feb 26 17:26:57 2018 -0800

Merge pull request #4716 from yifanzou/yifan_python_rc_validation

[BEAM-3339] Adding Python RC Validation Automation

 .../job_beam_ReleaseCandidate_Python.groovy|  51 
 .../run_release_candidate_python_validation.sh | 292 +
 2 files changed, 343 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (d29f7b8 -> 798e907)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d29f7b8  Merge pull request #4376 from 
holdenk/BEAM-3444-fix-flake8-errors-e999
 add d4a3102  [BEAM-3339] add python RC validation automation
 new 798e907  Merge pull request #4716 from 
yifanzou/yifan_python_rc_validation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../job_beam_ReleaseCandidate_Python.groovy|  51 
 .../run_release_candidate_python_validation.sh | 292 +
 2 files changed, 343 insertions(+)
 create mode 100644 .test-infra/jenkins/job_beam_ReleaseCandidate_Python.groovy
 create mode 100755 
release/src/main/groovy/run_release_candidate_python_validation.sh

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (ae79032 -> d29f7b8)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ae79032  [BEAM-591]: Update KafkaIO JavaDoc to reflect new timestamp 
API. (#4749)
 add 9850d3f  First pass at fixing all of E999 (invalid parsing) errors in 
Py3 found by flake8
 add 9579372  Fix some raise_from to reraise.
 add 9f7aa9b  vcfio somehow has some sort issues. It's not overly important 
and hopefully we can remove isort after the py3 migration is complete and just 
depend on pylint.
 new d29f7b8  Merge pull request #4376 from 
holdenk/BEAM-3444-fix-flake8-errors-e999

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../examples/complete/game/game_stats.py   |  4 ++--
 sdks/python/apache_beam/examples/complete/tfidf.py |  8 +--
 sdks/python/apache_beam/runners/common.py  |  4 +++-
 sdks/python/apache_beam/runners/direct/executor.py |  4 +++-
 .../runners/portability/fn_api_runner.py   |  2 +-
 .../runners/portability/fn_api_runner_test.py  |  4 +++-
 .../portability/universal_local_runner_test.py |  7 +++---
 .../apache_beam/runners/worker/data_plane.py   |  4 +++-
 .../apache_beam/runners/worker/data_plane_test.py  |  3 ++-
 .../apache_beam/runners/worker/sdk_worker.py   |  4 +++-
 sdks/python/apache_beam/transforms/util.py |  7 +-
 sdks/python/apache_beam/typehints/typecheck.py | 11 +
 .../python/apache_beam/typehints/typehints_test.py | 10 
 sdks/python/apache_beam/utils/retry.py |  3 ++-
 .../teardown.sh => sdks/python/run_mini_py3lint.sh | 28 +-
 sdks/python/run_pylint.sh  |  4 
 sdks/python/tox.ini| 21 ++--
 17 files changed, 95 insertions(+), 33 deletions(-)
 copy .test-infra/kubernetes/cassandra/LargeITCluster/teardown.sh => 
sdks/python/run_mini_py3lint.sh (63%)
 mode change 100644 => 100755

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Comment Edited] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377844#comment-16377844
 ] 

Kenneth Knowles edited comment on BEAM-3749 at 2/27/18 12:39 AM:
-

I don't think the description is quite right - BeamSql doesn't use the 
{{DefaultTrigger}}, does it? I think it uses whatever trigger is set up, which 
could be anything.

What about this?

{code}
PCollectionTuple inputTables = ...;
input.apply(Window.triggering(trigger).discardingFiredPanes()) // or 
accumulatingFiredPanes
.apply(BeamSql.query("..."));
{code}

This should allow the CLI to apply the trigger and accumulation mode outside of 
the SQL transform. This will use the continuation trigger the right way 
everywhere.


was (Author: kenn):
What about this?

{code}
PCollectionTuple inputTables = ...;
input.apply(Window.triggering(trigger).discardingFiredPanes()) // or 
accumulatingFiredPanes
.apply(BeamSql.query("..."));
{code}

This should allow the CLI to apply the trigger and accumulation mode outside of 
the SQL transform. This will use the continuation trigger the right way 
everywhere.

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377844#comment-16377844
 ] 

Kenneth Knowles commented on BEAM-3749:
---

What about this?

{code}
PCollectionTuple inputTables = ...;
input.apply(Window.triggering(trigger).discardingFiredPanes()) // or 
accumulatingFiredPanes
.apply(BeamSql.query("..."));
{code}

This should allow the CLI to apply the trigger and accumulation mode outside of 
the SQL transform. This will use the continuation trigger the right way 
everywhere.

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1405

[Apache Jenkins Server]

See 


Changes:

[github] Fixes typo in BigQueryIO javadoc

--
[...truncated 89.91 KB...]
'apache-beam-testing:bqjob_r117956c159230eb2_0161d4a0fd94_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-27 00:19:37,826 fdb2280d MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-27 00:20:07,042 fdb2280d MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-27 00:20:09,176 fdb2280d MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.12s,  CPU:0.26s,  MaxMemory:25380kb 
STDOUT: Upload complete.
Waiting on bqjob_r691be8fc36f1aa70_0161d4a1780f_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r691be8fc36f1aa70_0161d4a1780f_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r691be8fc36f1aa70_0161d4a1780f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-27 00:20:09,176 fdb2280d MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-27 00:20:25,830 fdb2280d MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-27 00:20:29,056 fdb2280d MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.22s,  CPU:0.27s,  MaxMemory:25432kb 
STDOUT: Upload complete.
Waiting on bqjob_r2c32b7fc70b8d95c_0161d4a1c56a_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r2c32b7fc70b8d95c_0161d4a1c56a_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r2c32b7fc70b8d95c_0161d4a1c56a_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-27 00:20:29,056 fdb2280d MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-27 00:20:58,419 fdb2280d MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-27 00:21:00,662 fdb2280d MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.23s,  CPU:0.26s,  MaxMemory:25212kb 
STDOUT: Upload complete.
Waiting on bqjob_r4fdf6539f411e874_0161d4a240c4_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4fdf6539f411e874_0161d4a240c4_1 ... (0s) 
Current status: DONE   
BigQuery error in load 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #264

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #961

[Apache Jenkins Server]

See 


Changes:

[github] Fixes typo in BigQueryIO javadoc

--
[...truncated 1.41 KB...]
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4243600083477744026.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7114854141122572049.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4152769862278813257.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1226582635770367558.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins931679236486298527.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins370441361712890859.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #189

[Apache Jenkins Server]

See 


Changes:

[github] Fixes typo in BigQueryIO javadoc

--
[...truncated 32.14 KB...]
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0-beta from the 
shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11 
from the shaded jar.

[jira] [Created] (BEAM-3750) Integration test will fail when use TestPipeline.convertToArgs in some special cases

[Mark Liu (JIRA)]

Mark Liu created BEAM-3750:
--

 Summary: Integration test will fail when use 
TestPipeline.convertToArgs in some special cases
 Key: BEAM-3750
 URL: https://issues.apache.org/jira/browse/BEAM-3750
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Mark Liu
Assignee: Mark Liu


ITs(Integration Tests) are using {{TestPipeline.convertToArgs}} to convert 
well-defined options to list of pipeline-recognizable arguments and pass them 
to pipeline main function. A example use case is 
{{WordCountIT}}([https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L70]).

However, incorrect format will be generated by this function if the option that 
we want to convert has attribute type {{List}}.

Ideally, we want to get rid of this complicated conversion step in testing 
which can avoid more problematic cases, but also make integration test better 
understandable.

Some discussions happened in [https://github.com/apache/beam/pull/4346.] One 
possible approach is to extract part of pipeline code so that integration test 
can reference directly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3739) @Parameter annotation does not work for UDFs in Beam SQL

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377748#comment-16377748
 ] 

Xu Mingmin commented on BEAM-3739:
--

[~samwagg], thanks to report the issue, will look into it. With a quick glance, 
it's because that {{DEFAULT}} keyword is not supported.

> @Parameter annotation does not work for UDFs in Beam SQL
> 
>
> Key: BEAM-3739
> URL: https://issues.apache.org/jira/browse/BEAM-3739
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.3.0
>Reporter: Samuel Waggoner
>Assignee: Xu Mingmin
>Priority: Major
>
> BeamSqlUdf javadoc indicates you can have optional parameters, but this 
> functionality is not working. I implemented the following copy/paste from the 
> doc 
> https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.html:
> {code:java}
> public static class MyLeftFunction implements BeamSqlUdf {
>  public String eval(
>  @Parameter(name = "s") String s,
>  @Parameter(name = "n", optional = true) Integer n) {
>  return s.substring(0, n == null ? 1 : n);
>  }
> }{code}
> I modify a query in BeamSqlExample.java to use it. With all parameters 
> supplied, it completes successfully:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
> BeamSql.query("select c1, leftfn('string1', 1) as c2, c3 from PCOLLECTION 
> where c1 > 1")
> .registerUdf("leftfn", MyLeftFunction.class));{code}
> With the optional parameter left off, I get an exception:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
>  BeamSql.query("select c1, leftfn('string1') as c2, c3 from PCOLLECTION where 
> c1 > 1")
>  .registerUdf("leftfn", MyLeftFunction.class));{code}
> {code:java}
> Exception in thread "main" java.lang.IllegalStateException: 
> java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported 
> yet!
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:75)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
>  at 
> org.apache.beam.sdk.extensions.sql.example.BeamSqlExample.main(BeamSqlExample.java:76)
> Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not 
> supported yet!
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:424)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:201)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.(BeamSqlFnExecutor.java:125)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
>  ... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3688) add setup/teardown for BeamSqlSeekableTable

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3688.

   Resolution: Fixed
Fix Version/s: 2.4.0

> add setup/teardown for BeamSqlSeekableTable
> ---
>
> Key: BEAM-3688
> URL: https://issues.apache.org/jira/browse/BEAM-3688
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Add interface \{{setup}} and \{{teardown}} for initial/cleanup work of 
> \{{BeamSqlSeekableTable}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3093.

   Resolution: Won't Fix
Fix Version/s: Not applicable

> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --
>
> Key: BEAM-3093
> URL: https://issues.apache.org/jira/browse/BEAM-3093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: Not applicable
>
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
> cannot be changed in application, to force to read from earliest/latest. 
> --This feature is important to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
> checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in 
> checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_beginning if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_latest, otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3699) RecordTimestamp should be the default Watermark in KafkaIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3699.

   Resolution: Duplicate
Fix Version/s: (was: 2.4.0)
   Not applicable

> RecordTimestamp should be the default Watermark in KafkaIO
> --
>
> Key: BEAM-3699
> URL: https://issues.apache.org/jira/browse/BEAM-3699
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the priority to get Watermark Instance in KafkaIO is:
> {code}
> getWatermarkFn().apply(curRecord)
>   getTimestampFn().apply(record)
>     Instant.now()
> {code}
> I would propose to change it as below to leverage {{KafkaRecord.timestamp}} 
> if no {{WatermarkFn()}} or {{TimestampFn()}} is available:
> {code}
> getWatermarkFn().apply(curRecord)
>   getTimestampFn().apply(record)
> KafkaRecord(Beam.KafkaIO).timestamp
> {code}
>  It equals to
> {code}
> getWatermarkFn().apply(curRecord)
>   getTimestampFn().apply(record)
> KafkaRawRecord(Kafka_client).timestamp
>   Instant.now()
> {code}
> [~rangadi] any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4317

[Apache Jenkins Server]

See 


Changes:

[github] Fixes typo in BigQueryIO javadoc

--
[...truncated 1.03 MB...]
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/util_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/testing
copying apache_beam/testing/data/standard_coders.yaml -> 
apache-beam-2.4.0.dev0/apache_beam/testing/data
copying apache_beam/testing/data/trigger_transcripts.yaml -> 
apache-beam-2.4.0.dev0/apache_beam/testing/data
copying apache_beam/transforms/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners.py -> 
apache-beam-2.4.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/transforms
copying apache_beam/transforms/core.py -> 
apache-beam-2.4.0.dev0/apache_beam/transforms
copying 

[jira] [Assigned] (BEAM-3748) ApiSurfaceTest checks the classpath, not the API surface

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3748:
-

Assignee: (was: Kenneth Knowles)

> ApiSurfaceTest checks the classpath, not the API surface
> 
>
> Key: BEAM-3748
> URL: https://issues.apache.org/jira/browse/BEAM-3748
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>
> The change to check the classloader is not wrong - it catches transitive 
> private deps that are nonetheless susceptible to clobber. However, it is 
> checking the wrong classpath. It needs to be an isolated post-shading 
> classpath, not the generic testing classpath.
> Likely all the uses are broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3207) Publicly document primitive transforms and their URNs - give "impulse" a URN

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3207.
-
   Resolution: Fixed
Fix Version/s: 2.4.0

> Publicly document primitive transforms and their URNs - give "impulse" a URN
> 
>
> Key: BEAM-3207
> URL: https://issues.apache.org/jira/browse/BEAM-3207
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>
> CC [~herohde]
> The "impulse" transform was discussed and introduced in some contexts, since 
> an SDK with only splittable DoFn needs some way to get things moving, but we 
> don't really have a central place where these specs live, except perhaps the 
> proto comments.
> TBD how best to address the fairly amorphous way these things are introduced 
> and propagated.
> Incidentally we have inconsistent version strings in the draft URNs in the 
> Java codebase, so there's plenty that would also improve in technical quality 
> by having more eyes on it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #999

[Apache Jenkins Server]

See 


Changes:

[github] Fixes typo in BigQueryIO javadoc

--
[...truncated 111.56 KB...]
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 814, in to_runner_api
for tag, out in self.named_outputs().items()},
  File 
"
 line 814, in 
for tag, out in self.named_outputs().items()},
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 141, in to_runner_api
coder_id=pickler.dumps(self.element_type),
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 686, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4316

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying 

[beam] branch master updated (dbaf098 -> dd1f047)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from dbaf098  rename method as suggested and declare as default methods.
 add d3e6c2b  Fixes typo in BigQueryIO javadoc
 new dd1f047  Merge pull request #4750: Fixes typo in BigQueryIO javadoc

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch bqio-javadoc deleted (was d3e6c2b)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch bqio-javadoc
in repository https://gitbox.apache.org/repos/asf/beam.git.


 was d3e6c2b  Fixes typo in BigQueryIO javadoc

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #4750: Fixes typo in BigQueryIO javadoc

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dd1f0473e93acd144f314c1241a5ca482514c0db
Merge: dbaf098 d3e6c2b
Author: Eugene Kirpichov 
AuthorDate: Mon Feb 26 13:42:28 2018 -0800

Merge pull request #4750: Fixes typo in BigQueryIO javadoc

Fixes typo in BigQueryIO javadoc

 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam-site] branch asf-site updated (10ec0c0 -> e51719a)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 10ec0c0  Prepare repository for deployment.
 add 3ef7f25  Don't pass the pom.xml as an input
 add 422e81b  This closes #304
 new e51719a  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/wordcount-example/index.html | 10 +-
 src/get-started/wordcount-example.md | 10 +-
 2 files changed, 10 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e51719af8b76669755cb0af04f5f8d7c2f051bb0
Author: Mergebot 
AuthorDate: Mon Feb 26 13:40:53 2018 -0800

Prepare repository for deployment.
---
 content/get-started/wordcount-example/index.html | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/content/get-started/wordcount-example/index.html 
b/content/get-started/wordcount-example/index.html
index eddefd4..a1448c3 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -710,30 +710,30 @@ instrumenting your pipeline code.
 To run this example in Java:
 
 $ 
mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
+ -Dexec.args="--output=counts" -Pdirect-runner
 
 
 
 $ mvn 
compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
+ -Dexec.args="--output=counts --runner=ApexRunner" -Papex-runner
 
 
 
 $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" 
-Pflink-runner
+ -Dexec.args="--runner=FlinkRunner --output=counts" -Pflink-runner
 
 
 
 $ mvn package exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
  -Dexec.args="--runner=FlinkRunner --flinkMaster=flink master 
--filesToStage=target/word-count-beam-bundled-0.1.jar \
-  --inputFile=/path/to/quickstart/pom.xml 
--output=/tmp/counts" -Pflink-runner
+  --output=/tmp/counts" -Pflink-runner
 
 You can monitor the running job by visiting the Flink dashboard at 
http://flink master:8081
 
 
 
 $ mvn 
compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
-Pspark-runner
+ -Dexec.args="--runner=SparkRunner --output=counts" -Pspark-runner
 
 
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Created] (BEAM-3748) ApiSurfaceTest checks the classpath, not the API surface

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3748:
-

 Summary: ApiSurfaceTest checks the classpath, not the API surface
 Key: BEAM-3748
 URL: https://issues.apache.org/jira/browse/BEAM-3748
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


The change to check the classloader is not wrong - it catches transitive 
private deps that are nonetheless susceptible to clobber. However, it is 
checking the wrong classpath. It needs to be an isolated post-shading 
classpath, not the generic testing classpath.

Likely all the uses are broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3505) Allow gRPC users to specify their own dialer

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3505.
-
   Resolution: Fixed
Fix Version/s: Not applicable

Fixed in https://github.com/apache/beam/pull/4456

> Allow gRPC users to specify their own dialer
> 
>
> Key: BEAM-3505
> URL: https://issues.apache.org/jira/browse/BEAM-3505
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
> Fix For: Not applicable
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> The implementation of grpcx.Dial makes a number of assumptions that are 
> reasonable in some environments, but it would be good to allow the user to 
> provide their own dialer, so they can specify transport details like 
> security, interceptors, and such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3299) Go SDK support for portable progress reporting

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3299.
-
   Resolution: Fixed
Fix Version/s: Not applicable

Fixed in https://github.com/apache/beam/pull/4219

> Go SDK support for portable progress reporting
> --
>
> Key: BEAM-3299
> URL: https://issues.apache.org/jira/browse/BEAM-3299
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3361) Go SDK needs to increase gRPC receive buffer size

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3361.
-
   Resolution: Fixed
Fix Version/s: Not applicable

Fixed in https://github.com/apache/beam/pull/4285

> Go SDK needs to increase gRPC receive buffer size
> -
>
> Key: BEAM-3361
> URL: https://issues.apache.org/jira/browse/BEAM-3361
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
> Fix For: Not applicable
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The Java gRPC server allows for sending messages larger than the default 4M 
> gRPC limit. The Go SDK needs to handle these messages. I propose increasing 
> the limit to 50M based on empirical observations, and we can tune this value 
> in the future if needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3368) Go model pipeline doesn't support External transform

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3368.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Go model pipeline doesn't support External transform
> 
>
> Key: BEAM-3368
> URL: https://issues.apache.org/jira/browse/BEAM-3368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
> Fix For: Not applicable
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The model pipeline translation process doesn't support the external 
> transform, causing the translation process to panic when it encounters it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3324) symtab.go shouldn't read entire file into memory

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3324.
-
   Resolution: Fixed
Fix Version/s: Not applicable

Fixed in https://github.com/apache/beam/pull/4269

> symtab.go shouldn't read entire file into memory
> 
>
> Key: BEAM-3324
> URL: https://issues.apache.org/jira/browse/BEAM-3324
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
> Fix For: Not applicable
>
>
> The implementation of symtab.go reads the entire binary into memory. This is 
> wasteful of memory, and it should just use os.File as the backing reader. If 
> performance becomes an issue, we can use a modest amount of memory to cache 
> lookups and avoid filesystem reads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3473) Go SDK: GroupByKey outputs cause the SDK harness to panic

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3473.
-
   Resolution: Fixed
Fix Version/s: Not applicable

Fixed in https://github.com/apache/beam/pull/4403

> Go SDK: GroupByKey outputs cause the SDK harness to panic
> -
>
> Key: BEAM-3473
> URL: https://issues.apache.org/jira/browse/BEAM-3473
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Major
> Fix For: Not applicable
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/exec/fn.go#L81
> The call to makeIter returns the ReusableInput, but Init() has not been 
> called on it. While this will be called for side inputs, this isn't happening 
> in the main input case, and causes a panic when the function is called from 
> user code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/02: Don't pass the pom.xml as an input

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 3ef7f25211c48aa8ac2c3276dd8b0e25c40b357f
Author: Ed Hartwell Goose 
AuthorDate: Sat Aug 26 12:21:35 2017 +0100

Don't pass the pom.xml as an input
---
 src/get-started/wordcount-example.md | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/get-started/wordcount-example.md 
b/src/get-started/wordcount-example.md
index 9947330..e24fa07 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -509,26 +509,26 @@ instrumenting your pipeline code.
 {:.runner-direct}
 ```
 $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts" -Pdirect-runner
+ -Dexec.args="--output=counts" -Pdirect-runner
 ```
 
 {:.runner-apex}
 ```
 $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
+ -Dexec.args="--output=counts --runner=ApexRunner" -Papex-runner
 ```
 
 {:.runner-flink-local}
 ```
 $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" 
-Pflink-runner
+ -Dexec.args="--runner=FlinkRunner --output=counts" -Pflink-runner
 ```
 
 {:.runner-flink-cluster}
 ```
 $ mvn package exec:java 
-Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
  -Dexec.args="--runner=FlinkRunner --flinkMaster= 
--filesToStage=target/word-count-beam-bundled-0.1.jar \
-  --inputFile=/path/to/quickstart/pom.xml 
--output=/tmp/counts" -Pflink-runner
+  --output=/tmp/counts" -Pflink-runner
 
 You can monitor the running job by visiting the Flink dashboard at 
http://:8081
 ```
@@ -536,7 +536,7 @@ You can monitor the running job by visiting the Flink 
dashboard at http://

[jira] [Resolved] (BEAM-3474) Go SDK: panic handling swallows the stack trace, hurting debuggability

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer resolved BEAM-3474.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Go SDK: panic handling swallows the stack trace, hurting debuggability
> --
>
> Key: BEAM-3474
> URL: https://issues.apache.org/jira/browse/BEAM-3474
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
> Fix For: Not applicable
>
>
> The panic handlers 
> https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/exec/util.go#L39
> and 
> https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/exec/util.go#L49
> should include the output of "runtime/debug"'s Stack() call so the user can 
> see what caused the panic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (283f624 -> 422e81b)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 283f624  This closes #387
 add 10ec0c0  Prepare repository for deployment.
 new 3ef7f25  Don't pass the pom.xml as an input
 new 422e81b  This closes #304

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/runners/dataflow/index.html | 50 +++
 src/get-started/wordcount-example.md  | 10 ++---
 2 files changed, 55 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #304

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 422e81bc27c5d7d7295c347c944cfc8fea91a703
Merge: 10ec0c0 3ef7f25
Author: Mergebot 
AuthorDate: Mon Feb 26 13:34:27 2018 -0800

This closes #304

 src/get-started/wordcount-example.md | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Commented] (BEAM-3474) Go SDK: panic handling swallows the stack trace, hurting debuggability

[Bill Neubauer (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377610#comment-16377610
 ] 

Bill Neubauer commented on BEAM-3474:
-

Fixed in https://github.com/apache/beam/pull/4404

> Go SDK: panic handling swallows the stack trace, hurting debuggability
> --
>
> Key: BEAM-3474
> URL: https://issues.apache.org/jira/browse/BEAM-3474
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>
> The panic handlers 
> https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/exec/util.go#L39
> and 
> https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/exec/util.go#L49
> should include the output of "runtime/debug"'s Stack() call so the user can 
> see what caused the panic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #998

[Apache Jenkins Server]

See 


--
[...truncated 122.54 KB...]
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 367, in run
if test_runner_api and self._verify_runner_api_compatible():
  File 
"
 line 570, in _verify_runner_api_compatible
self.visit(Visitor())
  File 
"
 line 410, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 762, in visit
visitor.enter_composite_transform(self)
  File 
"
 line 555, in enter_composite_transform
self.visit_transform(transform_node)
  File 
"
 line 561, in visit_transform
enable_trace=False),
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
save(cls)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 94, in wrapper
obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)

[jira] [Closed] (BEAM-1581) JSON source and sink

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-1581.
--
   Resolution: Won't Fix
Fix Version/s: Not applicable

AFAICT this has been superseded with AsJsons and ParseJsons, which can be used 
in combination with FileIO.

> JSON source and sink
> 
>
> Key: BEAM-1581
> URL: https://issues.apache.org/jira/browse/BEAM-1581
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Aviem Zur
>Priority: Major
> Fix For: Not applicable
>
>
> JSON source and sink to read/write JSON files.
> Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} 
> which are a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and 
> {{ParseJsons}}
> The {{PCollection}} of objects the user passes to the transform should be 
> embedded in a valid JSON file
> The most common pattern for this is a large object with an array member which 
> holds all the data objects and other members for metadata.
> Examples of public JSON APIs: https://www.sitepoint.com/10-example-json-files/
> Another pattern used is a file which is simply a JSON array of objects.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-1542) Need Source/Sink for Spanner

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-1542.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

This has been in usable shape since 2.1.0.

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.1.0
>
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Fixes typo in BigQueryIO javadoc

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch bqio-javadoc
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d3e6c2b786e24a1ee20aa79f3a043d90e309797b
Author: Eugene Kirpichov 
AuthorDate: Mon Feb 26 12:07:09 2018 -0800

Fixes typo in BigQueryIO javadoc
---
 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index a983514..f2a77f2 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -1373,7 +1373,7 @@ public class BigQueryIO {
  * Choose the frequency at which file writes are triggered.
  *
  * This is only applicable when the write method is set to {@link 
Method#FILE_LOADS}, and
- * only when writing a bounded {@link PCollection}.
+ * only when writing an unbounded {@link PCollection}.
  *
  * Every triggeringFrequency duration, a BigQuery load job will be 
generated for all the data
  * written since the last load job. BigQuery has limits on how many load 
jobs can be triggered

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch bqio-javadoc created (now d3e6c2b)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch bqio-javadoc
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at d3e6c2b  Fixes typo in BigQueryIO javadoc

This branch includes the following new commits:

 new d3e6c2b  Fixes typo in BigQueryIO javadoc

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Spark #1404

[Apache Jenkins Server]

See 


--
[...truncated 93.91 KB...]
'apache-beam-testing:bqjob_r201f21046367e2d4_0161d356fe74_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-26 18:19:11,439 b31c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-26 18:19:33,741 b31c MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-26 18:19:36,096 b31c MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.34s,  CPU:0.23s,  MaxMemory:25184kb 
STDOUT: Upload complete.
Waiting on bqjob_r3e9989fdf72f4206_0161d3575f1b_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r3e9989fdf72f4206_0161d3575f1b_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3e9989fdf72f4206_0161d3575f1b_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-26 18:19:36,097 b31c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-26 18:20:04,737 b31c MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-26 18:20:07,104 b31c MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.36s,  CPU:0.24s,  MaxMemory:25448kb 
STDOUT: Upload complete.
Waiting on bqjob_rc6ad5a751e8032c_0161d357d807_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_rc6ad5a751e8032c_0161d357d807_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_rc6ad5a751e8032c_0161d357d807_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-26 18:20:07,104 b31c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-26 18:20:27,410 b31c MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-26 18:20:29,810 b31c MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.39s,  CPU:0.22s,  MaxMemory:25360kb 
STDOUT: Upload complete.
Waiting on bqjob_r7ce8df7569192ca4_0161d3583097_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7ce8df7569192ca4_0161d3583097_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_Write_And_Read #2

[Apache Jenkins Server]

See 


--
[...truncated 13.43 KB...]
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/src/sdks/python,
 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_XmlIOIT_Write_And_Read] $ /bin/bash -xe 
/tmp/jenkins5730015950417330630.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.xmlioit_write_and_read_pkb_results.large 

Build failed in Jenkins: beam_PerformanceTests_Python #960

[Apache Jenkins Server]

See 


--
[...truncated 1.41 KB...]
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1129611986494423586.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5474090498311175823.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4809914391112055724.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8886638392695178253.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8556364186068943902.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1373854588036433808.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #188

[Apache Jenkins Server]

See 


--
[...truncated 29.89 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0-beta from the 
shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.

[jira] [Commented] (BEAM-3747) beam_PerformanceTests_JDBC is flaky

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377147#comment-16377147
 ] 

Łukasz Gajowy commented on BEAM-3747:
-

Thanks for creating this issue. I also noticed that but wanted to observe it a 
little bit more. I think this can be related: BEAM-2659 but I'm not sure yet.  

> beam_PerformanceTests_JDBC is flaky
> ---
>
> Key: BEAM-3747
> URL: https://issues.apache.org/jira/browse/BEAM-3747
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/]
> Latest failure is 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/262/]
> [ERROR] org.apache.beam.sdk.io.jdbc.JdbcIOIT Time elapsed: 0 s <<< ERROR!
> org.postgresql.util.PSQLException: The connection attempt failed.
> Łukasz can you take a look ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3747) beam_PerformanceTests_JDBC is flaky

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-3747:


 Summary: beam_PerformanceTests_JDBC is flaky
 Key: BEAM-3747
 URL: https://issues.apache.org/jira/browse/BEAM-3747
 Project: Beam
  Issue Type: Test
  Components: testing
Reporter: Chamikara Jayalath
Assignee: Łukasz Gajowy


[https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/]

Latest failure is 
[https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/262/]

[ERROR] org.apache.beam.sdk.io.jdbc.JdbcIOIT Time elapsed: 0 s <<< ERROR!
org.postgresql.util.PSQLException: The connection attempt failed.

Łukasz can you take a look ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reopened BEAM-3649:


> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
> Fix For: Not applicable
>
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3649.

   Resolution: Duplicate
Fix Version/s: Not applicable

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
> Fix For: Not applicable
>
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía closed BEAM-3649.
--

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
> Fix For: Not applicable
>
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377125#comment-16377125
 ] 

Ismaël Mejía commented on BEAM-3649:


Thanks for confirming, you can also try if you want the new support for native 
S3 via Beam Filesystem that was merged in 2.3.0.

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3649:
---
Fix Version/s: (was: 2.4.0)

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3746) Count.globally should override getIncompatibleGlobalWindowErrorMessage to tell the user the usage that is currently only in javadoc

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3746:
-

 Summary: Count.globally should override 
getIncompatibleGlobalWindowErrorMessage to tell the user the usage that is 
currently only in javadoc
 Key: BEAM-3746
 URL: https://issues.apache.org/jira/browse/BEAM-3746
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles


https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/transforms/Count.html#globally--

"Note: if the input collection uses a windowing strategy other than 
GlobalWindows, use Combine.globally(Count.combineFn()).withoutDefaults() 
instead."

But the actual crash a user gets is:

"java.lang.IllegalStateException: Default values are not supported in 
Combine.globally() if the output PCollection is not windowed by GlobalWindows. 
Instead, use Combine.globally().withoutDefaults() to output an empty 
PCollection if the input PCollection is empty, or 
Combine.globally().asSingletonView() to get the default output of the CombineFn 
if the input PCollection is empty."

There is a method that exists solely to make this actually useful, so we should 
use it!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4315

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying 

[jira] [Closed] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[Guillaume Balaine (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guillaume Balaine closed BEAM-3649.
---
   Resolution: Fixed
Fix Version/s: 2.4.0

Fixed by BEAM-2790

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
> Fix For: 2.4.0
>
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3649) HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

[Guillaume Balaine (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377051#comment-16377051
 ] 

Guillaume Balaine commented on BEAM-3649:
-

Hello Ismaël, thanks, I saw the fix after I rebased on master yesterday. This 
was indeed the error I was getting, I should just submit my patches faster ! 

Closing this.

> HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte 
> ByteBuffers
> --
>
> Key: BEAM-3649
> URL: https://issues.apache.org/jira/browse/BEAM-3649
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Guillaume Balaine
>Priority: Minor
>
> This happened last summer, when I wanted to use S3A as the backing HDFS 
> access implementation. 
> This is because while this method is called : 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145]
> This class does not implement ByteBuffer readable 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> I fixed it by manually incrementing the read position and copying the backing 
> array instead of buffering.
> [https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1]
> I know the s3 direct implementation is the preferred path, but this is 
> possible, and likely happens to a lot of developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #997

[Apache Jenkins Server]

See 


--
[...truncated 121.09 KB...]
==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 285, in run_pipeline
return_context=True)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 839, in to_runner_api_parameter
source=self.source.to_runner_api(context),
  File 
"
 line 94, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_Sink_And_ReadFiles #2

[Apache Jenkins Server]

See 


--
[...truncated 14.34 KB...]
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TextIOIT/src/sdks/python,
 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_Write_And_Read #1

[Apache Jenkins Server]

See 


--
[...truncated 16.02 KB...]
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth-1.4.1-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting cachetools>=2.0.0 (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting future<0.17dev,>=0.16.0 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting ply==3.8 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: avro, dill, protobuf, grpcio, hdfs, httplib2, 
funcsigs, pbr, mock, rsa, pyasn1-modules, oauth2client, pyvcf, typing, 
monotonic, fasteners, google-apitools, googleapis-common-protos, 
proto-google-cloud-datastore-v1, googledatastore, cachetools, google-auth, 
google-auth-httplib2, google-cloud-core, future, ply, google-gax, 
grpc-google-iam-v1, proto-google-cloud-pubsub-v1, gapic-google-cloud-pubsub-v1, 
google-cloud-pubsub, google-cloud-bigquery, pyhamcrest, apache-beam
  Running setup.py develop for apache-beam
Successfully installed apache-beam avro-1.8.2 cachetools-2.0.1 dill-0.2.6 
fasteners-0.14.1 funcsigs-1.0.2 future-0.16.0 
gapic-google-cloud-pubsub-v1-0.15.4 google-apitools-0.5.20 google-auth-1.4.1 
google-auth-httplib2-0.0.3 google-cloud-bigquery-0.25.0 
google-cloud-core-0.25.0 google-cloud-pubsub-0.26.0 google-gax-0.15.16 
googleapis-common-protos-1.5.3 googledatastore-7.0.1 grpc-google-iam-v1-0.11.4 
grpcio-1.9.1 hdfs-2.1.0 httplib2-0.9.2 mock-2.0.0 monotonic-1.4 
oauth2client-4.1.2 pbr-3.1.1 ply-3.8 proto-google-cloud-datastore-v1-0.90.4 
proto-google-cloud-pubsub-v1-0.15.4 protobuf-3.5.1 pyasn1-modules-0.2.1 
pyhamcrest-1.9.0 pyvcf-0.6.8 rsa-3.4.2 typing-3.6.4
[beam_PerformanceTests_XmlIOIT_Write_And_Read] $ /bin/bash -xe 
/tmp/jenkins6790567106374934134.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.xmlioit_write_and_read_pkb_results.large 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1800 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/xml 
--beam_it_class=org.apache.beam.sdk.io.xml.XmlIOIT#writeThenReadViaWriteAndRead 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_XmlIOIT_Write_And_Read/1/,--numberOfRecords=1,--charset=UTF-8]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-26 14:05:47,766 031b3560 MainThread INFO Verbose logging to: 

2018-02-26 14:05:47,766 031b3560 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-370-g8cefd72
2018-02-26 14:05:47,767 031b3560 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.xml.XmlIOIT#writeThenReadViaWriteAndRead
--beam_it_timeout=1800
--beam_it_module=sdks/java/io/xml
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_Sink_And_ReadFiles #1

[Apache Jenkins Server]

See 


--
[...truncated 15.62 KB...]
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/src/sdks/python,
 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_XmlIOIT_Sink_And_ReadFiles] $ /bin/bash -xe 
/tmp/jenkins3211768766171459158.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.xmlioit_sink_and_readFiles_pkb_results.large 

Build failed in Jenkins: beam_PerformanceTests_Spark #1403

[Apache Jenkins Server]

See 


Changes:

[xumingmingv] add setup/teardown for BeamSqlSeekableTable.

[xumingmingv] rename method as suggested and declare as default methods.

--
[...truncated 89.07 KB...]
'apache-beam-testing:bqjob_r2a49997ae99cccb8_0161d20cc8fa_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-26 12:18:30,919 a31980c9 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-26 12:18:59,580 a31980c9 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-26 12:19:03,091 a31980c9 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.50s,  CPU:0.25s,  MaxMemory:26432kb 
STDOUT: Upload complete.
Waiting on bqjob_r4ba1001ec880421a_0161d20d4283_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4ba1001ec880421a_0161d20d4283_1 ... (1s) 
Current status: RUNNING 
 Waiting on 
bqjob_r4ba1001ec880421a_0161d20d4283_1 ... (1s) Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4ba1001ec880421a_0161d20d4283_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-26 12:19:03,091 a31980c9 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-26 12:19:20,863 a31980c9 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-26 12:19:23,174 a31980c9 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.30s,  CPU:0.24s,  MaxMemory:25448kb 
STDOUT: Upload complete.
Waiting on bqjob_r7dac8191457cedb3_0161d20d95a8_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7dac8191457cedb3_0161d20d95a8_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r7dac8191457cedb3_0161d20d95a8_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-26 12:19:23,174 a31980c9 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-26 12:19:46,252 a31980c9 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-26 12:19:48,309 a31980c9 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.05s,  CPU:0.27s,  MaxMemory:25444kb 
STDOUT: Upload complete.
Waiting on 

Build failed in Jenkins: beam_PerformanceTests_Python #959

[Apache Jenkins Server]

See 


Changes:

[xumingmingv] add setup/teardown for BeamSqlSeekableTable.

[xumingmingv] rename method as suggested and declare as default methods.

--
[...truncated 696 B...]
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision dbaf09838c5d064fdd6d953b91ad9a1ce61af72b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f dbaf09838c5d064fdd6d953b91ad9a1ce61af72b
Commit message: "rename method as suggested and declare as default methods."
 > git rev-list --no-walk 2b70d605c9aeed0bf28c22ef222da7b9ad8704d6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2575861871634597877.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4029016668876202358.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins84222403588452940.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins822085707906792.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins320141837776064010.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1460219094022981490.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))

Build failed in Jenkins: beam_PerformanceTests_JDBC #262

[Apache Jenkins Server]

See 


Changes:

[xumingmingv] add setup/teardown for BeamSqlSeekableTable.

[xumingmingv] rename method as suggested and declare as default methods.

--
[...truncated 630.04 KB...]
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.20.1:integration-test (default) @ 
beam-sdks-java-io-jdbc ---
[INFO] Failsafe report directory: 

[INFO] parallel='all', perCoreThreadCount=true, threadCount=4, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0 s <<< 
FAILURE! - in org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] org.apache.beam.sdk.io.jdbc.JdbcIOIT  Time elapsed: 0 s  <<< ERROR!
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
at 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4314

[Apache Jenkins Server]

See 


Changes:

[xumingmingv] add setup/teardown for BeamSqlSeekableTable.

[xumingmingv] rename method as suggested and declare as default methods.

--
[...truncated 1.03 MB...]
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability

[jira] [Commented] (BEAM-959) IO validation messages need improvement

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376645#comment-16376645
 ] 

Jean-Baptiste Onofré commented on BEAM-959:
---

Agree, thanks Ismaël, I forgot to close this one !

> IO validation messages need improvement
> ---
>
> Key: BEAM-959
> URL: https://issues.apache.org/jira/browse/BEAM-959
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> The error messages upon validation of {{JdbcIO}} should be sentences that 
> explain to a user how to rectify the situation. As it is, they look like 
> internal SDK errors.
> Here is where they are: 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L319
> And here is how a user sees them: 
> http://stackoverflow.com/questions/40525269/jdbc-fetch-from-oracle-with-beam



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3726) Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move backwards

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376644#comment-16376644
 ] 

Ismaël Mejía commented on BEAM-3726:


[~pawelbartoszek] Has this by any chance been fixed by the recent fixes on 
Kinesis? Do you have an 'easy' way to reproduce it?

> Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move 
> backwards
> 
>
> Key: BEAM-3726
> URL: https://issues.apache.org/jira/browse/BEAM-3726
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> When the job is restored from savepoint Kinesis Reader throws almost always 
> {{java.lang.IllegalArgumentException: Attempting to move backwards}}
> After a few job restarts caused again by the same exception, job finally 
> starts up and continues to run with no further problems.
> Beam job is reading from 32 shards with parallelism set to 32. Using Flink 
> 1.3.2
>  
> {code:java}
> java.lang.IllegalArgumentException: Attempting to move backwards
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at org.apache.beam.sdk.util.MovingFunction.flush(MovingFunction.java:97)
> at org.apache.beam.sdk.util.MovingFunction.add(MovingFunction.java:114)
> at 
> org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:137)
> at 
> org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeAdvance(ReaderInvocationUtil.java:67)
> at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:264)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> at 
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> at 
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702){code}
>  
> Kinesis Reader transformation configuration:
> {code:java}
> pipeline.apply("KINESIS READER", KinesisIO.read()
> .withStreamName(streamName)
> .withInitialPositionInStream(InitialPositionInStream.LATEST)
> .withAWSClientsProvider(awsAccessKey, awsSecretKey, EU_WEST_1)){code}
>  
> When testing locally I managed to catch this exception. Just before executing 
> this 
> [link|https://github.com/apache/beam/blob/6c93105c2cb7be709c6b3e2e6cdcd09df2b48308/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java#L97]
>  that threw exception I captured the state of the class so that you can 
> replicate the issue
> {code:java}
> org.apache.beam.sdk.util.MovingFunction@71781a[sampleUpdateMs=5000,numSignificantBuckets=2,numSignificantSamples=10,function=org.apache.beam.sdk.transforms.Min$MinLongFn@7909d8d3,buckets={9223372036854775807,9223372036854775807,1519315344334,1519315343759,1519315343770,1519315344086,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807},numSamples={0,0,1,158,156,146,0,0,0,0,144,0},currentMsSinceEpoch=1519315585000,currentIndex=2]{code}
>  
> the add function of MovingFunction was called with nowMsSinceEpoch = 
> 1519315583591
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-959) IO validation messages need improvement

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía closed BEAM-959.
-

> IO validation messages need improvement
> ---
>
> Key: BEAM-959
> URL: https://issues.apache.org/jira/browse/BEAM-959
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> The error messages upon validation of {{JdbcIO}} should be sentences that 
> explain to a user how to rectify the situation. As it is, they look like 
> internal SDK errors.
> Here is where they are: 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L319
> And here is how a user sees them: 
> http://stackoverflow.com/questions/40525269/jdbc-fetch-from-oracle-with-beam



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-959) IO validation messages need improvement

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-959.
---
   Resolution: Fixed
Fix Version/s: Not applicable

I am closing this one since it is quite old and already fixed, also messages in 
more recent IOs follow the pattern proposed here.

> IO validation messages need improvement
> ---
>
> Key: BEAM-959
> URL: https://issues.apache.org/jira/browse/BEAM-959
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> The error messages upon validation of {{JdbcIO}} should be sentences that 
> explain to a user how to rectify the situation. As it is, they look like 
> internal SDK errors.
> Here is where they are: 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L319
> And here is how a user sees them: 
> http://stackoverflow.com/questions/40525269/jdbc-fetch-from-oracle-with-beam



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-1592) Unify HdfsIO and HadoopInputFormatIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía closed BEAM-1592.
--

> Unify HdfsIO and HadoopInputFormatIO
> 
>
> Key: BEAM-1592
> URL: https://issues.apache.org/jira/browse/BEAM-1592
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Reporter: Stephen Sisk
>Priority: Major
> Fix For: Not applicable
>
>
> HIFIO is currently in PR (https://github.com/apache/beam/pull/1994)  and as 
> per discussion in 
> https://lists.apache.org/thread.html/803857877804165e798cf31edf079e6603eb9682b7690d52124c31e7@%3Cdev.beam.apache.org%3E,
>  we'd like to check HIFIO in as-is, then unify the two since they share a lot 
> of code. 
> [~dhalp...@google.com] has mentioned: "the FileInputFormat reader gets to 
> call some special APIs that the
> generic InputFormat reader cannot -- so they are not completely redundant. 
> Specifically, FileInputFormat reader can do size-based splitting." 
> Dan recommended: "See if we can "inline" the FileInputFormat specific parts 
> of HdfsIO inside of HadoopInputFormatIO via reflection. If so, we can get the 
> best of both worlds with shared code." 
> This seems reasonable to me. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-1592) Unify HdfsIO and HadoopInputFormatIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-1592:
--

Assignee: (was: Jean-Baptiste Onofré)

> Unify HdfsIO and HadoopInputFormatIO
> 
>
> Key: BEAM-1592
> URL: https://issues.apache.org/jira/browse/BEAM-1592
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Reporter: Stephen Sisk
>Priority: Major
> Fix For: Not applicable
>
>
> HIFIO is currently in PR (https://github.com/apache/beam/pull/1994)  and as 
> per discussion in 
> https://lists.apache.org/thread.html/803857877804165e798cf31edf079e6603eb9682b7690d52124c31e7@%3Cdev.beam.apache.org%3E,
>  we'd like to check HIFIO in as-is, then unify the two since they share a lot 
> of code. 
> [~dhalp...@google.com] has mentioned: "the FileInputFormat reader gets to 
> call some special APIs that the
> generic InputFormat reader cannot -- so they are not completely redundant. 
> Specifically, FileInputFormat reader can do size-based splitting." 
> Dan recommended: "See if we can "inline" the FileInputFormat specific parts 
> of HdfsIO inside of HadoopInputFormatIO via reflection. If so, we can get the 
> best of both worlds with shared code." 
> This seems reasonable to me. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-1592) Unify HdfsIO and HadoopInputFormatIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-1592.

   Resolution: Invalid
Fix Version/s: Not applicable

I am closing this one since this is not longer valid after the arrival of 
BeamFileSystem.

> Unify HdfsIO and HadoopInputFormatIO
> 
>
> Key: BEAM-1592
> URL: https://issues.apache.org/jira/browse/BEAM-1592
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Reporter: Stephen Sisk
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> HIFIO is currently in PR (https://github.com/apache/beam/pull/1994)  and as 
> per discussion in 
> https://lists.apache.org/thread.html/803857877804165e798cf31edf079e6603eb9682b7690d52124c31e7@%3Cdev.beam.apache.org%3E,
>  we'd like to check HIFIO in as-is, then unify the two since they share a lot 
> of code. 
> [~dhalp...@google.com] has mentioned: "the FileInputFormat reader gets to 
> call some special APIs that the
> generic InputFormat reader cannot -- so they are not completely redundant. 
> Specifically, FileInputFormat reader can do size-based splitting." 
> Dan recommended: "See if we can "inline" the FileInputFormat specific parts 
> of HdfsIO inside of HadoopInputFormatIO via reflection. If so, we can get the 
> best of both worlds with shared code." 
> This seems reasonable to me. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #996

[Apache Jenkins Server]

See 


Changes:

[xumingmingv] add setup/teardown for BeamSqlSeekableTable.

[xumingmingv] rename method as suggested and declare as default methods.

--
[...truncated 116.78 KB...]
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 284, in save
f = self.dispatch.get(t)
  File 
"
 line 376, in get
return self[key]
  File 
"
 line 381, in __missing__
if issubclass(key, type):
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_empty_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 306, in test_flattened_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 367, in run
if test_runner_api and self._verify_runner_api_compatible():
  File 
"
 line 570, in _verify_runner_api_compatible
self.visit(Visitor())
  File 
"
 line 410, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 767, in visit
visitor.visit_transform(self)
  File 
"
 line 562, in visit_transform
enable_trace=False)
  File 
"
 line 221, in loads
return dill.loads(s)
  File 
"
 line 277, in 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4313

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying 

[beam] 01/02: add setup/teardown for BeamSqlSeekableTable.

[xumingming]

This is an automated email from the ASF dual-hosted git repository.

xumingming pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b0991a47a0e1a7bba416be1c0b3781c298966731
Author: mingmxu 
AuthorDate: Sun Feb 25 22:14:25 2018 -0800

add setup/teardown for BeamSqlSeekableTable.
---
 .../apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java | 10 ++
 .../extensions/sql/impl/transform/BeamJoinTransforms.java| 12 +++-
 .../sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java  |  8 
 3 files changed, 29 insertions(+), 1 deletion(-)

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
index d274dd9..95165a5 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
@@ -29,7 +29,17 @@ import org.apache.beam.sdk.values.Row;
 @Experimental
 public interface BeamSqlSeekableTable extends Serializable{
   /**
+   * prepare the instance.
+   */
+  void setup();
+
+  /**
* return a list of {@code Row} with given key set.
*/
   List seekRow(Row lookupSubRow);
+
+  /**
+   * cleanup resources of the instance.
+   */
+  void teardown();
 }
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
index 648b973..2b44814 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
@@ -219,7 +219,12 @@ public class BeamJoinTransforms {
 
 @Override
 public PCollection expand(PCollection input) {
-  return input.apply("join_as_lookup", ParDo.of(new DoFn() {
+  return input.apply("join_as_lookup", ParDo.of(new DoFn(){
+@Setup
+public void setup(){
+  seekableTable.setup();
+}
+
 @ProcessElement
 public void processElement(ProcessContext context) {
   Row factRow = context.element();
@@ -230,6 +235,11 @@ public class BeamJoinTransforms {
   }
 }
 
+@Teardown
+public void teardown(){
+  seekableTable.teardown();
+}
+
 private Row extractJoinSubRow(Row factRow) {
   List joinSubsetValues =
   factJoinIdx
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
index ca639bd..6a06123 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
@@ -131,6 +131,14 @@ public class BeamJoinRelUnboundedVsBoundedTest extends 
BaseRelTest {
 public List seekRow(Row lookupSubRow) {
   return Arrays.asList(Row.withRowType(getRowType()).addValues(1, 
"SITE1").build());
 }
+
+@Override
+public void setup() {
+}
+
+@Override
+public void teardown() {
+}
   }
 
   @Test

-- 
To stop receiving notification emails like this one, please contact
xumingm...@apache.org.

[beam] 02/02: rename method as suggested and declare as default methods.

[xumingming]

This is an automated email from the ASF dual-hosted git repository.

xumingming pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dbaf09838c5d064fdd6d953b91ad9a1ce61af72b
Author: mingmxu 
AuthorDate: Mon Feb 12 08:02:31 2018 -0800

rename method as suggested and declare as default methods.
---
 .../org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java  | 4 ++--
 .../sdk/extensions/sql/impl/transform/BeamJoinTransforms.java | 4 ++--
 .../sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java   | 8 
 3 files changed, 4 insertions(+), 12 deletions(-)

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
index 95165a5..7b256c8 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
@@ -31,7 +31,7 @@ public interface BeamSqlSeekableTable extends Serializable{
   /**
* prepare the instance.
*/
-  void setup();
+  default void setUp(){};
 
   /**
* return a list of {@code Row} with given key set.
@@ -41,5 +41,5 @@ public interface BeamSqlSeekableTable extends Serializable{
   /**
* cleanup resources of the instance.
*/
-  void teardown();
+  default void tearDown(){};
 }
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
index 2b44814..1be47b4 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
@@ -222,7 +222,7 @@ public class BeamJoinTransforms {
   return input.apply("join_as_lookup", ParDo.of(new DoFn(){
 @Setup
 public void setup(){
-  seekableTable.setup();
+  seekableTable.setUp();
 }
 
 @ProcessElement
@@ -237,7 +237,7 @@ public class BeamJoinTransforms {
 
 @Teardown
 public void teardown(){
-  seekableTable.teardown();
+  seekableTable.tearDown();
 }
 
 private Row extractJoinSubRow(Row factRow) {
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
index 6a06123..ca639bd 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
@@ -131,14 +131,6 @@ public class BeamJoinRelUnboundedVsBoundedTest extends 
BaseRelTest {
 public List seekRow(Row lookupSubRow) {
   return Arrays.asList(Row.withRowType(getRowType()).addValues(1, 
"SITE1").build());
 }
-
-@Override
-public void setup() {
-}
-
-@Override
-public void teardown() {
-}
   }
 
   @Test

-- 
To stop receiving notification emails like this one, please contact
xumingm...@apache.org.

[beam] branch master updated (2b70d60 -> dbaf098)

[xumingming]

This is an automated email from the ASF dual-hosted git repository.

xumingming pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2b70d60  Merge pull request #4731: [BEAM-3732] Fix broken maven 
profiles
 new b0991a4  add setup/teardown for BeamSqlSeekableTable.
 new dbaf098  rename method as suggested and declare as default methods.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java | 10 ++
 .../extensions/sql/impl/transform/BeamJoinTransforms.java| 12 +++-
 2 files changed, 21 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
xumingm...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #995

[Apache Jenkins Server]

See 


--
[...truncated 124.56 KB...]
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 839, in to_runner_api_parameter
source=self.source.to_runner_api(context),
  File 
"
 line 94, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 82, in 
pickle_urn, wrappers_pb2.BytesValue(value=pickler.dumps(self
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
save(cls)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 94, in wrapper
obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple

[jira] [Comment Edited] (BEAM-2393) BoundedSource is not fault-tolerant in FlinkRunner Streaming mode

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376528#comment-16376528
 ] 

Grzegorz Kołakowski edited comment on BEAM-2393 at 2/26/18 8:48 AM:


I pushed my initial fix for BEAM-2393 to my 
[fork|https://github.com/grzegorz8/beam/tree/BEAM-2393] (3 commits). I've used 
{{BoundedToUnboundedSourceAdapter}} as suggested.

However, I still struggle with one issue. I've made an additional fix in 
{{UnboundedSourceWrapper}}, which makes reader state update and element 
emission atomic with respect to snapshot. Now the {{reader.advance()}} and 
{{emitElement()}} are in the same synchronized section. It is necessary for two 
reasons. Firstly, without the fix the snapshot is usually taken between 
{{reader.advance()}} and {{emitElement(),}} which I suppose may lead to data 
loss while recovering from such a snapshot. Secondly, 
{{BoundedToUnboundedSourceAdapter.ResidualElements}} and 
{{BoundedToUnboundedSourceAdapter.ResidualSource}} are requested for the next 
element while their states are still uninitialized, which results in 
NoSuchElementException.

Unfortunately, I'm afraid the fix affects snapshoting, namely, it seems to be 
starving now. It takes a minute in average to obtain a savepoint:
{code:java}
$ time ./flink savepoint e6126b77abd47f5a3d3ea383608b4f53 
 Triggering savepoint for job e6126b77abd47f5a3d3ea383608b4f53.
 Waiting for response...
 Savepoint completed. Path: 
 You can resume your program from this savepoint with the run command.

real 1m13.095s{code}
I'm afraid I'm not able to debug the problem on my own. I would appreciate any 
help.


was (Author: grzegorz_kolakowski):
I pushed my initial fix for BEAM-2393 to my 
[fork|https://github.com/grzegorz8/beam/tree/BEAM-2393] (3 commits). I've used 
{{BoundedToUnboundedSourceAdapter}} as suggested.

However, I still struggle with one issue. I've made an additional fix in 
{{UnboundedSourceWrapper}}, which makes reader state update and element 
emission atomic with respect to snapshot. Now the {{reader.advance()}} and 
{{emitElement()}} are in the same synchronized section. It is necessary for two 
reasons, Firstly, without the fix the snapshot is usually taken between 
{{reader.advance()}} and {{emitElement(),}} which I suppose may lead to data 
loss while recovering from such a snapshot. Secondly, 
{{BoundedToUnboundedSourceAdapter.ResidualElements}} and 
{{BoundedToUnboundedSourceAdapter.ResidualSource}} are requested for the next 
element while their state is still uninitialized, which results in 
NoSuchElementException.

Unfortunately, I'm afraid the fix affects snapshoting, namely, it seems to be 
starving now. It takes a minute in average to obtain a savepoint:
{code:java}
$ time ./flink savepoint e6126b77abd47f5a3d3ea383608b4f53 
 Triggering savepoint for job e6126b77abd47f5a3d3ea383608b4f53.
 Waiting for response...
 Savepoint completed. Path: 
 You can resume your program from this savepoint with the run command.

real 1m13.095s{code}
I'm afraid I'm not able to debug the problem on my own. I would appreciate any 
help.

> BoundedSource is not fault-tolerant in FlinkRunner Streaming mode
> -
>
> Key: BEAM-2393
> URL: https://issues.apache.org/jira/browse/BEAM-2393
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
>
> {{BoundedSourceWrapper}} does not implement snapshot() and restore(), when 
> the failure to restart, it will send duplicate data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2393) BoundedSource is not fault-tolerant in FlinkRunner Streaming mode

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376528#comment-16376528
 ] 

Grzegorz Kołakowski commented on BEAM-2393:
---

I pushed my initial fix for BEAM-2393 to my 
[fork|https://github.com/grzegorz8/beam/tree/BEAM-2393] (3 commits). I've used 
{{BoundedToUnboundedSourceAdapter}} as suggested.

However, I still struggle with one issue. I've made an additional fix in 
{{UnboundedSourceWrapper}}, which makes reader state update and element 
emission atomic with respect to snapshot. Now the {{reader.advance()}} and 
{{emitElement()}} are in the same synchronized section. It is necessary for two 
reasons, Firstly, without the fix the snapshot is usually taken between 
{{reader.advance()}} and {{emitElement(),}} which I suppose may lead to data 
loss while recovering from such a snapshot. Secondly, 
{{BoundedToUnboundedSourceAdapter.ResidualElements}} and 
{{BoundedToUnboundedSourceAdapter.ResidualSource}} are requested for the next 
element while their state is still uninitialized, which results in 
NoSuchElementException.

Unfortunately, I'm afraid the fix affects snapshoting, namely, it seems to be 
starving now. It takes a minute in average to obtain a savepoint:
{code:java}
$ time ./flink savepoint e6126b77abd47f5a3d3ea383608b4f53 
 Triggering savepoint for job e6126b77abd47f5a3d3ea383608b4f53.
 Waiting for response...
 Savepoint completed. Path: 
 You can resume your program from this savepoint with the run command.

real 1m13.095s{code}
I'm afraid I'm not able to debug the problem on my own. I would appreciate any 
help.

> BoundedSource is not fault-tolerant in FlinkRunner Streaming mode
> -
>
> Key: BEAM-2393
> URL: https://issues.apache.org/jira/browse/BEAM-2393
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Major
>
> {{BoundedSourceWrapper}} does not implement snapshot() and restore(), when 
> the failure to restart, it will send duplicate data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)