Build failed in Jenkins: beam_PostCommit_Python_Verify #4487

[Apache Jenkins Server]

See 


--
[...truncated 1.14 MB...]
  File "dataflow_worker/native_operations.py", line 95, in 
dataflow_worker.native_operations.NativeWriteOperation.finish
self.writer.__exit__(None, None, None)
  File 
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativeavroio.py", line 
161, in __exit__
self._data_file_writer.close()
  File "/usr/local/lib/python2.7/dist-packages/avro/datafile.py", line 219, in 
close
self.flush()
  File "/usr/local/lib/python2.7/dist-packages/avro/datafile.py", line 215, in 
flush
self.writer.flush()
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filesystemio.py", 
line 186, in write
self._uploader.put(b)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsio.py", 
line 569, in put
self._conn.send_bytes(data.tobytes())
IOError: [Errno 32] Broken pipe

root: INFO: 2018-03-23T05:53:48.620Z: JOB_MESSAGE_ERROR: Traceback (most recent 
call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", 
line 609, in do_work
work_executor.execute()
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", 
line 170, in execute
op.finish()
  File "dataflow_worker/native_operations.py", line 93, in 
dataflow_worker.native_operations.NativeWriteOperation.finish
def finish(self):
  File "dataflow_worker/native_operations.py", line 94, in 
dataflow_worker.native_operations.NativeWriteOperation.finish
with self.scoped_finish_state:
  File "dataflow_worker/native_operations.py", line 95, in 
dataflow_worker.native_operations.NativeWriteOperation.finish
self.writer.__exit__(None, None, None)
  File 
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativeavroio.py", line 
161, in __exit__
self._data_file_writer.close()
  File "/usr/local/lib/python2.7/dist-packages/avro/datafile.py", line 219, in 
close
self.flush()
  File "/usr/local/lib/python2.7/dist-packages/avro/datafile.py", line 215, in 
flush
self.writer.flush()
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filesystemio.py", 
line 186, in write
self._uploader.put(b)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsio.py", 
line 569, in put
self._conn.send_bytes(data.tobytes())
IOError: [Errno 32] Broken pipe

root: INFO: 2018-03-23T05:54:14.111Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/DoOnce/Read.out" materialized.
root: INFO: 2018-03-23T05:54:14.143Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/PreFinalize/MapToVoidKey0.out" materialized.
root: INFO: 2018-03-23T05:54:14.156Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0.out" materialized.
root: INFO: 2018-03-23T05:54:14.179Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/WriteBundles/MapToVoidKey0.out" materialized.
root: INFO: 2018-03-23T05:54:14.211Z: JOB_MESSAGE_BASIC: Executing operation 
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)
root: INFO: 2018-03-23T05:54:14.241Z: JOB_MESSAGE_BASIC: Executing operation 
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)
root: INFO: 2018-03-23T05:54:14.270Z: JOB_MESSAGE_BASIC: Executing operation 
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)
root: INFO: 2018-03-23T05:54:14.313Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0).output"
 materialized.
root: INFO: 2018-03-23T05:54:14.344Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0).output"
 materialized.
root: INFO: 2018-03-23T05:54:14.374Z: JOB_MESSAGE_DEBUG: Value 
"write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0).output"
 materialized.
root: INFO: 2018-03-23T05:54:14.419Z: JOB_MESSAGE_BASIC: Executing operation 
group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
root: INFO: 2018-03-23T05:54:14.864Z: JOB_MESSAGE_ERROR: Traceback (most recent 
call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", 
line 609, in do_work
work_executor.execute()
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", 
line 167, in execute
op.start()
  File "apache_beam/runners/worker/operations.py", line 339, in 
apache_beam.runners.worker.operations.DoOperation.start
def start(self):
  File "apache_beam/runners/worker/operations.py", line 340, in 
apache_beam.runners.worker.operations.DoOperation.start
with self.scoped_start_state:
  File "apache_beam/runners/worker/operations.py", line 372, in 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3756

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3824) Use WriteToBigQuery in Python mobile gaming examples.

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-3824.
---
   Resolution: Fixed
Fix Version/s: 2.5.0

> Use  WriteToBigQuery in Python mobile gaming examples. 
> ---
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

[jira] [Assigned] (BEAM-3824) Use WriteToBigQuery in Python mobile gaming examples.

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-3824:
-

Assignee: Ahmet Altay  (was: David Cavazos)

> Use  WriteToBigQuery in Python mobile gaming examples. 
> ---
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

[jira] [Work logged] (BEAM-3824) Use WriteToBigQuery in Python mobile gaming examples.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?focusedWorklogId=83473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83473
 ]

ASF GitHub Bot logged work on BEAM-3824:


Author: ASF GitHub Bot
Created on: 23/Mar/18 05:27
Start Date: 23/Mar/18 05:27
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #4932: [BEAM-3824] Convert 
big query writes to beam.io.WriteToBigQuery in mobile gaming example
URL: https://github.com/apache/beam/pull/4932
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats.py 
b/sdks/python/apache_beam/examples/complete/game/game_stats.py
index 1f13ed180f6..11b81fe1bd2 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats.py
@@ -40,7 +40,7 @@
 publishing.
 
 To run the Java injector:
-/examples/java8$ mvn compile exec:java \
+/examples/java$ mvn compile exec:java \
 -Dexec.mainClass=org.apache.beam.examples.complete.game.injector.Injector \
 -Dexec.args="$PROJECT_ID $PUBSUB_TOPIC none"
 
@@ -68,13 +68,6 @@
 --dataset $BIGQUERY_DATASET \
 --runner DataflowRunner \
 --temp_location gs://$BUCKET/user_score/temp
-
-
-NOTE [BEAM-2354]: This example is not yet runnable by DataflowRunner.
-The runner still needs support for:
-  * the --save_main_session flag when streaming is enabled
-  * combiners
-
 """
 
 from __future__ import absolute_import
@@ -182,22 +175,19 @@ def get_schema(self):
 return ', '.join(
 '%s:%s' % (col, self.schema[col]) for col in self.schema)
 
-  def get_table(self, pipeline):
-"""Utility to construct an output table reference."""
-project = pipeline.options.view_as(GoogleCloudOptions).project
-return '%s:%s.%s' % (project, self.dataset, self.table_name)
+  def get_schema(self):
+"""Build the output table schema."""
+return ', '.join(
+'%s:%s' % (col, self.schema[col]) for col in self.schema)
 
   def expand(self, pcoll):
-table = self.get_table(pcoll.pipeline)
+project = pcoll.pipeline.options.view_as(GoogleCloudOptions).project
 return (
 pcoll
 | 'ConvertToRow' >> beam.Map(
 lambda elem: {col: elem[col] for col in self.schema})
-| beam.io.Write(beam.io.BigQuerySink(
-table,
-schema=self.get_schema(),
-create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
-write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)))
+| beam.io.WriteToBigQuery(
+self.table_name, self.dataset, project, self.get_schema()))
 
 
 # [START abuse_detect]
diff --git 
a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py 
b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
index b286a6a5ddf..e99ab23ec67 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
@@ -175,22 +175,14 @@ def get_schema(self):
 return ', '.join(
 '%s:%s' % (col, self.schema[col]) for col in self.schema)
 
-  def get_table(self, pipeline):
-"""Utility to construct an output table reference."""
-project = pipeline.options.view_as(GoogleCloudOptions).project
-return '%s:%s.%s' % (project, self.dataset, self.table_name)
-
   def expand(self, pcoll):
-table = self.get_table(pcoll.pipeline)
+project = pcoll.pipeline.options.view_as(GoogleCloudOptions).project
 return (
 pcoll
 | 'ConvertToRow' >> beam.Map(
 lambda elem: {col: elem[col] for col in self.schema})
-| beam.io.Write(beam.io.BigQuerySink(
-table,
-schema=self.get_schema(),
-create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
-write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)))
+| beam.io.WriteToBigQuery(
+self.table_name, self.dataset, project, self.get_schema()))
 
 
 # [START main]
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board.py 
b/sdks/python/apache_beam/examples/complete/game/leader_board.py
index e207f26712e..99a8e092822 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board.py
@@ -48,7 +48,7 @@
 publishing.
 
 To run the Java injector:
-/examples/java8$ mvn compile exec:java \

[beam] branch master updated (f5c07e3 -> 068c76a)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f5c07e3  Merge pull request #4746: correct BigQuery.write JavaDoc 
example
 add 9cb25ee  Convert big query writes to beam.io.WriteToBigQuery
 new 068c76a  Merge pull request #4932 from aaltay/mobi

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../examples/complete/game/game_stats.py   | 26 +++---
 .../examples/complete/game/hourly_team_score.py| 14 +++-
 .../examples/complete/game/leader_board.py | 22 --
 3 files changed, 15 insertions(+), 47 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4932 from aaltay/mobi

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 068c76ab8d3e3b2c0bcb0a72dba71c47f732f0f3
Merge: f5c07e3 9cb25ee
Author: Ahmet Altay 
AuthorDate: Thu Mar 22 22:27:28 2018 -0700

Merge pull request #4932 from aaltay/mobi

[BEAM-3824] Convert big query writes to beam.io.WriteToBigQuery in mobile 
gaming example

 .../examples/complete/game/game_stats.py   | 26 +++---
 .../examples/complete/game/hourly_team_score.py| 14 +++-
 .../examples/complete/game/leader_board.py | 22 --
 3 files changed, 15 insertions(+), 47 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83469
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 05:24
Start Date: 23/Mar/18 05:24
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r176641575
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -113,11 +113,8 @@ def get_version():
 'futures>=3.1.1,<4.0.0',
 ]
 
-REQUIRED_SETUP_PACKAGES = [
-'nose>=1.0',
 
 Review comment:
   @udim Is it required to run `python setup.py test`. If not we can drop it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83469)
Time Spent: 10h 10m  (was: 10h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83468
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 05:20
Start Date: 23/Mar/18 05:20
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r176641300
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -113,11 +113,8 @@ def get_version():
 'futures>=3.1.1,<4.0.0',
 ]
 
-REQUIRED_SETUP_PACKAGES = [
-'nose>=1.0',
 
 Review comment:
   @cclauss Are you referring to a specific issue? `REQUIRED_PIP_VERSION` is 
just the minimum version we require, we are not force downgrading to that 
version. If there is a known issue in compatibility we can consider raising the 
minimum supported version


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83468)
Time Spent: 10h  (was: 9h 50m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83467
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 05:12
Start Date: 23/Mar/18 05:12
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #4930: [BEAM-3861] 
Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#issuecomment-375547198
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83467)
Time Spent: 7h 40m  (was: 7.5h)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3919) checkpoint can not work with flink 1.4.1,1.4.2

[eisig (JIRA)]

eisig created BEAM-3919:
---

 Summary: checkpoint can not work with flink 1.4.1,1.4.2
 Key: BEAM-3919
 URL: https://issues.apache.org/jira/browse/BEAM-3919
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 2.4.0, 2.3.0
Reporter: eisig
Assignee: Aljoscha Krettek


When submmit application to flink cluster(1.4.1,1.4.2) with checkpoint enabled. 

Job fail whith exception:

java.lang.NoSuchMethodError: 
org.apache.flink.streaming.api.operators.HeapInternalTimerService.snapshotTimersForKeyGroup(Lorg/apache/flink/core/memory/DataOutputViewStreamWrapper;I)V

 

It seems that 
`org.apache.flink.streaming.api.operators.HeapInternalTimerService.snapshotTimersForKeyGroup`.
  was changed in flink1.4.1.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5204

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4486

[Apache Jenkins Server]

See 


Changes:

[alex] Correct BigQuery.write JavaDoc example

[ccy] Replace side inputs when applying PTransformOverrides

[Pablo] Updating dataflow API protocol buffers.

--
[...truncated 1.13 MB...]
root: INFO: 2018-03-23T04:09:51.865Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-22_21_09_51-16872146442058122105.
root: INFO: 2018-03-23T04:09:54.060Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-23T04:09:54.156Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-23T04:09:54.360Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-23T04:09:54.380Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-23T04:09:54.401Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step group: GroupByKey not followed by a combiner.
root: INFO: 2018-03-23T04:09:54.427Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-23T04:09:54.454Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-23T04:09:54.486Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-23T04:09:54.516Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-23T04:09:54.541Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/PreFinalize/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-23T04:09:54.570Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-23T04:09:54.581Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/PreFinalize/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-23T04:09:54.613Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-23T04:09:54.645Z: JOB_MESSAGE_DETAILED: Fusing consumer 
pair_with_one into split
root: INFO: 2018-03-23T04:09:54.674Z: JOB_MESSAGE_DETAILED: Fusing consumer 
group/Reify into pair_with_one
root: INFO: 2018-03-23T04:09:54.707Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/GroupByKey/Reify into 
write/Write/WriteImpl/WindowInto(WindowIntoFn)
root: INFO: 2018-03-23T04:09:54.731Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/WindowInto(WindowIntoFn) into write/Write/WriteImpl/Pair
root: INFO: 2018-03-23T04:09:54.756Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/WriteBundles/WriteBundles into format
root: INFO: 2018-03-23T04:09:54.782Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/Pair into write/Write/WriteImpl/WriteBundles/WriteBundles
root: INFO: 2018-03-23T04:09:54.812Z: JOB_MESSAGE_DETAILED: Fusing consumer 
split into read/Read
root: INFO: 2018-03-23T04:09:54.836Z: JOB_MESSAGE_DETAILED: Fusing consumer 
count into group/GroupByWindow
root: INFO: 2018-03-23T04:09:54.857Z: JOB_MESSAGE_DETAILED: Fusing consumer 
format into count
root: INFO: 2018-03-23T04:09:54.889Z: JOB_MESSAGE_DETAILED: Fusing consumer 
group/Write into group/Reify
root: INFO: 2018-03-23T04:09:54.914Z: JOB_MESSAGE_DETAILED: Fusing consumer 
group/GroupByWindow into group/Read
root: INFO: 2018-03-23T04:09:54.928Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/Extract into 
write/Write/WriteImpl/GroupByKey/GroupByWindow
root: INFO: 2018-03-23T04:09:54.951Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/GroupByKey/Write into 
write/Write/WriteImpl/GroupByKey/Reify
root: INFO: 2018-03-23T04:09:54.975Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/GroupByKey/GroupByWindow into 
write/Write/WriteImpl/GroupByKey/Read
root: INFO: 2018-03-23T04:09:54.987Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/PreFinalize/MapToVoidKey0 into 
write/Write/WriteImpl/InitializeWrite
root: INFO: 2018-03-23T04:09:55.018Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0 into 
write/Write/WriteImpl/InitializeWrite
root: INFO: 2018-03-23T04:09:55.040Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/WriteBundles/MapToVoidKey0 into 
write/Write/WriteImpl/InitializeWrite
root: INFO: 2018-03-23T04:09:55.072Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/PreFinalize/MapToVoidKey0 into 
write/Write/WriteImpl/InitializeWrite
root: INFO: 2018-03-23T04:09:55.098Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0 into 
write/Write/WriteImpl/InitializeWrite
root: INFO: 2018-03-23T04:09:55.130Z: JOB_MESSAGE_DETAILED: 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1167

[Apache Jenkins Server]

See 


Changes:

[alex] Correct BigQuery.write JavaDoc example

--
[...truncated 776.20 KB...]
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "compute/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-22_20_33_24-5178576805132623104]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-22_20_33_24-5178576805132623104?project=apache-beam-testing
root: INFO: Job 2018-03-22_20_33_24-5178576805132623104 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-23T03:33:24.944Z: JOB_MESSAGE_WARNING: Job 
2018-03-22_20_33_24-5178576805132623104 might autoscale up to 250 workers.
root: INFO: 2018-03-23T03:33:24.956Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-22_20_33_24-5178576805132623104. The number of workers 
will be between 1 and 250.
root: INFO: 2018-03-23T03:33:24.973Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-22_20_33_24-5178576805132623104.
root: INFO: 2018-03-23T03:33:27.335Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-23T03:33:27.498Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-23T03:33:28.365Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-23T03:33:28.395Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-23T03:33:28.428Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-23T03:33:28.458Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-23T03:33:28.490Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-23T03:33:28.536Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-23T03:33:28.562Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11 for input s10.out
root: INFO: 2018-03-23T03:33:28.584Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_1
root: INFO: 2018-03-23T03:33:28.607Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-23T03:33:28.644Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-23T03:33:28.677Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-23T03:33:28.702Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-23T03:33:28.740Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11-u13 for input s12-reify-value0-c11
root: INFO: 2018-03-23T03:33:28.766Z: JOB_MESSAGE_DETAILED: Fusing 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #6273

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4485

[Apache Jenkins Server]

See 


--
[...truncated 956.40 KB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5203

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1166

[Apache Jenkins Server]

See 


Changes:

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[ccy] Replace side inputs when applying PTransformOverrides

[Pablo] Updating dataflow API protocol buffers.

[boyuanz] Add cython annotation to make DistributionAccumulator faster

[ehudm] Reduce precommit test timeouts for Java and Go.

[altay] More graceful fallback when grpc is not present.

--
[...truncated 1.24 MB...]
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "compute/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-22_19_05_11-7449992449654972149]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-22_19_05_11-7449992449654972149?project=apache-beam-testing
root: INFO: Job 2018-03-22_19_05_11-7449992449654972149 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-23T02:05:11.364Z: JOB_MESSAGE_WARNING: Job 
2018-03-22_19_05_11-7449992449654972149 might autoscale up to 250 workers.
root: INFO: 2018-03-23T02:05:11.379Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-22_19_05_11-7449992449654972149. The number of workers 
will be between 1 and 250.
root: INFO: 2018-03-23T02:05:11.400Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-22_19_05_11-7449992449654972149.
root: INFO: 2018-03-23T02:05:13.893Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-23T02:05:14.049Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-23T02:05:14.839Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-23T02:05:14.864Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-23T02:05:14.891Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-23T02:05:14.913Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-23T02:05:14.938Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-23T02:05:14.974Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-23T02:05:14.998Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11 for input s10.out
root: INFO: 2018-03-23T02:05:15.028Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_1
root: INFO: 2018-03-23T02:05:15.061Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-23T02:05:15.094Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-23T02:05:15.124Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 

[beam] branch master updated (a3bfe6f -> f5c07e3)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a3bfe6f  Merge pull request #4926: Updating dataflow API protocol 
buffers
 add 21a9d73  Correct BigQuery.write JavaDoc example
 new f5c07e3  Merge pull request #4746: correct BigQuery.write JavaDoc 
example

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] 01/01: Merge pull request #4746: correct BigQuery.write JavaDoc example

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f5c07e37f2673aaeb9cea66ccddc964811b12362
Merge: a3bfe6f 21a9d73
Author: Chamikara Jayalath 
AuthorDate: Thu Mar 22 19:07:40 2018 -0700

Merge pull request #4746: correct BigQuery.write JavaDoc example

 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=83441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83441
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:52
Start Date: 23/Mar/18 01:52
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-375514123
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83441)
Time Spent: 0.5h  (was: 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3824) Use WriteToBigQuery in Python mobile gaming examples.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?focusedWorklogId=83440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83440
 ]

ASF GitHub Bot logged work on BEAM-3824:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:51
Start Date: 23/Mar/18 01:51
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4932: [BEAM-3824] 
Convert big query writes to beam.io.WriteToBigQuery in mobile gaming example
URL: https://github.com/apache/beam/pull/4932#issuecomment-375513968
 
 
   LGTM.
   
   Please self merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83440)
Time Spent: 50m  (was: 40m)

> Use  WriteToBigQuery in Python mobile gaming examples. 
> ---
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  

Build failed in Jenkins: beam_PostCommit_Python_Verify #4484

[Apache Jenkins Server]

See 


Changes:

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[boyuanz] Add cython annotation to make DistributionAccumulator faster

[ankurgoenka] Send worker_id in all channels to runner Adding

[ehudm] Reduce precommit test timeouts for Java and Go.

[altay] More graceful fallback when grpc is not present.

--
[...truncated 1.20 MB...]
"output_name": "out", 
"user_name": "pair_with_one.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "ref_AppliedPTransform_pair_with_one_5", 
"user_name": "pair_with_one"
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "group_and_sum/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "group_and_sum/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CombineValuesDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CombineValuesDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "group_and_sum/Combine/ParDo(CombineValuesDoFn).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": 
"ref_AppliedPTransform_group_and_sum/Combine/ParDo(CombineValuesDoFn)_9", 
"user_name": "group_and_sum/Combine/ParDo(CombineValuesDoFn)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s6", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83436
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175934686
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
 
 Review comment:
   Removed. It was convenient for me but I realized it adds 15s to the run.
   You can always pass `--recreate` on the command line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83436)
Time Spent: 9h 40m  (was: 9.5h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83435
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175908871
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -113,11 +113,8 @@ def get_version():
 'futures>=3.1.1,<4.0.0',
 ]
 
-REQUIRED_SETUP_PACKAGES = [
-'nose>=1.0',
 
 Review comment:
   @aaltay setup_requires= lists packages required to run setup.py. I couldn't 
see why nose is required for that.
   Specifying nose in the tests_require= keyword arg is sufficient.
   
   @cclauss I would like to leave it alone in this PR, and I don't have any 
objection to raising the required version.
   Note that creating a new virtualenv installs (at least for me) a version of 
pip that's newer than the system installed version (9.0.2 vs 9.0.1).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83435)
Time Spent: 9.5h  (was: 9h 20m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83438
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175935020
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
+# Don't warn that these commands aren't installed.
+whitelist_externals =
+  find
+  time
+
 [testenv:py27]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
-whitelist_externals=find
 commands =
-  python --version
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83438)
Time Spent: 9h 40m  (was: 9.5h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83434=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83434
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175926483
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -64,7 +64,7 @@ build.dependsOn buildPython
 task lint (dependsOn: 'setupTest') {
   doLast {
 exec {
-  commandLine 'tox', '-e', 'lint', '-c', 'tox.ini'
+  commandLine 'tox', '-e', 'py27-lint', '-c', 'tox.ini'
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83434)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83432
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175903622
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
 
 Review comment:
   These are not packages, but options for pip when installing the dist 
tarball. I reworded the comment above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83432)
Time Spent: 9h 10m  (was: 9h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83437
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r176620089
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -27,44 +28,78 @@ select = E3
 
 # Shared environment options.
 [testenv]
-# Set [] options for pip install, e.g., pip install apache-beam[test].
+# Set [] options for pip installation of apache-beam tarball.
 extras = test
-# Always recreate the virtual environment.
-recreate = True
-# Pass these environment variables to the test environment.
-passenv = TRAVIS*
 # Don't warn that these commands aren't installed.
 whitelist_externals =
   find
   time
 
 [testenv:py27]
 commands =
+  python --version
+  pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  #python setup.py test
   {toxinidir}/run_tox_cleanup.sh
 
 [testenv:py27-cython]
 
 Review comment:
   Please ignore. Testing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83437)
Time Spent: 9h 40m  (was: 9.5h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83433=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83433
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175935645
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
+# Don't warn that these commands aren't installed.
+whitelist_externals =
+  find
+  time
+
 [testenv:py27]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
-whitelist_externals=find
 commands =
-  python --version
-  pip --version
-  # Clean up all previous python generated files.
-  - find apache_beam -type f -name '*.pyc' -delete
-  pip install -e .[test]
+  {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-passenv = TRAVIS*
+  {toxinidir}/run_tox_cleanup.sh
 
-[testenv:py27cython]
+[testenv:py27-cython]
 # cython tests are only expected to work in linux (2.x and 3.x)
 # If we want to add other platforms in the future, it should be:
 # `platform = linux2|darwin|...`
 # See https://docs.python.org/2/library/sys.html#sys.platform for platform 
codes
 platform = linux2
-# autocomplete_test depends on nose when invoked directly.
 deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
   cython==0.25.2
-whitelist_externals=
-  find
-  time
 commands =
-  python --version
-  pip --version
-  # Clean up all previous python generated files.
-  - find apache_beam -type f -name '*.pyc' -delete
-  # Clean up all previous cython generated files.
-  - find apache_beam -type f -name '*.c' -delete
-  - find apache_beam -type f -name '*.so' -delete
-  - find target/build -type f -name '*.c' -delete
-  - find target/build -type f -name '*.so' -delete
-  time pip install -e .[test]
+  {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-  # Clean up all cython generated files. Ignore if deletion fails.
-  - find apache_beam -type f -name '*.c' -delete
-  - find apache_beam -type f -name '*.so' -delete
-  - find target/build -type f -name '*.c' -delete
-  - find target/build -type f -name '*.so' -delete
-passenv = TRAVIS*
+  {toxinidir}/run_tox_cleanup.sh
 
-[testenv:py27gcp]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-whitelist_externals=find
+[testenv:py27-gcp]
+extras = test,gcp
 commands =
-  python --version
-  pip --version
-  pip install -e .[test,gcp]
-  # Clean up all previous python generated files.
-  - find apache_beam -type f -name '*.pyc' -delete
+  {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-passenv = TRAVIS*
+  {toxinidir}/run_tox_cleanup.sh
 
-[testenv:lint_py2]
-deps=
-  nose==1.3.7
+[testenv:py27-lint]
+deps =
   pycodestyle==2.3.1
   pylint==1.7.2
   future==0.16.0
   isort==4.2.15
   flake8==3.5.0
-whitelist_externals=time
 commands =
   python --version
-  pip --version
-  time pip install -e .[test]
   time {toxinidir}/run_pylint.sh
-passenv = TRAVIS*
 
-[testenv:lint_py3]
-deps=
-  nose==1.3.7
+[testenv:py3-lint]
+deps =
   pycodestyle==2.3.1
   pylint==1.7.2
   future==0.16.0
   isort==4.2.15
   flake8==3.5.0
-whitelist_externals=time
+setenv =
+BEAM_EXPERIMENTAL_PY3=1
 commands =
-  time pip install -e .[test]
+  python --version
   time {toxinidir}/run_mini_py3lint.sh
-passenv = TRAVIS*
-
 
 [testenv:docs]
-deps=
-  nose==1.3.7
-  grpcio-tools==1.3.5
+extras = docs
+deps =
   Sphinx==1.6.5
   sphinx_rtd_theme==0.2.4
-whitelist_externals=time
 commands =
   python --version
   pip --version
   time pip install -e .[test,gcp,docs]
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above 

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=83439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83439
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:45
Start Date: 23/Mar/18 01:45
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175934778
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
 
 Review comment:
   Removed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83439)
Time Spent: 9h 50m  (was: 9h 40m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4926: Updating dataflow API protocol buffers

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a3bfe6f47e48a9bd49428e678a49e23c57dccf0c
Merge: 99c1fcb 7a90d88
Author: Chamikara Jayalath 
AuthorDate: Thu Mar 22 18:36:58 2018 -0700

Merge pull request #4926: Updating dataflow API protocol buffers

 .../clients/dataflow/dataflow_v1b3_messages.py | 149 ++---
 1 file changed, 127 insertions(+), 22 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] branch master updated (99c1fcb -> a3bfe6f)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 99c1fcb  Merge pull request #4914: [BEAM-3898] Replace side inputs 
when applying PTransformOverrides
 add 7a90d88  Updating dataflow API protocol buffers.
 new a3bfe6f  Merge pull request #4926: Updating dataflow API protocol 
buffers

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../clients/dataflow/dataflow_v1b3_messages.py | 149 ++---
 1 file changed, 127 insertions(+), 22 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Work logged] (BEAM-3898) PTransformOverride does not replace side input usage of replaced transform

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3898?focusedWorklogId=83431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83431
 ]

ASF GitHub Bot logged work on BEAM-3898:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:35
Start Date: 23/Mar/18 01:35
Worklog Time Spent: 10m 
  Work Description: chamikaramj closed pull request #4914: [BEAM-3898] 
Replace side inputs when applying PTransformOverrides
URL: https://github.com/apache/beam/pull/4914
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 71d97ba5d21..74bd4cb17d0 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -185,6 +185,7 @@ def _replace(self, override):
 output_map = {}
 output_replacements = {}
 input_replacements = {}
+side_input_replacements = {}
 
 class TransformUpdater(PipelineVisitor): # pylint: 
disable=used-before-assignment
   A visitor that replaces the matching PTransforms."""
@@ -312,12 +313,28 @@ def visit_transform(self, transform_node):
 replace_input = True
 break
 
+replace_side_inputs = False
+for side_input in transform_node.side_inputs:
+  if side_input.pvalue in output_map:
+replace_side_inputs = True
+break
+
 if replace_input:
   new_input = [
   input if not input in output_map else output_map[input]
   for input in transform_node.inputs]
   input_replacements[transform_node] = new_input
 
+if replace_side_inputs:
+  new_side_inputs = []
+  for side_input in transform_node.side_inputs:
+if side_input.pvalue in output_map:
+  side_input.pvalue = output_map[side_input.pvalue]
+  new_side_inputs.append(side_input)
+else:
+  new_side_inputs.append(side_input)
+  side_input_replacements[transform_node] = new_side_inputs
+
 self.visit(InputOutputUpdater(self))
 
 for transform in output_replacements:
@@ -326,6 +343,9 @@ def visit_transform(self, transform_node):
 for transform in input_replacements:
   transform.inputs = input_replacements[transform]
 
+for transform in side_input_replacements:
+  transform.side_inputs = side_input_replacements[transform]
+
   def _check_replacement(self, override):
 
 class ReplacementValidator(PipelineVisitor):


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83431)
Time Spent: 40m  (was: 0.5h)

> PTransformOverride does not replace side input usage of replaced transform
> --
>
> Key: BEAM-3898
> URL: https://issues.apache.org/jira/browse/BEAM-3898
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the PTransformOverride mechanism allows specification of a 
> replacement procedure where transform A is replaced with another transform B. 
>  However, the current mechanism does not replace usages where the output of A 
> is being read as a side input.  We should fix this behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3898) PTransformOverride does not replace side input usage of replaced transform

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3898?focusedWorklogId=83430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83430
 ]

ASF GitHub Bot logged work on BEAM-3898:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:35
Start Date: 23/Mar/18 01:35
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4914: [BEAM-3898] 
Replace side inputs when applying PTransformOverrides
URL: https://github.com/apache/beam/pull/4914#issuecomment-375511224
 
 
   LGTM. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83430)
Time Spent: 0.5h  (was: 20m)

> PTransformOverride does not replace side input usage of replaced transform
> --
>
> Key: BEAM-3898
> URL: https://issues.apache.org/jira/browse/BEAM-3898
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, the PTransformOverride mechanism allows specification of a 
> replacement procedure where transform A is replaced with another transform B. 
>  However, the current mechanism does not replace usages where the output of A 
> is being read as a side input.  We should fix this behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4914: [BEAM-3898] Replace side inputs when applying PTransformOverrides

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 99c1fcbdace73c07d83ced51053cfd8b2db31aad
Merge: 0221de7 a853ff7
Author: Chamikara Jayalath 
AuthorDate: Thu Mar 22 18:35:34 2018 -0700

Merge pull request #4914: [BEAM-3898] Replace side inputs when applying 
PTransformOverrides

 sdks/python/apache_beam/pipeline.py | 20 
 1 file changed, 20 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] branch master updated (0221de7 -> 99c1fcb)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0221de7  This closes #4936
 add a853ff7  Replace side inputs when applying PTransformOverrides
 new 99c1fcb  Merge pull request #4914: [BEAM-3898] Replace side inputs 
when applying PTransformOverrides

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline.py | 20 
 1 file changed, 20 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=83426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83426
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:25
Start Date: 23/Mar/18 01:25
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-375509677
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83426)
Time Spent: 20m  (was: 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=83425=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83425
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 23/Mar/18 01:25
Start Date: 23/Mar/18 01:25
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #4943: [BEAM-3906] 
Automate Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943
 
 
   DESCRIPTION HERE
   
   In the 2.4.0 release, we have added Wheel files for Python SDK. Changes in 
this pull request aim to automate the Quickstarts and MobileGaming examples to 
make an easy release validation process.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83425)
Time Spent: 10m
Remaining Estimate: 0h

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1499

[Apache Jenkins Server]

See 


Changes:

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[boyuanz] Add cython annotation to make DistributionAccumulator faster

[ankurgoenka] Send worker_id in all channels to runner Adding

[ehudm] Reduce precommit test timeouts for Java and Go.

--
[...truncated 66.09 KB...]
2018-03-23 01:15:18,208 9b0139e7 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-23 01:15:41,965 9b0139e7 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-23 01:15:45,438 9b0139e7 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r1f7a36fc4c2e77b1_0162506cfbc9_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r1f7a36fc4c2e77b1_0162506cfbc9_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r1f7a36fc4c2e77b1_0162506cfbc9_1 ... (0s) Current status: DONE   
2018-03-23 01:15:45,438 9b0139e7 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-23 01:16:14,113 9b0139e7 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-23 01:16:17,698 9b0139e7 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r325c6cf4faa6c3a9_0162506d7953_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r325c6cf4faa6c3a9_0162506d7953_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r325c6cf4faa6c3a9_0162506d7953_1 ... (0s) Current status: DONE   
2018-03-23 01:16:17,699 9b0139e7 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-23 01:16:45,027 9b0139e7 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-23 01:16:48,406 9b0139e7 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3e1a246953a2_0162506df25c_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r3e1a246953a2_0162506df25c_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r3e1a246953a2_0162506df25c_1 ... (0s) Current status: DONE   
2018-03-23 01:16:48,407 9b0139e7 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-23 01:17:05,779 9b0139e7 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-23 01:17:09,048 9b0139e7 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r28c8b6562911bc99_0162506e4381_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r28c8b6562911bc99_0162506e4381_1 
... (0s) Current status: RUNNING  

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3753

[Apache Jenkins Server]

See 

[beam] branch master updated: More graceful fallback when grpc is not present.

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 9be0148  More graceful fallback when grpc is not present.
 new 0221de7  This closes #4936
9be0148 is described below

commit 9be0148970158f253d202b7eddc221270d6c12a5
Author: Robert Bradshaw 
AuthorDate: Thu Mar 22 01:38:01 2018 -0700

More graceful fallback when grpc is not present.
---
 sdks/python/apache_beam/runners/direct/direct_runner.py | 14 ++
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/sdks/python/apache_beam/runners/direct/direct_runner.py 
b/sdks/python/apache_beam/runners/direct/direct_runner.py
index 531423a..062509f 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner.py
@@ -83,8 +83,10 @@ class SwitchingDirectRunner(PipelineRunner):
 class _FnApiRunnerSupportVisitor(PipelineVisitor):
   """Visitor determining if a Pipeline can be run on the FnApiRunner."""
 
-  def __init__(self):
+  def accept(self, pipeline):
 self.supported_by_fnapi_runner = True
+pipeline.visit(self)
+return self.supported_by_fnapi_runner
 
   def visit_transform(self, applied_ptransform):
 transform = applied_ptransform.transform
@@ -115,9 +117,13 @@ class SwitchingDirectRunner(PipelineRunner):
 
 # Check whether all transforms used in the pipeline are supported by the
 # FnApiRunner.
-visitor = _FnApiRunnerSupportVisitor()
-pipeline.visit(visitor)
-if not visitor.supported_by_fnapi_runner:
+use_fnapi_runner = _FnApiRunnerSupportVisitor().accept(pipeline)
+
+# Also ensure grpc is available.
+try:
+  # pylint: disable=unused-variable
+  import grpc
+except ImportError:
   use_fnapi_runner = False
 
 if use_fnapi_runner:

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #300

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1055

[Apache Jenkins Server]

See 


Changes:

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[boyuanz] Add cython annotation to make DistributionAccumulator faster

[ankurgoenka] Send worker_id in all channels to runner Adding

[ehudm] Reduce precommit test timeouts for Java and Go.

--
[...truncated 61.60 KB...]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 212 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.056s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.186s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [ 14.580 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  9.042 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.184 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [ 25.766 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  9.321 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [ 12.061 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.577 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 56.395 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam 

[jira] [Commented] (BEAM-2767) BigQueryIO result different for REPEATED field between DirectRunner and DataflowRunner

[Maximilian Roos (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410645#comment-16410645
 ] 

Maximilian Roos commented on BEAM-2767:
---

In the DataFlow Runner, a BQ Read on a nested column creates a collection with 
items like:

 

{{dict(name='John', orders=[123, 456]}}}

 

But in the DirectRunner, the same item is unnested like:

 

{{dict(name='John', orders=123)}}
{{# new item in collection}}
{{dict(name='John', orders=456)}}

 

This obviously breaks anything that previously worked!

 

> BigQueryIO result different for REPEATED field between DirectRunner and 
> DataflowRunner
> --
>
> Key: BEAM-2767
> URL: https://issues.apache.org/jira/browse/BEAM-2767
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp, runner-dataflow, runner-direct
>Affects Versions: 2.0.0
>Reporter: Andre
>Assignee: Udi Meiri
>Priority: Minor
>
> When running a query against BigQueryIO with a REPEATED RECORD field the 
> behavior is different between DirectRunner and DataflowRunner. The field 
> containing the repeated record has to be cast to access the records. 
> Apparently the following implementations work for each runner but I would 
> expect them to be the same as my pipeline otherwise only runs on one.
> DirectRunner:
> {code:java}
> ArrayList orderLines = (ArrayList) 
> c.element().get("RepeatedField");
> {code}
> DataflowRunner:
> {code:java}
> ImmutableList orderLines = (ImmutableList) 
> c.element().get("RepeatedField");
> {code}
>   
> For example when using the ImmutableList implementation on DirectRunner the 
> following exception is thrown:
> {code:java}
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to 
> com.google.common.collect.ImmutableList
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #5301

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83413
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:52
Start Date: 23/Mar/18 00:52
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #4930: [BEAM-3861] Complete 
streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#issuecomment-375504394
 
 
   LGTM. Please squash, I can merge after tests pass.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83413)
Time Spent: 7.5h  (was: 7h 20m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2767) BigQueryIO result different for REPEATED field between DirectRunner and DataflowRunner

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410634#comment-16410634
 ] 

Chamikara Jayalath commented on BEAM-2767:
--

Can you explain the difference between repeated values for Python ?

 

 

> BigQueryIO result different for REPEATED field between DirectRunner and 
> DataflowRunner
> --
>
> Key: BEAM-2767
> URL: https://issues.apache.org/jira/browse/BEAM-2767
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp, runner-dataflow, runner-direct
>Affects Versions: 2.0.0
>Reporter: Andre
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> When running a query against BigQueryIO with a REPEATED RECORD field the 
> behavior is different between DirectRunner and DataflowRunner. The field 
> containing the repeated record has to be cast to access the records. 
> Apparently the following implementations work for each runner but I would 
> expect them to be the same as my pipeline otherwise only runs on one.
> DirectRunner:
> {code:java}
> ArrayList orderLines = (ArrayList) 
> c.element().get("RepeatedField");
> {code}
> DataflowRunner:
> {code:java}
> ImmutableList orderLines = (ImmutableList) 
> c.element().get("RepeatedField");
> {code}
>   
> For example when using the ImmutableList implementation on DirectRunner the 
> following exception is thrown:
> {code:java}
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to 
> com.google.common.collect.ImmutableList
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2767) BigQueryIO result different for REPEATED field between DirectRunner and DataflowRunner

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-2767:


Assignee: Udi Meiri  (was: Chamikara Jayalath)

> BigQueryIO result different for REPEATED field between DirectRunner and 
> DataflowRunner
> --
>
> Key: BEAM-2767
> URL: https://issues.apache.org/jira/browse/BEAM-2767
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp, runner-dataflow, runner-direct
>Affects Versions: 2.0.0
>Reporter: Andre
>Assignee: Udi Meiri
>Priority: Minor
>
> When running a query against BigQueryIO with a REPEATED RECORD field the 
> behavior is different between DirectRunner and DataflowRunner. The field 
> containing the repeated record has to be cast to access the records. 
> Apparently the following implementations work for each runner but I would 
> expect them to be the same as my pipeline otherwise only runs on one.
> DirectRunner:
> {code:java}
> ArrayList orderLines = (ArrayList) 
> c.element().get("RepeatedField");
> {code}
> DataflowRunner:
> {code:java}
> ImmutableList orderLines = (ImmutableList) 
> c.element().get("RepeatedField");
> {code}
>   
> For example when using the ImmutableList implementation on DirectRunner the 
> following exception is thrown:
> {code:java}
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to 
> com.google.common.collect.ImmutableList
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1165

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Send worker_id in all channels to runner Adding

--
[...truncated 770.43 KB...]
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s15", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": 

[beam] branch master updated (f477db1 -> c509d17)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f477db1  Merge pull request #4891 from ehrencrona/2264
 add feb0c87  Add cython annotation to make DistributionAccumulator faster
 new c509d17  Merge pull request #4928 from boyuanzz/distribution

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/transforms/cy_combiners.pxd | 10 ++
 1 file changed, 10 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4928 from boyuanzz/distribution

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c509d174b4340676ccfe67c9fe4a69ea16d70437
Merge: f477db1 feb0c87
Author: Ahmet Altay 
AuthorDate: Thu Mar 22 17:34:58 2018 -0700

Merge pull request #4928 from boyuanzz/distribution

Add cython annotations to make DistributionAccumulator faster

 sdks/python/apache_beam/transforms/cy_combiners.pxd | 10 ++
 1 file changed, 10 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-2264) Re-use credential instead of generating a new one one each GCS call

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2264?focusedWorklogId=83402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83402
 ]

ASF GitHub Bot logged work on BEAM-2264:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:29
Start Date: 23/Mar/18 00:29
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #4891: [BEAM-2264] 
Credentials were not being reused between GCS calls
URL: https://github.com/apache/beam/pull/4891#issuecomment-375501005
 
 
   Thank you @ehrencrona 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83402)
Time Spent: 40m  (was: 0.5h)

> Re-use credential instead of generating a new one one each GCS call
> ---
>
> Key: BEAM-2264
> URL: https://issues.apache.org/jira/browse/BEAM-2264
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ahmet Altay
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We should cache the credential used within a Pipeline and re-use it instead 
> of generating a new one on each GCS call. When executing (against 2.0.0 RC2):
> {code}
> python -m apache_beam.examples.wordcount --input 
> "gs://dataflow-samples/shakespeare/*" --output local_counts
> {code}
> Note that we seemingly generate a new access token each time instead of when 
> a refresh is required.
> {code}
>   super(GcsIO, cls).__new__(cls, storage_client))
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 1 files. Estimation 
> took 0.286200046539 seconds
> INFO:root:Running pipeline with DirectRunner.
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 43 files. Estimation 
> took 0.205624818802 seconds
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> ... many more times ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2264) Re-use credential instead of generating a new one one each GCS call

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2264?focusedWorklogId=83403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83403
 ]

ASF GitHub Bot logged work on BEAM-2264:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:29
Start Date: 23/Mar/18 00:29
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #4891: [BEAM-2264] 
Credentials were not being reused between GCS calls
URL: https://github.com/apache/beam/pull/4891
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py 
b/sdks/python/apache_beam/io/gcp/gcsio.py
index f687686fd64..c7986cdb672 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio.py
@@ -146,6 +146,8 @@ class GcsIOError(IOError, retry.PermanentException):
 class GcsIO(object):
   """Google Cloud Storage I/O client."""
 
+  local_state = threading.local()
+
   def __new__(cls, storage_client=None):
 if storage_client:
   # This path is only used for testing.
@@ -155,7 +157,7 @@ def __new__(cls, storage_client=None):
   # creating more than one storage client for each thread, since each
   # initialization requires the relatively expensive step of initializing
   # credentaials.
-  local_state = threading.local()
+  local_state = GcsIO.local_state
   if getattr(local_state, 'gcsio_instance', None) is None:
 credentials = auth.get_service_credentials()
 storage_client = storage.StorageV1(


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83403)
Time Spent: 50m  (was: 40m)

> Re-use credential instead of generating a new one one each GCS call
> ---
>
> Key: BEAM-2264
> URL: https://issues.apache.org/jira/browse/BEAM-2264
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ahmet Altay
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We should cache the credential used within a Pipeline and re-use it instead 
> of generating a new one on each GCS call. When executing (against 2.0.0 RC2):
> {code}
> python -m apache_beam.examples.wordcount --input 
> "gs://dataflow-samples/shakespeare/*" --output local_counts
> {code}
> Note that we seemingly generate a new access token each time instead of when 
> a refresh is required.
> {code}
>   super(GcsIO, cls).__new__(cls, storage_client))
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 1 files. Estimation 
> took 0.286200046539 seconds
> INFO:root:Running pipeline with DirectRunner.
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 43 files. Estimation 
> took 0.205624818802 seconds
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> ... many more times ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (5b7f095 -> f477db1)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5b7f095  Reduce precommit test timeouts for Java and Go.
 add 4045e32  [BEAM-2264] Credentials were not being reused between GCS 
calls
 new f477db1  Merge pull request #4891 from ehrencrona/2264

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/gcp/gcsio.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4891 from ehrencrona/2264

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f477db12e7833a31b87a88460f699c9ca428a56c
Merge: 5b7f095 4045e32
Author: Ahmet Altay 
AuthorDate: Thu Mar 22 17:29:21 2018 -0700

Merge pull request #4891 from ehrencrona/2264

[BEAM-2264] Credentials were not being reused between GCS calls

 sdks/python/apache_beam/io/gcp/gcsio.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-2264) Re-use credential instead of generating a new one one each GCS call

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2264?focusedWorklogId=83401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83401
 ]

ASF GitHub Bot logged work on BEAM-2264:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:23
Start Date: 23/Mar/18 00:23
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #4891: [BEAM-2264] 
Credentials were not being reused between GCS calls
URL: https://github.com/apache/beam/pull/4891#issuecomment-375499986
 
 
   Thank you!  This is a great change.  LGTM.
   
   R: @aaltay for merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83401)
Time Spent: 0.5h  (was: 20m)

> Re-use credential instead of generating a new one one each GCS call
> ---
>
> Key: BEAM-2264
> URL: https://issues.apache.org/jira/browse/BEAM-2264
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ahmet Altay
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We should cache the credential used within a Pipeline and re-use it instead 
> of generating a new one on each GCS call. When executing (against 2.0.0 RC2):
> {code}
> python -m apache_beam.examples.wordcount --input 
> "gs://dataflow-samples/shakespeare/*" --output local_counts
> {code}
> Note that we seemingly generate a new access token each time instead of when 
> a refresh is required.
> {code}
>   super(GcsIO, cls).__new__(cls, storage_client))
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 1 files. Estimation 
> took 0.286200046539 seconds
> INFO:root:Running pipeline with DirectRunner.
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 43 files. Estimation 
> took 0.205624818802 seconds
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> ... many more times ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2264) Re-use credential instead of generating a new one one each GCS call

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2264?focusedWorklogId=83400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83400
 ]

ASF GitHub Bot logged work on BEAM-2264:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:20
Start Date: 23/Mar/18 00:20
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #4891: [BEAM-2264] 
Credentials were not being reused between GCS calls
URL: https://github.com/apache/beam/pull/4891#issuecomment-375499544
 
 
   R: @charlesccychen


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83400)
Time Spent: 20m  (was: 10m)

> Re-use credential instead of generating a new one one each GCS call
> ---
>
> Key: BEAM-2264
> URL: https://issues.apache.org/jira/browse/BEAM-2264
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ahmet Altay
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We should cache the credential used within a Pipeline and re-use it instead 
> of generating a new one on each GCS call. When executing (against 2.0.0 RC2):
> {code}
> python -m apache_beam.examples.wordcount --input 
> "gs://dataflow-samples/shakespeare/*" --output local_counts
> {code}
> Note that we seemingly generate a new access token each time instead of when 
> a refresh is required.
> {code}
>   super(GcsIO, cls).__new__(cls, storage_client))
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 1 files. Estimation 
> took 0.286200046539 seconds
> INFO:root:Running pipeline with DirectRunner.
> INFO:root:Starting the size estimation of the input
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Finished the size estimation of the input at 43 files. Estimation 
> took 0.205624818802 seconds
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> ... many more times ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Reduce precommit test timeouts for Java and Go.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5b7f095d4b4f7896efc47b3d9d9c690a3d7356d6
Merge: 6740ead d39229d
Author: Lukasz Cwik 
AuthorDate: Thu Mar 22 17:20:50 2018 -0700

Reduce precommit test timeouts for Java and Go.

 .test-infra/jenkins/job_beam_PreCommit_Go_GradleBuild.groovy| 2 +-
 .test-infra/jenkins/job_beam_PreCommit_Java_GradleBuild.groovy  | 2 +-
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (6740ead -> 5b7f095)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6740ead  Merge pull request #4587 from angoenka/multiprocess_new
 add d39229d  Reduce precommit test timeouts for Java and Go.
 new 5b7f095  Reduce precommit test timeouts for Java and Go.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/job_beam_PreCommit_Go_GradleBuild.groovy| 2 +-
 .test-infra/jenkins/job_beam_PreCommit_Java_GradleBuild.groovy  | 2 +-
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83396
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:05
Start Date: 23/Mar/18 00:05
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #4930: [BEAM-3861] 
Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#issuecomment-375497207
 
 
   @aaltay PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83396)
Time Spent: 7h 20m  (was: 7h 10m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4483

[Apache Jenkins Server]

See 


--
[...truncated 1.12 MB...]
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey2.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s20"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey2"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-22_16_57_54-5194148899779802903]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-22_16_57_54-5194148899779802903?project=apache-beam-testing
root: INFO: Job 2018-03-22_16_57_54-5194148899779802903 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-22T23:57:54.598Z: JOB_MESSAGE_WARNING: Job 
2018-03-22_16_57_54-5194148899779802903 might autoscale up to 250 workers.
root: INFO: 2018-03-22T23:57:54.606Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-22_16_57_54-5194148899779802903. The number of workers 
will be between 1 and 250.
root: INFO: 2018-03-22T23:57:54.618Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-22_16_57_54-5194148899779802903.
root: INFO: 2018-03-22T23:57:57.360Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-22T23:57:57.520Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-22T23:57:57.771Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-22T23:57:57.787Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-22T23:57:57.798Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step group: GroupByKey not followed by a combiner.
root: INFO: 2018-03-22T23:57:57.811Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-22T23:57:57.825Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-22T23:57:57.842Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-22T23:57:57.866Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-22T23:57:57.877Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/PreFinalize/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-22T23:57:57.889Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-22T23:57:57.900Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/PreFinalize/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-22T23:57:57.910Z: JOB_MESSAGE_DETAILED: Fusing consumer 
write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1 into 
write/Write/WriteImpl/Extract
root: INFO: 2018-03-22T23:57:57.920Z: JOB_MESSAGE_DETAILED: Fusing consumer 
pair_with_one into split
root: INFO: 2018-03-22T23:57:57.934Z: JOB_MESSAGE_DETAILED: Fusing consumer 
group/Reify into 

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83395
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:03
Start Date: 23/Mar/18 00:03
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #4930: [BEAM-3861] 
Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#issuecomment-375496804
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83395)
Time Spent: 7h 10m  (was: 7h)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83393
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:01
Start Date: 23/Mar/18 00:01
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#4930: [BEAM-3861] Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#discussion_r176607403
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
 ##
 @@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""PubSub verifier used for end-to-end test."""
+
+import logging
+import time
+from collections import Counter
+
+from hamcrest.core.base_matcher import BaseMatcher
+
+__all__ = ['PubSubMessageMatcher']
+
+
+# Protect against environments where pubsub library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+  from google.cloud import pubsub
+except ImportError:
+  pubsub = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+DEFAULT_TIMEOUT = 5 * 60
+MAX_MESSAGES_IN_ONE_PULL = 50
+
+
+class PubSubMessageMatcher(BaseMatcher):
+  """Matcher that verifies messages from given subscription.
+
+  This matcher can block the test and keep pulling messages from given
+  subscription until all expected messages are shown or timeout.
+  """
+
+  def __init__(self, project, sub_name, expected_msg, timeout=DEFAULT_TIMEOUT):
+"""Initialize PubSubMessageMatcher object.
+
+Args:
+  project: A name string of project.
+  sub_name: A name string of subscription which is attached to output.
+  expected_msg: A string list that contains expected message data pulled
+from the subscription.
+  timeout: Timeout in seconds to wait for all expected messages appears.
+"""
+if pubsub is None:
+  raise ImportError(
+  'PubSub dependencies are not installed.')
+if not project:
+  raise ValueError('Invalid project %s.' % project)
+if not sub_name:
+  raise ValueError('Invalid subscription %s.' % sub_name)
+if not expected_msg or not isinstance(expected_msg, list):
+  raise ValueError('Invalid expected messages %s.' % expected_msg)
+
+self.project = project
+self.sub_name = sub_name
+self.expected_msg = expected_msg
+self.timeout = timeout
+self.messages = None
+
+  def _matches(self, _):
+if not self.messages:
+  subscription = (pubsub
+  .Client(project=self.project)
+  .subscription(self.sub_name))
+  self.messages = self._wait_for_messages(subscription,
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83393)
Time Spent: 6h 50m  (was: 6h 40m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83394
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:01
Start Date: 23/Mar/18 00:01
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#4930: [BEAM-3861] Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#discussion_r176607424
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
 ##
 @@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""PubSub verifier used for end-to-end test."""
+
+import logging
+import time
+from collections import Counter
+
+from hamcrest.core.base_matcher import BaseMatcher
+
+__all__ = ['PubSubMessageMatcher']
+
+
+# Protect against environments where pubsub library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+  from google.cloud import pubsub
+except ImportError:
+  pubsub = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+DEFAULT_TIMEOUT = 5 * 60
+MAX_MESSAGES_IN_ONE_PULL = 50
+
+
+class PubSubMessageMatcher(BaseMatcher):
+  """Matcher that verifies messages from given subscription.
+
+  This matcher can block the test and keep pulling messages from given
+  subscription until all expected messages are shown or timeout.
+  """
+
+  def __init__(self, project, sub_name, expected_msg, timeout=DEFAULT_TIMEOUT):
+"""Initialize PubSubMessageMatcher object.
+
+Args:
+  project: A name string of project.
+  sub_name: A name string of subscription which is attached to output.
+  expected_msg: A string list that contains expected message data pulled
+from the subscription.
+  timeout: Timeout in seconds to wait for all expected messages appears.
+"""
+if pubsub is None:
+  raise ImportError(
+  'PubSub dependencies are not installed.')
+if not project:
+  raise ValueError('Invalid project %s.' % project)
+if not sub_name:
+  raise ValueError('Invalid subscription %s.' % sub_name)
+if not expected_msg or not isinstance(expected_msg, list):
+  raise ValueError('Invalid expected messages %s.' % expected_msg)
+
+self.project = project
+self.sub_name = sub_name
+self.expected_msg = expected_msg
+self.timeout = timeout
+self.messages = None
+
+  def _matches(self, _):
+if not self.messages:
+  subscription = (pubsub
+  .Client(project=self.project)
+  .subscription(self.sub_name))
+  self.messages = self._wait_for_messages(subscription,
+  len(self.expected_msg),
+  self.timeout)
+return Counter(self.messages) == Counter(self.expected_msg)
+
+  def _wait_for_messages(self, subscription, expected_num, timeout):
+"""Wait for messages from given subscription."""
+logging.debug('Start pulling messages from %s', subscription.full_name)
+total_messages = []
+start_time = time.time()
+while time.time() - start_time <= timeout:
+  pulled = subscription.pull(max_messages=MAX_MESSAGES_IN_ONE_PULL)
+  for ack_id, message in pulled:
+total_messages.append(message.data)
+subscription.acknowledge([ack_id])
+  if len(total_messages) >= expected_num:
+return total_messages
+  time.sleep(1)
+
+raise RuntimeError('Timeout after %d sec. Received %d messages from %s.' %
+   (timeout, len(total_messages), subscription.full_name))
+
+  def describe_to(self, description):
+description.append_text(
+'Expected %d messages.' % len(self.expected_msg))
+
+  def describe_mismatch(self, _, mismatch_description):
+diff = set(self.expected_msg) - set(self.messages)
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use 

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83392
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 23/Mar/18 00:01
Start Date: 23/Mar/18 00:01
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#4930: [BEAM-3861] Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#discussion_r176607385
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
 ##
 @@ -47,24 +50,32 @@ class StreamingWordCountIT(unittest.TestCase):
 
   def setUp(self):
 self.test_pipeline = TestPipeline(is_integration_test=True)
+self.project = self.test_pipeline.get_option('project')
+self.identifier = self._generate_identifier()
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83392)
Time Spent: 6h 40m  (was: 6.5h)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83391
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:56
Start Date: 22/Mar/18 23:56
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4930: 
[BEAM-3861] Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#discussion_r176606739
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
 ##
 @@ -44,34 +44,41 @@ def run_pipeline(self, pipeline):
 
 self.result = super(TestDataflowRunner, self).run_pipeline(pipeline)
 if self.result.has_job:
-  project = pipeline._options.view_as(GoogleCloudOptions).project
-  region_id = pipeline._options.view_as(GoogleCloudOptions).region
-  job_id = self.result.job_id()
   # TODO(markflyhigh)(BEAM-1890): Use print since Nose dosen't show logs
   # in some cases.
-  print (
-  'Found: https://console.cloud.google.com/dataflow/jobsDetail'
-  '/locations/%s/jobs/%s?project=%s' % (region_id, job_id, project))
+  print('Found: %s.' % self.build_console_url(pipeline.options))
 
 if not options.view_as(StandardOptions).streaming:
   self.result.wait_until_finish()
 else:
-  # TODO: Ideally, we want to wait until workers start successfully.
-  self.wait_until_running()
+  self.wait_until_in_state(PipelineState.RUNNING)
 
 if on_success_matcher:
   from hamcrest import assert_that as hc_assert_that
   hc_assert_that(self.result, pickler.loads(on_success_matcher))
 
+if options.view_as(StandardOptions).streaming:
+  self.result.cancel()
 
 Review comment:
   I missed the success matchers above. This is fine as it is.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83391)
Time Spent: 6.5h  (was: 6h 20m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3910) Support floating point values in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3910?focusedWorklogId=83389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83389
 ]

ASF GitHub Bot logged work on BEAM-3910:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:55
Start Date: 22/Mar/18 23:55
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4941: 
BEAM-3910: Add float support for the Go SDK.
URL: https://github.com/apache/beam/pull/4941#discussion_r176606245
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/coderx/float.go
 ##
 @@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package coderx
+
+import (
+   "fmt"
+   "math"
+   "math/bits"
+   "reflect"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
+)
+
+func init() {
+   runtime.RegisterFunction(encFloat)
+   runtime.RegisterFunction(decFloat)
+}
+
+func encFloat(v typex.T) []byte {
+   var val float64
+   switch n := v.(type) {
+   case float32:
+   val = float64(n)
+   case float64:
+   val = n
+   default:
+   panic(fmt.Sprintf("received unknown value type: want a float, 
got %T", n))
+   }
+
+   return encVarUintZ(bits.ReverseBytes64(math.Float64bits(val)))
+}
+
+func decFloat(t reflect.Type, data []byte) (typex.T, error) {
+   uval, err := decVarUintZ(reflectx.Uint64, data)
+   if err != nil {
+   return nil, fmt.Errorf("invalid float encoding for: %v", data)
+   }
+
+   n := math.Float64frombits(bits.ReverseBytes64(uval.(uint64)))
+   switch t.Kind() {
+   case reflect.Float64:
+   return n, nil
+   case reflect.Float32:
+   return float32(n), nil
+   default:
+   panic(fmt.Sprintf("unreachable statement: expected a float, got 
%v", t))
+   }
+}
+
+// NewFloat returns a coder for the given float type. It uses the same
+// encoding scheme as the gob package.
+func NewFloat(t reflect.Type) (*coder.CustomCoder, error) {
 
 Review comment:
   Given there are only 2 types, perhaps do them separately to avoid the 
reflect.Type everywhere? I.e.,
   
   NewFloat32
   NewFloat64
   encFloat32
   etc
   
   Presumably faster, too, but not sure if it matters.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83389)
Time Spent: 0.5h  (was: 20m)
Remaining Estimate: 23.5h  (was: 23h 40m)

> Support floating point values in Go SDK
> ---
>
> Key: BEAM-3910
> URL: https://issues.apache.org/jira/browse/BEAM-3910
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 0.5h
>  Remaining Estimate: 23.5h
>
> The Go SDK supports all the integer types of the language, but does not 
> support floats.
> My plan for coding is to use the same technique the gob package uses, which 
> results in a compact encoding for simple values.
> [https://golang.org/src/encoding/gob/encode.go?#L210|https://golang.org/src/encoding/gob/encode.go#L210]
>  with rationale explained in 
> https://golang.org/pkg/encoding/gob/#hdr-Encoding_Details
> The resulting uint is then encoded using the existing coders in coderx.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3910) Support floating point values in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3910?focusedWorklogId=83390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83390
 ]

ASF GitHub Bot logged work on BEAM-3910:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:55
Start Date: 22/Mar/18 23:55
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4941: 
BEAM-3910: Add float support for the Go SDK.
URL: https://github.com/apache/beam/pull/4941#discussion_r176606474
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto
 ##
 @@ -46,6 +46,8 @@ message Type {
 UINT32 = 10;
 UINT64 = 11;
 STRING = 12;
+FLOAT32 = 13;
 
 Review comment:
   Don't we need changes to EncodeType/DecodeType as well that use them?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83390)
Time Spent: 0.5h  (was: 20m)
Remaining Estimate: 23.5h  (was: 23h 40m)

> Support floating point values in Go SDK
> ---
>
> Key: BEAM-3910
> URL: https://issues.apache.org/jira/browse/BEAM-3910
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 0.5h
>  Remaining Estimate: 23.5h
>
> The Go SDK supports all the integer types of the language, but does not 
> support floats.
> My plan for coding is to use the same technique the gob package uses, which 
> results in a compact encoding for simple values.
> [https://golang.org/src/encoding/gob/encode.go?#L210|https://golang.org/src/encoding/gob/encode.go#L210]
>  with rationale explained in 
> https://golang.org/pkg/encoding/gob/#hdr-Encoding_Details
> The resulting uint is then encoded using the existing coders in coderx.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3418) Python Fnapi - Support Multiple SDK workers on a single VM

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3418?focusedWorklogId=83388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83388
 ]

ASF GitHub Bot logged work on BEAM-3418:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:45
Start Date: 22/Mar/18 23:45
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #4587: [BEAM-3418] Send 
worker_id in all grpc channels to runner harness
URL: https://github.com/apache/beam/pull/4587
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index e503da9eb5b..9db1cab96f4 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -169,10 +169,17 @@ def __init__(self, packages, options, 
environment_version, pipeline_url):
 if job_type.startswith('FNAPI_'):
   runner_harness_override = (
   dependency.get_runner_harness_container_image())
+  self.debug_options.experiments = self.debug_options.experiments or []
   if runner_harness_override:
-self.debug_options.experiments = self.debug_options.experiments or []
 self.debug_options.experiments.append(
 'runner_harness_container_image=' + runner_harness_override)
+  # Add use_multiple_sdk_containers flag if its not already present. Do not
+  # add the flag if 'no_use_multiple_sdk_containers' is present.
+  # TODO: Cleanup use_multiple_sdk_containers once we deprecate Python SDK
+  # till version 2.4.
+  if ('use_multiple_sdk_containers' not in self.proto.experiments and
+  'no_use_multiple_sdk_containers' not in self.proto.experiments):
+self.debug_options.experiments.append('use_multiple_sdk_containers')
 # Experiments
 if self.debug_options.experiments:
   for experiment in self.debug_options.experiments:
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py 
b/sdks/python/apache_beam/runners/worker/data_plane.py
index f554646c659..7c79c4cc7a7 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -34,6 +34,7 @@
 from apache_beam.coders import coder_impl
 from apache_beam.portability.api import beam_fn_api_pb2
 from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.worker_id_interceptor import 
WorkerIdInterceptor
 
 # This module is experimental. No backwards-compatibility guarantees.
 
@@ -311,6 +312,9 @@ def create_data_channel(self, remote_grpc_port):
   # controlled in a layer above.
   options=[("grpc.max_receive_message_length", -1),
("grpc.max_send_message_length", -1)])
+  # Add workerId to the grpc channel
+  grpc_channel = grpc.intercept_channel(grpc_channel,
+WorkerIdInterceptor())
   self._data_channel_cache[url] = GrpcClientDataChannel(
   beam_fn_api_pb2_grpc.BeamFnDataStub(grpc_channel))
 return self._data_channel_cache[url]
diff --git a/sdks/python/apache_beam/runners/worker/log_handler.py 
b/sdks/python/apache_beam/runners/worker/log_handler.py
index 6d8a1d92671..152659e0a3f 100644
--- a/sdks/python/apache_beam/runners/worker/log_handler.py
+++ b/sdks/python/apache_beam/runners/worker/log_handler.py
@@ -25,6 +25,7 @@
 
 from apache_beam.portability.api import beam_fn_api_pb2
 from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.worker_id_interceptor import 
WorkerIdInterceptor
 
 # This module is experimental. No backwards-compatibility guarantees.
 
@@ -48,7 +49,9 @@ class FnApiLogRecordHandler(logging.Handler):
 
   def __init__(self, log_service_descriptor):
 super(FnApiLogRecordHandler, self).__init__()
-self._log_channel = grpc.insecure_channel(log_service_descriptor.url)
+self._log_channel = grpc.intercept_channel(
+grpc.insecure_channel(log_service_descriptor.url),
+WorkerIdInterceptor())
 self._logging_stub = beam_fn_api_pb2_grpc.BeamFnLoggingStub(
 self._log_channel)
 self._log_entry_queue = queue.Queue()
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py 
b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 1988490013c..c77659b3479 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -34,6 +34,7 @@
 from apache_beam.portability.api import beam_fn_api_pb2_grpc
 from 

[beam] branch master updated (501dc4c -> 6740ead)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 501dc4c  [BEAM-3851] Option to preserve element timestamp while 
publishing to Kafka. (#4868)
 add 1f52bb8  Send worker_id in all channels to runner Adding 
use_multiple_sdk_containers flag for FNAPI pipelines.
 new 6740ead  Merge pull request #4587 from angoenka/multiprocess_new

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/dataflow/internal/apiclient.py |  9 ++-
 .../apache_beam/runners/worker/data_plane.py   |  4 ++
 .../apache_beam/runners/worker/log_handler.py  |  5 +-
 .../apache_beam/runners/worker/sdk_worker.py   |  4 +-
 .../runners/worker/worker_id_interceptor.py| 59 +
 .../runners/worker/worker_id_interceptor_test.py   | 75 ++
 sdks/python/container/boot.go  |  1 +
 sdks/python/setup.py   |  2 +-
 8 files changed, 155 insertions(+), 4 deletions(-)
 create mode 100644 
sdks/python/apache_beam/runners/worker/worker_id_interceptor.py
 create mode 100644 
sdks/python/apache_beam/runners/worker/worker_id_interceptor_test.py

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4587 from angoenka/multiprocess_new

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6740ead918da15601b7134b3ecf47272e41f74d3
Merge: 501dc4c 1f52bb8
Author: Ahmet Altay 
AuthorDate: Thu Mar 22 16:45:50 2018 -0700

Merge pull request #4587 from angoenka/multiprocess_new

[BEAM-3418] Send worker_id in all grpc channels to runner harness

 .../runners/dataflow/internal/apiclient.py |  9 ++-
 .../apache_beam/runners/worker/data_plane.py   |  4 ++
 .../apache_beam/runners/worker/log_handler.py  |  5 +-
 .../apache_beam/runners/worker/sdk_worker.py   |  4 +-
 .../runners/worker/worker_id_interceptor.py| 59 +
 .../runners/worker/worker_id_interceptor_test.py   | 75 ++
 sdks/python/container/boot.go  |  1 +
 sdks/python/setup.py   |  2 +-
 8 files changed, 155 insertions(+), 4 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Updated] (BEAM-3707) Implement portable non-lifted Combines in Java SDK

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-3707:
--
Description: 
Get a portable version of non-lifted Combines working in Java.

Non-lifted combines should be done as an opaque ParDo after a GBK within a 
Combine composite transform.

This task also requires implementing the anonymous case, where the ParDo and 
GBK are not in a composite and the runner cannot identify the ParDo as a 
Combine at all. For now this case only occurs when the Combine has side inputs.

  was:
Get a portable version of non-lifted Combines working in Java.

Non-lifted combines should be done as an opaque ParDo after a GBK, created 
SDK-side. Runners won't be able to identify it as a Combine, but it should work 
without any changes to any runners.


> Implement portable non-lifted Combines in Java SDK
> --
>
> Key: BEAM-3707
> URL: https://issues.apache.org/jira/browse/BEAM-3707
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: features, portability
>
> Get a portable version of non-lifted Combines working in Java.
> Non-lifted combines should be done as an opaque ParDo after a GBK within a 
> Combine composite transform.
> This task also requires implementing the anonymous case, where the ParDo and 
> GBK are not in a composite and the runner cannot identify the ParDo as a 
> Combine at all. For now this case only occurs when the Combine has side 
> inputs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3889) Revise python streaming mobile gaming examples.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-3889:
-

Assignee: Ahmet Altay  (was: David Cavazos)

> Revise python streaming mobile gaming examples.
> ---
>
> Key: BEAM-3889
> URL: https://issues.apache.org/jira/browse/BEAM-3889
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.5.0
>
>
> I spent some time trying to run streaming examples [1] on Direct runner and 
> ran into the following issues:
>  * Examples use deprecated code as per warnings 
>  ** DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  project = pipeline.options.view_as(GoogleCloudOptions).project
> /tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py:337:
>  DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  pipeline.replace_all(_get_transform_overrides(pipeline.options))
>  * Path to Injector has a typo (java8 instead of java)
>  * I was not able to use the injector to produce an output on a pubsub topic, 
> but perhaps I missed some instructions. Injector would exit after:
>  ** Warning! You are not using service account credentials to authenticate.
> You need to use service account credentials for this example,
> since user-level credentials do not have enough pubsub quota,
> and so you will run out of PubSub quota very quickly.
>  ** Note that I set an environment variable GOOGLE_APPLICATION_CREDENTIALS 
> and also tried passing 
> -DGOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json.
>  * Examples say they are not applicable for Dataflow runner, is it still the 
> case?
>  * Running injector requires mvn, which is being replaced with gradle and 
> also requires familiarity with java toolchain, even though the example is for 
> Python SDK.
>  
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3889) Revise python streaming mobile gaming examples.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev closed BEAM-3889.
-
   Resolution: Duplicate
Fix Version/s: 2.5.0

> Revise python streaming mobile gaming examples.
> ---
>
> Key: BEAM-3889
> URL: https://issues.apache.org/jira/browse/BEAM-3889
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Major
> Fix For: 2.5.0
>
>
> I spent some time trying to run streaming examples [1] on Direct runner and 
> ran into the following issues:
>  * Examples use deprecated code as per warnings 
>  ** DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  project = pipeline.options.view_as(GoogleCloudOptions).project
> /tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py:337:
>  DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  pipeline.replace_all(_get_transform_overrides(pipeline.options))
>  * Path to Injector has a typo (java8 instead of java)
>  * I was not able to use the injector to produce an output on a pubsub topic, 
> but perhaps I missed some instructions. Injector would exit after:
>  ** Warning! You are not using service account credentials to authenticate.
> You need to use service account credentials for this example,
> since user-level credentials do not have enough pubsub quota,
> and so you will run out of PubSub quota very quickly.
>  ** Note that I set an environment variable GOOGLE_APPLICATION_CREDENTIALS 
> and also tried passing 
> -DGOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json.
>  * Examples say they are not applicable for Dataflow runner, is it still the 
> case?
>  * Running injector requires mvn, which is being replaced with gradle and 
> also requires familiarity with java toolchain, even though the example is for 
> Python SDK.
>  
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3875) Update Spark runner to Spark version 2.3.0

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3875?focusedWorklogId=83380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83380
 ]

ASF GitHub Bot logged work on BEAM-3875:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:08
Start Date: 22/Mar/18 23:08
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #4893: [BEAM-3875] Update 
Spark runner to use Spark version 2.3.0
URL: https://github.com/apache/beam/pull/4893#issuecomment-375486692
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83380)
Time Spent: 50m  (was: 40m)

> Update Spark runner to Spark version 2.3.0
> --
>
> Key: BEAM-3875
> URL: https://issues.apache.org/jira/browse/BEAM-3875
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Spark 2.3.0 was released in Feb 2018 and apart of the usual fixes, it has 
> early support to run jobs on Kubernetes, a new data source API and early 
> support for continuous processing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3875) Update Spark runner to Spark version 2.3.0

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3875?focusedWorklogId=83379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83379
 ]

ASF GitHub Bot logged work on BEAM-3875:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:07
Start Date: 22/Mar/18 23:07
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #4893: [BEAM-3875] Update 
Spark runner to use Spark version 2.3.0
URL: https://github.com/apache/beam/pull/4893#issuecomment-375486692
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83379)
Time Spent: 40m  (was: 0.5h)

> Update Spark runner to Spark version 2.3.0
> --
>
> Key: BEAM-3875
> URL: https://issues.apache.org/jira/browse/BEAM-3875
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Spark 2.3.0 was released in Feb 2018 and apart of the usual fixes, it has 
> early support to run jobs on Kubernetes, a new data source API and early 
> support for continuous processing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3910) Support floating point values in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3910?focusedWorklogId=83378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83378
 ]

ASF GitHub Bot logged work on BEAM-3910:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:01
Start Date: 22/Mar/18 23:01
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on issue #4941: BEAM-3910: Add float 
support for the Go SDK.
URL: https://github.com/apache/beam/pull/4941#issuecomment-375485190
 
 
   @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83378)
Time Spent: 20m  (was: 10m)
Remaining Estimate: 23h 40m  (was: 23h 50m)

> Support floating point values in Go SDK
> ---
>
> Key: BEAM-3910
> URL: https://issues.apache.org/jira/browse/BEAM-3910
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 20m
>  Remaining Estimate: 23h 40m
>
> The Go SDK supports all the integer types of the language, but does not 
> support floats.
> My plan for coding is to use the same technique the gob package uses, which 
> results in a compact encoding for simple values.
> [https://golang.org/src/encoding/gob/encode.go?#L210|https://golang.org/src/encoding/gob/encode.go#L210]
>  with rationale explained in 
> https://golang.org/pkg/encoding/gob/#hdr-Encoding_Details
> The resulting uint is then encoded using the existing coders in coderx.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-419) Non-transient non-serializable instance field in CombineFnUtil$NonSerializableBoundedKeyedCombineFn

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira resolved BEAM-419.
--
   Resolution: Invalid
Fix Version/s: Not applicable

The findbug issue turned out to be a inapplicable, the class was technically 
serializable, but was non-serializable by throwing exceptions if it tried to be 
serialized. So there was no problem having a non-serializable field in the 
class.

I replaced the reference to this issue in the xml file with a comment 
explaining why it should be ignored.

> Non-transient non-serializable instance field in 
> CombineFnUtil$NonSerializableBoundedKeyedCombineFn
> ---
>
> Key: BEAM-419
> URL: https://issues.apache.org/jira/browse/BEAM-419
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Daniel Oliveira
>Priority: Minor
>  Labels: findbugs, newbie, starter
> Fix For: Not applicable
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> [FindBugs 
> SE_BAD_FIELD|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L363]:
>  Non-transient non-serializable instance field in serializable class
> Applies to: 
> [CombineFnUtil$NonSerializableBoundedKeyedCombineFn.context|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CombineFnUtil.java#L170].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3910) Support floating point values in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3910?focusedWorklogId=83377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83377
 ]

ASF GitHub Bot logged work on BEAM-3910:


Author: ASF GitHub Bot
Created on: 22/Mar/18 23:01
Start Date: 22/Mar/18 23:01
Worklog Time Spent: 10m 
  Work Description: wcn3 opened a new pull request #4941: BEAM-3910: Add 
float support for the Go SDK.
URL: https://github.com/apache/beam/pull/4941
 
 
   Adds support for floating point types with an efficient encoding scheme.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83377)
Time Spent: 10m
Remaining Estimate: 23h 50m  (was: 24h)

> Support floating point values in Go SDK
> ---
>
> Key: BEAM-3910
> URL: https://issues.apache.org/jira/browse/BEAM-3910
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 10m
>  Remaining Estimate: 23h 50m
>
> The Go SDK supports all the integer types of the language, but does not 
> support floats.
> My plan for coding is to use the same technique the gob package uses, which 
> results in a compact encoding for simple values.
> [https://golang.org/src/encoding/gob/encode.go?#L210|https://golang.org/src/encoding/gob/encode.go#L210]
>  with rationale explained in 
> https://golang.org/pkg/encoding/gob/#hdr-Encoding_Details
> The resulting uint is then encoded using the existing coders in coderx.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83373
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 22:45
Start Date: 22/Mar/18 22:45
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375481906
 
 
   Changes are made:
   1. add comments in Injector that indicate the team list should sync to the 
list in MobileGamingJavaUtils.
   2. Use map to create arguments for each examples.
   3. Rewrite command generation method in MobileGamingJavaUtils.
   4. Add "-q" option for each quickstart command that stop writing [Info] 
messages into console during maven compile. This make the console output 
cleaner.
   PTAL @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83373)
Time Spent: 75h 50m  (was: 75h 40m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 75h 50m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3913) Support custom runner primitives in Fusion

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3913?focusedWorklogId=83367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83367
 ]

ASF GitHub Bot logged work on BEAM-3913:


Author: ASF GitHub Bot
Created on: 22/Mar/18 22:24
Start Date: 22/Mar/18 22:24
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4938: 
[BEAM-3913] Allow Fusion to Continue with unknown PTransforms
URL: https://github.com/apache/beam/pull/4938#discussion_r176589982
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
 ##
 @@ -276,16 +281,27 @@ private static boolean unknownTransformFusion(
   @SuppressWarnings("unused") Environment environment,
   @SuppressWarnings("unused") Collection 
stagePCollections,
   @SuppressWarnings("unused") QueryablePipeline pipeline) {
-throw new IllegalArgumentException(
-String.format("Unknown URN %s", 
transform.getTransform().getSpec().getUrn()));
+LOG.debug(
+"Unknown {} {} will not fuse into an existing {}",
+PTransform.class.getSimpleName(),
+transform.getTransform(),
+ExecutableStage.class.getSimpleName(),
+PTransform.class.getSimpleName());
+return false;
   }
 
+  // Things with unknown URNs either execute within their own stage or are 
executed by the runner.
+  // In either case, assume the
 
 Review comment:
   comment is incomplete.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83367)
Time Spent: 20m  (was: 10m)

> Support custom runner primitives in Fusion
> --
>
> Key: BEAM-3913
> URL: https://issues.apache.org/jira/browse/BEAM-3913
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Effectively, register runner-provided transform URNs in the 
> GreedyPCollectionFusers to understand runner-specific primitives, and don't 
> ever fuse them.
>  
> It's probably appropriate to continue to crash when encountering truly 
> unknown URNs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3706) Update CombinePayload to improved model for Portability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3706?focusedWorklogId=83358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83358
 ]

ASF GitHub Bot logged work on BEAM-3706:


Author: ASF GitHub Bot
Created on: 22/Mar/18 22:15
Start Date: 22/Mar/18 22:15
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #4924: [BEAM-3706] Removing 
side inputs from CombinePayload proto.
URL: https://github.com/apache/beam/pull/4924#issuecomment-375475383
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83358)
Time Spent: 1h  (was: 50m)

> Update CombinePayload to improved model for Portability
> ---
>
> Key: BEAM-3706
> URL: https://issues.apache.org/jira/browse/BEAM-3706
> Project: Beam
>  Issue Type: Sub-task
>  Components: beam-model
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> This will mean changing the proto definition in beam_runner_api, most likely 
> trimming out fields that are no longer necessary and adding any new ones that 
> could be useful. The majority of work will probably be in investigating if 
> some existing fields can actually be removed (SideInputs and Parameters for 
> example).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #155

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3918) test failure BeamSqlDslAggregationTest.testTriggeredTumble

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3918:


 Summary: test failure BeamSqlDslAggregationTest.testTriggeredTumble
 Key: BEAM-3918
 URL: https://issues.apache.org/jira/browse/BEAM-3918
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql, testing
Reporter: Xu Mingmin
Assignee: Kenneth Knowles


I cannot pass the test case added in 
[https://github.com/apache/beam/pull/4826,] here's the error message: (run with 
{{mvn clean install -pl sdks/java/extensions/sql/}})
{code}
[ERROR] Failures:
[ERROR]   BeamSqlDslAggregationTest.testTriggeredTumble:384 Windowed 
Query/BeamProjectRel.Transform/BEAMPROJECTREL_1566_149/ParMultiDo(BeamSqlProject).output:
Expected: iterable over [,
 ,
 ]
 in any order
 but: No item matches: ,
 
 in []
{code}

What it confuses me is, the failure is not found in precommit jobs.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83348
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375456963
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83348)
Time Spent: 75.5h  (was: 75h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 75.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83344
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375382898
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83344)
Time Spent: 74h 50m  (was: 74h 40m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 74h 50m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83349
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375459972
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83349)
Time Spent: 75h 40m  (was: 75.5h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 75h 40m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83345
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375408771
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83345)
Time Spent: 75h  (was: 74h 50m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 75h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83343
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375371298
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83343)
Time Spent: 74h 40m  (was: 74.5h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 74h 40m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83340
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375459972
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83340)
Time Spent: 74h 10m  (was: 74h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 74h 10m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83342
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375196050
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83342)
Time Spent: 74.5h  (was: 74h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 74.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83346
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375434961
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83346)
Time Spent: 75h 10m  (was: 75h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 75h 10m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83341=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83341
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375179867
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83341)
Time Spent: 74h 20m  (was: 74h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 74h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83347
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:15
Start Date: 22/Mar/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375450609
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83347)
Time Spent: 75h 20m  (was: 75h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 75h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3749.

Resolution: Won't Fix

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5202

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=83338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83338
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:04
Start Date: 22/Mar/18 21:04
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#4930: [BEAM-3861] Complete streaming wordcount test in Python SDK
URL: https://github.com/apache/beam/pull/4930#discussion_r176571812
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
 ##
 @@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""PubSub verifier used for end-to-end test."""
+
+import logging
+import time
+from collections import Counter
+
+from hamcrest.core.base_matcher import BaseMatcher
+
+__all__ = ['PubSubMessageMatcher']
+
+
+# Protect against environments where pubsub library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+  from google.cloud import pubsub
+except ImportError:
+  pubsub = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+DEFAULT_TIMEOUT = 5 * 60
+MAX_MESSAGES_IN_ONE_PULL = 50
+
+
+class PubSubMessageMatcher(BaseMatcher):
+  """Matcher that verifies messages from given subscription.
+
+  This matcher can block the test and keep pulling messages from given
+  subscription until all expected messages are shown or timeout.
+  """
+
+  def __init__(self, project, sub_name, expected_msg, timeout=DEFAULT_TIMEOUT):
+"""Initialize PubSubMessageMatcher object.
+
+Args:
+  project: A name string of project.
+  sub_name: A name string of subscription which is attached to output.
+  expected_msg: A string list that contains expected message data pulled
+from the subscription.
+  timeout: Timeout in seconds to wait for all expected messages appears.
+"""
+if pubsub is None:
+  raise ImportError(
+  'PubSub dependencies are not installed.')
+if not project:
+  raise ValueError('Invalid project %s.' % project)
+if not sub_name:
+  raise ValueError('Invalid subscription %s.' % sub_name)
+if not expected_msg or not isinstance(expected_msg, list):
+  raise ValueError('Invalid expected messages %s.' % expected_msg)
+
+self.project = project
+self.sub_name = sub_name
+self.expected_msg = expected_msg
+self.timeout = timeout
+self.messages = None
+
+  def _matches(self, _):
+if not self.messages:
+  subscription = (pubsub
+  .Client(project=self.project)
+  .subscription(self.sub_name))
+  self.messages = self._wait_for_messages(subscription,
 
 Review comment:
   just realize this method returns received messages, will return instead of 
raise exception in all cases.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83338)
Time Spent: 6h 20m  (was: 6h 10m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=83339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83339
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 22/Mar/18 21:04
Start Date: 22/Mar/18 21:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-375456963
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83339)
Time Spent: 74h  (was: 73h 50m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 74h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #154

[Apache Jenkins Server]

See 


--
GitHub pull request #4788 of commit d63df0d71693e95b714db11532f01c440421ae72, 
no merge conflicts.
Setting status of d63df0d71693e95b714db11532f01c440421ae72 to PENDING with url 
https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/154/ and 
message: 'Build started sha1 is merged.'
Using context: Jenkins: ./gradlew :release:runQuickstartsJava
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4788/*:refs/remotes/origin/pr/4788/*
 > git rev-parse refs/remotes/origin/pr/4788/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4788/merge^{commit} # timeout=10
Checking out Revision 979973fc0f1977b346a5cf74f3b6195326225edc 
(refs/remotes/origin/pr/4788/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 979973fc0f1977b346a5cf74f3b6195326225edc
Commit message: "Merge d63df0d71693e95b714db11532f01c440421ae72 into 
501dc4cb17bb943aaa095feab959a9fed1aac20c"
 > git rev-list --no-walk 97946107e3b49154fa405334ba787302d2e6c3d1 # timeout=10
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[Gradle] - Launching build.
[src] $ 
 
-Pver= -Prepourl= :release:runQuickstartsJava
Parallel execution with configuration on demand is an incubating feature.
Applying build_rules.gradle to src

FAILURE: Build failed with an exception.

* What went wrong:
Task 'runQuickstartsJava' not found in project ':release'.

* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace 
option to get the stack trace. Run with --info or --debug option to get more 
log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 23s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure