[jira] [Commented] (BEAM-3399) Documentation wrong for Session Windows (Python SDK)

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478552#comment-16478552
 ] 

Ahmet Altay commented on BEAM-3399:
---

Python code is still wrong:

items | 'window' >> beam.WindowInto(window.Sessions(10)))

> Documentation wrong for Session Windows (Python SDK)
> 
>
> Key: BEAM-3399
> URL: https://issues.apache.org/jira/browse/BEAM-3399
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Affects Versions: 2.2.0
>Reporter: Ralf Hein
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: starter
> Fix For: Not applicable
>
>
> The documentation for setting Session Windows via python SDK is wrong 
> (Section 7.3.3).
> https://beam.apache.org/documentation/programming-guide/#setting-your-pcollections-windowing-function
> It says "time gap of at least 10 minutes" when introducing the example code, 
> but the code actually only sets a duration gap of 10 seconds.
> Reason: The duration needs to be passed in seconds, not minutes. So the 
> example code should read something like this:
> beam.WindowInto(window.Sessions(10*60)))
> Took me quite some time to figure this out ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #280

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=102770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102770
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 17/May/18 04:38
Start Date: 17/May/18 04:38
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5290: [BEAM-3983] 
Restore BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/extensions/sql/pom.xml b/sdks/java/extensions/sql/pom.xml
index d893205e96d..63f5022e964 100644
--- a/sdks/java/extensions/sql/pom.xml
+++ b/sdks/java/extensions/sql/pom.xml
@@ -402,6 +402,12 @@
   provided
 
 
+
+  org.apache.beam
+  beam-sdks-java-io-google-cloud-platform
+  provided
+
+
 
 
   junit
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQueryTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQueryTable.java
new file mode 100644
index 000..6bfd839c018
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQueryTable.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.impl.schema.BeamIOType;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
+import org.apache.beam.sdk.io.gcp.bigquery.WriteResult;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * {@code BeamBigQueryTable} represent a BigQuery table as a target.
+ * This provider does not currently support being a source.
+ *
+ */
+@Experimental
+public class BeamBigQueryTable extends BaseBeamTable implements Serializable {
+  private String tableSpec;
+
+  public BeamBigQueryTable(Schema beamSchema, String tableSpec) {
+super(beamSchema);
+this.tableSpec = tableSpec;
+  }
+
+  @Override
+  public BeamIOType getSourceType() {
+return BeamIOType.BOUNDED;
+  }
+
+  @Override
+  public PCollection buildIOReader(Pipeline pipeline) {
+throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public PTransform, POutput> buildIOWriter() {
+return new PTransform() {
+  @Override
+  public WriteResult expand(PCollection input) {
+return input.apply(BigQueryIO.write()
+  .withSchema(BigQueryUtils.toTableSchema(getSchema()))
+  .withFormatFunction(BigQueryUtils.toTableRow())
+  .to(tableSpec));
+  }
+};
+  }
+
+  public String getTableSpec() {
+return tableSpec;
+  }
+}
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
new file mode 100644
index 000..4d1bd2d9155
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=102769=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102769
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 17/May/18 04:38
Start Date: 17/May/18 04:38
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5290: [BEAM-3983] 
Restore BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-389742085
 
 
   Boss.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102769)
Time Spent: 33h 10m  (was: 33h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 33h 10m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1677

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-3983] Add utils for converting to BigQuery types

[apilloud] [BEAM-3983][SQL] Add BigQuery table provider

[apilloud] [BEAM-4248] Copy enums from com.google.cloud

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 9ba58eea7bbaffdb16f849836cf51c1f59282d06 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9ba58eea7bbaffdb16f849836cf51c1f59282d06
Commit message: "Merge pull request #5290: [BEAM-3983] Restore BigQuery SQL 
Support with copied enums"
 > git rev-list --no-walk ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5320334607513520364.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk

[jira] [Commented] (BEAM-3377) assert_that not working for streaming

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478493#comment-16478493
 ] 

María GH commented on BEAM-3377:


I added streaming_wordcount_debugging.py, including:
 *  PrintFn (DoFn) to inspect element, window, and timestamp.
 *  AddTimestampFn (DoFn) to modify timestamps.
 *  assert_that via check_gbk_format and equal_to_per_window (matchers).
 *  Change parameter custom_windowing to use_global_window to reuse the 
workflow's windowing when possible.
 *  Update test_stream_test.py test to use assert_that for either global or 
per-window spaces appropriately.

It works in the DirectRunner, but it doesn't in the DataflowRunner on two 
grounds:
 1) Using a controlled emission of meesages: streaming_wordcount_it_test fails 
when using AddTimestampDoFn.
 2) Using a simple matcher to check the format (word:count): It reports:
 Caused by: java.lang.ClassCastException: 
org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to 
org.apache.beam.sdk.transforms.windowing.GlobalWindow.
 I have created 
[assert-addts|https://github.com/mariapython/incubator-beam/tree/assert-addts] 
(private branch) and added two commits for 1 and 2, so that this work can be 
revisited in the future.
 1) Show AddTimestampDoFn fails Dataflow streaming wordcount
 2) Show a simple format checker fails Dataflow streaming wordcount

> assert_that not working for streaming
> -
>
> Key: BEAM-3377
> URL: https://issues.apache.org/jira/browse/BEAM-3377
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: María GH
>Priority: Major
>  Labels: starter
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> assert_that does not work for AfterWatermark timers.
> Easy way to reproduce: modify test_gbk_execution [1] in this form:
>  
> {code:java}
>  def test_this(self):
> test_stream = (TestStream()
>.add_elements(['a', 'b', 'c'])
>.advance_watermark_to(20))
> def fnc(x):
>   print 'fired_elem:', x
>   return x
> options = PipelineOptions()
> options.view_as(StandardOptions).streaming = True
> p = TestPipeline(options=options)
> records = (p
>| test_stream
>| beam.WindowInto(
>FixedWindows(15),
>
> trigger=trigger.AfterWatermark(early=trigger.AfterCount(2)),
>accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
>| beam.Map(lambda x: ('k', x))
>| beam.GroupByKey())
> assert_that(records, equal_to([
> ('k', ['a', 'b', 'c'])]))
> p.run()
> {code}
> This test will pass, but if the .advance_watermark_to(20) is removed, the 
> test will fail. However, both cases fire the same elements:
>   fired_elem: ('k', ['a', 'b', 'c'])
>   fired_elem: ('k', ['a', 'b', 'c'])
> In the passing case, they correspond to the sorted_actual inside the 
> assert_that. In the failing case:
>   sorted_actual: [('k', ['a', 'b', 'c']), ('k', ['a', 'b', 'c'])]
>   sorted_actual: []
> [1] 
> https://github.com/mariapython/incubator-beam/blob/direct-timers-show/sdks/python/apache_beam/testing/test_stream_test.py#L120



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #279

[Apache Jenkins Server]

See 


Changes:

[github] [BEAM-4300] Fix ValidatesRunner tests in Python: run with same 
mechanism

--
[...truncated 10.36 MB...]
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 17, 2018 3:24:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 17, 2018 3:24:32 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0517032426-c1429b03/output/results/staging/
May 17, 2018 3:24:32 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash LWyOpvYN2RKW1cKs_QOCpw> to 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1676

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6
Commit message: "Merge pull request #5306: [BEAM-4263] Bugfix: Read BQ bytes 
processed from correct field"
 > git rev-list --no-walk ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5896598651452274744.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=102758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102758
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 17/May/18 02:13
Start Date: 17/May/18 02:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5053: 
[BEAM-3981] Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#discussion_r188822509
 
 

 ##
 File path: sdks/python/apache_beam/coders/coders.py
 ##
 @@ -210,11 +216,15 @@ def as_cloud_object(self):
   def __repr__(self):
 return self.__class__.__name__
 
+  # pylint: disable=protected-access
   def __eq__(self, other):
-# pylint: disable=protected-access
 return (self.__class__ == other.__class__
 and self._dict_without_impl() == other._dict_without_impl())
-# pylint: enable=protected-access
+
+  def __hash__(self):
 
 Review comment:
   That's true. Although that wouldn't honor the contract between __eq__ and 
__hash__.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102758)
Time Spent: 19h  (was: 18h 50m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 19h
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3999) Futurize and fix python 2 compatibility for internal subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3999?focusedWorklogId=102757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102757
 ]

ASF GitHub Bot logged work on BEAM-3999:


Author: ASF GitHub Bot
Created on: 17/May/18 02:00
Start Date: 17/May/18 02:00
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5334: 
[BEAM-3999] Futurize internal subpackage
URL: https://github.com/apache/beam/pull/5334#discussion_r188820867
 
 

 ##
 File path: sdks/python/apache_beam/internal/util.py
 ##
 @@ -20,9 +20,13 @@
 For internal use only. No backwards compatibility guarantees.
 """
 
+from __future__ import absolute_import
+
 import logging
 import threading
 import weakref
+from builtins import next
 
 Review comment:
   I'd like to understand the reason for adding `from builtins import next`. Is 
it  done by a conversion tool because we have an occurrence of `next(v_iter)` 
in the file, and the tool cannot infer whether or not v_iter implements a 
custom iterator? If yes: will such changes be required  by linter? 
   
   Is my understanding correct that as long as `v_iter.__class__`  does not 
implement a custom iterator (and it is not, in our case), the code without the 
import is still equivalent in Python 2 and Python 3?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102757)
Time Spent: 1h  (was: 50m)

> Futurize and fix python 2 compatibility for internal subpackage
> ---
>
> Key: BEAM-3999
> URL: https://issues.apache.org/jira/browse/BEAM-3999
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3999) Futurize and fix python 2 compatibility for internal subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3999?focusedWorklogId=102756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102756
 ]

ASF GitHub Bot logged work on BEAM-3999:


Author: ASF GitHub Bot
Created on: 17/May/18 02:00
Start Date: 17/May/18 02:00
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5334: 
[BEAM-3999] Futurize internal subpackage
URL: https://github.com/apache/beam/pull/5334#discussion_r188820867
 
 

 ##
 File path: sdks/python/apache_beam/internal/util.py
 ##
 @@ -20,9 +20,13 @@
 For internal use only. No backwards compatibility guarantees.
 """
 
+from __future__ import absolute_import
+
 import logging
 import threading
 import weakref
+from builtins import next
 
 Review comment:
   I'd like to understand the reason for adding `from builtins import next`. Is 
it  done by a conversion tool because we have an occurrence of `next(v_iter)` 
in the file, and the tool cannot infer whether or not v_iter implements a 
custom iterator? If yes: will such changes be required  by linter? 
   
   Is my understanding correct that as long as `v_iter.__class__` class does 
not implement a custom iterator (and it is not, in our case), the code without 
the import is still equivalent in Python 2 and Python 3?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102756)
Time Spent: 50m  (was: 40m)

> Futurize and fix python 2 compatibility for internal subpackage
> ---
>
> Key: BEAM-3999
> URL: https://issues.apache.org/jira/browse/BEAM-3999
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3999) Futurize and fix python 2 compatibility for internal subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3999?focusedWorklogId=102754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102754
 ]

ASF GitHub Bot logged work on BEAM-3999:


Author: ASF GitHub Bot
Created on: 17/May/18 01:59
Start Date: 17/May/18 01:59
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5334: 
[BEAM-3999] Futurize internal subpackage
URL: https://github.com/apache/beam/pull/5334#discussion_r188820867
 
 

 ##
 File path: sdks/python/apache_beam/internal/util.py
 ##
 @@ -20,9 +20,13 @@
 For internal use only. No backwards compatibility guarantees.
 """
 
+from __future__ import absolute_import
+
 import logging
 import threading
 import weakref
+from builtins import next
 
 Review comment:
   I'd like to understand the reason for adding `from builtins import next`. Is 
it  done by a conversion tool because we have an occurrence of `next(v_iter)` 
in the file, and the tool cannot infer whether or not v_iter implements a 
custom iterator? If yes: will such changes be required  by linter? 
   
   Is my understanding correct that as long as v_iter's class does not 
implement a custom iterator (and it is not, in our case), the code without the 
import is still equivalent in Python 2 and Python 3?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102754)
Time Spent: 0.5h  (was: 20m)

> Futurize and fix python 2 compatibility for internal subpackage
> ---
>
> Key: BEAM-3999
> URL: https://issues.apache.org/jira/browse/BEAM-3999
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3999) Futurize and fix python 2 compatibility for internal subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3999?focusedWorklogId=102755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102755
 ]

ASF GitHub Bot logged work on BEAM-3999:


Author: ASF GitHub Bot
Created on: 17/May/18 01:59
Start Date: 17/May/18 01:59
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5334: 
[BEAM-3999] Futurize internal subpackage
URL: https://github.com/apache/beam/pull/5334#discussion_r188820867
 
 

 ##
 File path: sdks/python/apache_beam/internal/util.py
 ##
 @@ -20,9 +20,13 @@
 For internal use only. No backwards compatibility guarantees.
 """
 
+from __future__ import absolute_import
+
 import logging
 import threading
 import weakref
+from builtins import next
 
 Review comment:
   I'd like to understand the reason for adding `from builtins import next`. Is 
it  done by a conversion tool because we have an occurrence of `next(v_iter)` 
in the file, and the tool cannot infer whether or not v_iter implements a 
custom iterator? If yes: will such changes be required  by linter? 
   
   Is my understanding correct that as long as `v_iter`'s class does not 
implement a custom iterator (and it is not, in our case), the code without the 
import is still equivalent in Python 2 and Python 3?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102755)
Time Spent: 40m  (was: 0.5h)

> Futurize and fix python 2 compatibility for internal subpackage
> ---
>
> Key: BEAM-3999
> URL: https://issues.apache.org/jira/browse/BEAM-3999
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102749
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 01:38
Start Date: 17/May/18 01:38
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-389717316
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102749)
Time Spent: 14h 40m  (was: 14.5h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 14h 40m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102748
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 01:38
Start Date: 17/May/18 01:38
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-389717242
 
 
   Updated the PR based on our discussion.
   PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102748)
Time Spent: 14.5h  (was: 14h 20m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 14.5h
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1675

[Apache Jenkins Server]

See 


--
GitHub pull request #5251 of commit 43e7dbeea0ffb974082cb3cc1bf4e6f22b9e7fb3, 
no merge conflicts.
Setting status of 43e7dbeea0ffb974082cb3cc1bf4e6f22b9e7fb3 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesRunner_Dataflow/1675/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesRunner 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5251/*:refs/remotes/origin/pr/5251/*
 > git rev-parse refs/remotes/origin/pr/5251/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5251/merge^{commit} # timeout=10
Checking out Revision aea3278422b40c00dc5d10f8a057c6df18d24aaa 
(refs/remotes/origin/pr/5251/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f aea3278422b40c00dc5d10f8a057c6df18d24aaa
Commit message: "Merge 43e7dbeea0ffb974082cb3cc1bf4e6f22b9e7fb3 into 
ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins4464134076579792224.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=102746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102746
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 17/May/18 01:17
Start Date: 17/May/18 01:17
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #5180: [BEAM-4065] Basic 
performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#issuecomment-389714115
 
 
   Please let me know when this is ready for another look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102746)
Time Spent: 10h  (was: 9h 50m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3433) Allow BigQueryIO to use a different project for the load job in batch mode.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3433?focusedWorklogId=102745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102745
 ]

ASF GitHub Bot logged work on BEAM-3433:


Author: ASF GitHub Bot
Created on: 17/May/18 01:15
Start Date: 17/May/18 01:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #5178: [BEAM-3433] Allow 
a GCP project to be explicitly set for a load job
URL: https://github.com/apache/beam/pull/5178#issuecomment-389713747
 
 
   LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102745)
Time Spent: 1h 40m  (was: 1.5h)

> Allow BigQueryIO to use a different project for the load job in batch mode.
> ---
>
> Key: BEAM-3433
> URL: https://issues.apache.org/jira/browse/BEAM-3433
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Peterson
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> BigQueryIO is currently configured to always run a batch load job using the 
> same projects as the destination table: 
> https://github.com/apache/beam/blob/192b4c70927901860312f8c8acd27bd47e4a4259/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L256
> This may not always be desirable, since a pipeline may have write access to a 
> dataset in a different project, but not jobs.create access in that project. 
> This parameter should be settable in the interface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1674

[Apache Jenkins Server]

See 


Changes:

[kmj] Bugfix: Read BQ bytes processed from correct field.

--
Started by GitHub push by chamikaramj
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6
Commit message: "Merge pull request #5306: [BEAM-4263] Bugfix: Read BQ bytes 
processed from correct field"
 > git rev-list --no-walk d41df7b17f49569d37a7cdcc618408daba55 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5870590270194592050.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk

[jira] [Work logged] (BEAM-4263) BigQuery connector reads the table size value from a deprecated field

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4263?focusedWorklogId=102744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102744
 ]

ASF GitHub Bot logged work on BEAM-4263:


Author: ASF GitHub Bot
Created on: 17/May/18 01:12
Start Date: 17/May/18 01:12
Worklog Time Spent: 10m 
  Work Description: chamikaramj closed pull request #5306: [BEAM-4263] 
Bugfix: Read BQ bytes processed from correct field.
URL: https://github.com/apache/beam/pull/5306
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
index f380b7d391b..979f8b9d1b3 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
@@ -102,7 +102,7 @@ private BigQueryQuerySource(
   @Override
   public long getEstimatedSizeBytes(PipelineOptions options) throws Exception {
 BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
-return dryRunQueryIfNeeded(bqOptions).getTotalBytesProcessed();
+return dryRunQueryIfNeeded(bqOptions).getQuery().getTotalBytesProcessed();
   }
 
   @Override
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
index b6fbe4905b4..213ee3d45f0 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
@@ -560,6 +560,40 @@ public void testEstimatedSizeWithStreamingBuffer() throws 
Exception {
 assertEquals(118, bqSource.getEstimatedSizeBytes(options));
   }
 
+  @Test
+  public void testBigQueryQuerySourceEstimatedSize() throws Exception {
+
+List data = ImmutableList.of(
+new TableRow().set("name", "A").set("number", 10L),
+new TableRow().set("name", "B").set("number", 11L),
+new TableRow().set("name", "C").set("number", 12L));
+
+PipelineOptions options = PipelineOptionsFactory.create();
+BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
+bqOptions.setProject("project");
+String stepUuid = "testStepUuid";
+
+String query = FakeBigQueryServices.encodeQuery(data);
+BigQueryQuerySource bqSource = BigQueryQuerySource.create(
+stepUuid,
+ValueProvider.StaticValueProvider.of(query),
+true /* flattenResults */,
+true /* useLegacySql */,
+fakeBqServices,
+TableRowJsonCoder.of(),
+BigQueryIO.TableRowParser.INSTANCE,
+QueryPriority.BATCH,
+null);
+
+fakeJobService.expectDryRunQuery(
+bqOptions.getProject(),
+query,
+new JobStatistics().setQuery(
+new JobStatistics2().setTotalBytesProcessed(100L)));
+
+assertEquals(100, bqSource.getEstimatedSizeBytes(bqOptions));
+  }
+
   @Test
   public void testBigQueryQuerySourceInitSplit() throws Exception {
 TableReference dryRunTable = new TableReference();


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102744)
Time Spent: 10m
Remaining Estimate: 0h

> BigQuery connector reads the table size value from a deprecated field
> -
>
> Key: BEAM-4263
> URL: https://issues.apache.org/jira/browse/BEAM-4263
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Kenneth Jung
>Assignee: Kenneth Jung
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The BigQuery connector in the GCP IO module reads the totalBytesProcessed 
> value from a deprecated field in the job statistics:
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs]
> The non-deprecated replacement is the totalBytesProcessed 

[beam] branch master updated (d41df7b -> ab4a2d8)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d41df7b  [BEAM-4300] Fix ValidatesRunner tests in Python: run with 
same mechanism as ITs (#5372)
 add abe3ac7  Bugfix: Read BQ bytes processed from correct field.
 new ab4a2d8  Merge pull request #5306: [BEAM-4263] Bugfix: Read BQ bytes 
processed from correct field

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/io/gcp/bigquery/BigQueryQuerySource.java   |  2 +-
 .../sdk/io/gcp/bigquery/BigQueryIOReadTest.java| 34 ++
 2 files changed, 35 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] 01/01: Merge pull request #5306: [BEAM-4263] Bugfix: Read BQ bytes processed from correct field

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ab4a2d819bf6e9a7ed7004414af334bbbf4ca8e6
Merge: d41df7b abe3ac7
Author: Chamikara Jayalath 
AuthorDate: Wed May 16 18:12:23 2018 -0700

Merge pull request #5306: [BEAM-4263] Bugfix: Read BQ bytes processed from 
correct field

 .../sdk/io/gcp/bigquery/BigQueryQuerySource.java   |  2 +-
 .../sdk/io/gcp/bigquery/BigQueryIOReadTest.java| 34 ++
 2 files changed, 35 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Commented] (BEAM-4318) Enforce ErrorProne analysis in Spark runner project

[Teng Peng (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478315#comment-16478315
 ] 

Teng Peng commented on BEAM-4318:
-

Thanks for the instruction. I have run ./gradlew :beam-runners-spark:assemble, 
but I do not see any warnings.

 

Mac:Beam pt$ ./gradlew :beam-runners-spark:assemble
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
createPerformanceTestHarness with default configuration for project beam
Adding 47 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-java-core
Applying build_rules.gradle to beam-sdks-java-core
applyJavaNature with 
[shadowClosure:org.codehaus.groovy.runtime.ComposedClosure@536a44fb] for 
project beam-sdks-java-core
applyAvroNature with default configuration for project beam-sdks-java-core

> Configure project :beam-runners-spark
Applying build_rules.gradle to beam-runners-spark
applyJavaNature with [failOnWarning:true] for project beam-runners-spark
Generating :runQuickstartJavaSpark

> Configure project :beam-model-pipeline
Applying build_rules.gradle to beam-model-pipeline
applyJavaNature with [enableFindbugs:false, enableErrorProne:false] for project 
beam-model-pipeline
applyGrpcNature with default configuration for project beam-model-pipeline

> Configure project :beam-runners-core-construction-java
Applying build_rules.gradle to beam-runners-core-construction-java
applyJavaNature with default configuration for project 
beam-runners-core-construction-java

> Configure project :beam-runners-core-java
Applying build_rules.gradle to beam-runners-core-java
applyJavaNature with default configuration for project beam-runners-core-java

> Configure project :beam-sdks-java-io-kafka
Applying build_rules.gradle to beam-sdks-java-io-kafka
applyJavaNature with [enableFindbugs:false] for project beam-sdks-java-io-kafka

> Configure project :beam-model-job-management
Applying build_rules.gradle to beam-model-job-management
applyJavaNature with [enableFindbugs:false, enableErrorProne:false] for project 
beam-model-job-management
applyGrpcNature with default configuration for project beam-model-job-management

> Configure project :beam-model-fn-execution
Applying build_rules.gradle to beam-model-fn-execution
applyJavaNature with [enableFindbugs:false, enableErrorProne:false] for project 
beam-model-fn-execution
applyGrpcNature with default configuration for project beam-model-fn-execution

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3s
37 actionable tasks: 37 up-to-date

> Enforce ErrorProne analysis in Spark runner project
> ---
>
> Key: BEAM-4318
> URL: https://issues.apache.org/jira/browse/BEAM-4318
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-spark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-spark:assemble}}
> # Fix each ErrorProne warning from the {{runners/spark}} project.
> # In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (BEAM-4318) Enforce ErrorProne analysis in Spark runner project

[Teng Peng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teng Peng updated BEAM-4318:

Comment: was deleted

(was: I am working on it.)

> Enforce ErrorProne analysis in Spark runner project
> ---
>
> Key: BEAM-4318
> URL: https://issues.apache.org/jira/browse/BEAM-4318
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-spark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-spark:assemble}}
> # Fix each ErrorProne warning from the {{runners/spark}} project.
> # In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102741
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 00:37
Start Date: 17/May/18 00:37
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-389708022
 
 
   @valentyn I am converting all the staticmethods to module methods as mocking 
static methods is even more difficult.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102741)
Time Spent: 14h 20m  (was: 14h 10m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 14h 20m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=102740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102740
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 17/May/18 00:33
Start Date: 17/May/18 00:33
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #5392: [BEAM-4267] 
JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#issuecomment-389707561
 
 
   CC: @angoenka @axelmagn @jkff @tgroh 
   
   Note that this is a WIP and depends on some unsubmitted PRs (`IdGenerators` 
and a refactoring of `DockerEnvironmentFactory`) as well as some unimplemented 
code (an `ArtifactRetrievalService` implementation and a cross-platform 
friendly `ServerFactory`).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102740)
Time Spent: 20m  (was: 10m)

> Implement a reusable library that can run an ExecutableStage with a given 
> Environment
> -
>
> Key: BEAM-4267
> URL: https://issues.apache.org/jira/browse/BEAM-4267
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Axel Magnuson
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Build off of the interfaces introduced in 
> [BEAM-3327|https://github.com/apache/beam/pull/5152] to provide a reusable 
> execution library to runners.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=102739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102739
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 17/May/18 00:29
Start Date: 17/May/18 00:29
Worklog Time Spent: 10m 
  Work Description: bsidhom opened a new pull request #5392: [BEAM-4267] 
JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392
 
 
   This implementation caches environments using weak references. This allows 
environments to be reused by multiple clients in the context of the same job. 
Note that runners will need to implement their own mechanisms of sharing this 
bundle factory between clients. For example, in the case of Flink, this will 
likely be the job of `FlinkExecutableStageContext`.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102739)
Time Spent: 10m
Remaining Estimate: 0h

> Implement a reusable library that can run an ExecutableStage with a given 
> Environment
> -
>
> Key: BEAM-4267
> URL: https://issues.apache.org/jira/browse/BEAM-4267
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Axel Magnuson
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Build off of the interfaces introduced in 
> [BEAM-3327|https://github.com/apache/beam/pull/5152] to provide a reusable 
> execution library to runners.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #177

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] Fix error-prone and some static analysis warnings in

[daniel.o.programmer] [BEAM-4277] Adding unshaded jars used by Dataflow to poms.

[lukasz.gajowy] [BEAM-4140] Utilize beam_location perfkit flag

[swegner] Upgrade Byte Buddy and FindBugs annotations dependencies.

[swegner] Fix ErrorProne warnings

[swegner] Remove isTriggerSpecified from WindowingStategy equality checks.

[lcwik] Add window mapping transform.

[lcwik] Use the coder from the side input PCollection definition by removing the

[lcwik] Remove garbage collection from PValueCache

[Pablo] Fixing incorrect assert in dependency_test.py

--
[...truncated 364.85 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
xmlioit0writethenreadall--05161706-xenx-harness-9npq.c.apache-beam-testing.internal/10.128.0.16
 to 224.155.188.35.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #184

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] Fix error-prone and some static analysis warnings in

[daniel.o.programmer] [BEAM-4277] Adding unshaded jars used by Dataflow to poms.

[lukasz.gajowy] [BEAM-4140] Utilize beam_location perfkit flag

[swegner] Upgrade Byte Buddy and FindBugs annotations dependencies.

[swegner] Fix ErrorProne warnings

[swegner] Remove isTriggerSpecified from WindowingStategy equality checks.

[lcwik] Add window mapping transform.

[lcwik] Use the coder from the side input PCollection definition by removing the

[lcwik] Remove garbage collection from PValueCache

[Pablo] Fixing incorrect assert in dependency_test.py

--
[...truncated 361.60 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05161708-niou-harness-88qq.c.apache-beam-testing.internal/10.128.0.2
 to 164.23.198.104.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 

[jira] [Commented] (BEAM-4318) Enforce ErrorProne analysis in Spark runner project

[Teng Peng (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478270#comment-16478270
 ] 

Teng Peng commented on BEAM-4318:
-

I am working on it.

> Enforce ErrorProne analysis in Spark runner project
> ---
>
> Key: BEAM-4318
> URL: https://issues.apache.org/jira/browse/BEAM-4318
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-spark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-spark:assemble}}
> # Fix each ErrorProne warning from the {{runners/spark}} project.
> # In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (BEAM-4198) Automatically infer JSON schema from Pubsub messages

[Teng Peng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teng Peng updated BEAM-4198:

Comment: was deleted

(was: Quiet interesting. I am working on it.)

> Automatically infer JSON schema from Pubsub messages
> 
>
> Key: BEAM-4198
> URL: https://issues.apache.org/jira/browse/BEAM-4198
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Anton Kedin
>Priority: Major
>
> JsonToRow transform allows JSON String->Row conversion but requires users to 
> know and specify the correct schema upfront. It would be great to be able to 
> infer the schema from a message sample.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #179

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] Fix error-prone and some static analysis warnings in

[daniel.o.programmer] [BEAM-4277] Adding unshaded jars used by Dataflow to poms.

[lukasz.gajowy] [BEAM-4140] Utilize beam_location perfkit flag

[swegner] Upgrade Byte Buddy and FindBugs annotations dependencies.

[swegner] Fix ErrorProne warnings

[swegner] Remove isTriggerSpecified from WindowingStategy equality checks.

[lcwik] Add window mapping transform.

[lcwik] Use the coder from the side input PCollection definition by removing the

[lcwik] Remove garbage collection from PValueCache

[Pablo] Fixing incorrect assert in dependency_test.py

--
[...truncated 207.22 KB...]
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.225.6.39:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #178

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #178

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102735
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 00:10
Start Date: 17/May/18 00:10
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188807218
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
 
 Review comment:
   Marking _download_file static has made it impossible to extend it and we 
will not be able to provide its implementation in the subclass without monkey 
patching it. 
   I will remove the _download_file from TestStager as its not getting 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102734
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 00:09
Start Date: 17/May/18 00:09
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188807218
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
 
 Review comment:
   Marking _download_file static has made it impossible to extend it and we 
will not be able to provide its implementation in the subclass without monkey 
patching it.


This 

Build failed in Jenkins: beam_PerformanceTests_JDBC #581

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] Fix error-prone and some static analysis warnings in

[daniel.o.programmer] [BEAM-4277] Adding unshaded jars used by Dataflow to poms.

[lukasz.gajowy] [BEAM-4140] Utilize beam_location perfkit flag

[swegner] Upgrade Byte Buddy and FindBugs annotations dependencies.

[swegner] Fix ErrorProne warnings

[swegner] Remove isTriggerSpecified from WindowingStategy equality checks.

[lcwik] Add window mapping transform.

[lcwik] Use the coder from the side input PCollection definition by removing the

[lcwik] Remove garbage collection from PValueCache

[Pablo] Fixing incorrect assert in dependency_test.py

--
[...truncated 96.25 KB...]
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 13,5,main]) completed. Took 0.016 secs.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 13,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 059121a02b8254a45ad34bf1a34d598f
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 8,5,main]) completed. Took 0.045 secs.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
3a101c48fba9616798805484a0cf8586
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 13,5,main]) completed. Took 0.064 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 13,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 13,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 13,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
f7ba51434c9610fdd0cc060d066fbfc5
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 13,5,main]) completed. Took 0.036 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 13,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
88db5cede7606017ee3177f5b0f7fc11
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 13,5,main]) completed. Took 0.062 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 13,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 13,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 13,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
36edd6d4112cc3a2198bdbc161376bfe
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 13,5,main]) completed. Took 0.029 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
13,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
5f430a896cd8b13ed1962fdb7793e4e9
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is 

[jira] [Work logged] (BEAM-4300) Duplicated code for ValidatesRunner / PostCommit tests in Python

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4300?focusedWorklogId=102733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102733
 ]

ASF GitHub Bot logged work on BEAM-4300:


Author: ASF GitHub Bot
Created on: 17/May/18 00:07
Start Date: 17/May/18 00:07
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #5372: [BEAM-4300] Fix 
ValidatesRunner tests in Python: run with same mechanism as ITs
URL: https://github.com/apache/beam/pull/5372
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy 
b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
index 392a93598c7..44611063633 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
@@ -20,9 +20,11 @@ import common_job_properties
 
 // This job runs the suite of Python ValidatesRunner tests against the
 // Dataflow runner.
-job('beam_PostCommit_Python_ValidatesRunner_Dataflow') {
+job('beam_PostCommit_Py_VR_Dataflow') {
   description('Runs Python ValidatesRunner suite on the Dataflow runner.')
 
+  previousNames('beam_PostCommit_Python_ValidatesRunner_Dataflow')
+
   // Set common parameters.
   common_job_properties.setTopLevelMainJobProperties(delegate)
 
@@ -35,8 +37,12 @@ job('beam_PostCommit_Python_ValidatesRunner_Dataflow') {
   'Google Cloud Dataflow Runner Python ValidatesRunner Tests',
   'Run Python Dataflow ValidatesRunner')
 
-  // Execute shell command to test Python SDK.
+  // Execute gradle task to test Python SDK.
   steps {
-shell('cd ' + common_job_properties.checkoutDir + ' && bash 
sdks/python/run_validatesrunner.sh')
+gradle {
+  rootBuildScriptDir(common_job_properties.checkoutDir)
+  tasks(':beam-sdks-python:validatesRunnerTests')
+  common_job_properties.setGradleSwitches(delegate)
+}
   }
 }
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py 
b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
index 289e07892ec..2307371d380 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
@@ -256,7 +256,7 @@ def _populate_requirements_cache(requirements_file, 
cache_dir):
   # the requirements file and will not download package dependencies.
   cmd_args = [
   _get_python_executable(), '-m', 'pip', 'download', '--dest', cache_dir,
-  '-r', requirements_file,
+  '-r', requirements_file, '--exists-action', 'i',
   # Download from PyPI source distributions.
   '--no-binary', ':all:']
   logging.info('Executing command: %s', cmd_args)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index df5a09823c3..aa7d736a0d0 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -187,11 +187,20 @@ task localWordCount(dependsOn: 'installGcpTest') {
   }
 }
 
+task postCommitVRTests(dependsOn: 'installGcpTest') {
+  doLast {
+exec {
+  executable 'sh'
+  args '-c', ". ${envdir}/bin/activate && ./run_postcommit.sh IT"
+}
+  }
+}
+
 task validatesRunnerTests(dependsOn: 'installGcpTest') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && ./run_postcommit.sh"
+  args '-c', ". ${envdir}/bin/activate && ./run_postcommit.sh 
ValidatesRunner"
 }
   }
 }
@@ -209,5 +218,5 @@ task postCommit() {
   dependsOn "preCommit"
   dependsOn "localWordCount"
   dependsOn "hdfsIntegrationTest"
-  dependsOn "validatesRunnerTests"
+  dependsOn "postCommitVRTests"
 }
diff --git a/sdks/python/run_postcommit.sh b/sdks/python/run_postcommit.sh
index 582c432b7f5..d26a1c95b9b 100755
--- a/sdks/python/run_postcommit.sh
+++ b/sdks/python/run_postcommit.sh
@@ -19,11 +19,17 @@
 # This script will be run by Jenkins as a post commit test. In order to run
 # locally make the following changes:
 #
-# LOCAL_PATH   -> Path of tox and virtualenv if you have them already 
installed.
 # GCS_LOCATION -> Temporary location to use for service tests.
 # PROJECT  -> Project name to use for service jobs.
 #
-# Execute from the root of the repository: sdks/python/run_postcommit.sh
+
+if [ -z "$1" ]; then
+  printf "Usage: \n$> ./run_postcommit.sh  [gcp_location] 
[gcp_project]"
+  printf "\n\ttest_type: ValidatesRunner or IT"
+  printf "\n\tgcp_location: A gs:// path to stage artifacts and output results"
+  printf "\n\tgcp_project: A GCP project to run Dataflow pipelines\n"
+  exit 1
+fi
 
 set -e
 set -v
@@ -31,20 +37,24 @@ set -v
 # Run tests on the 

[beam] branch master updated: [BEAM-4300] Fix ValidatesRunner tests in Python: run with same mechanism as ITs (#5372)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new d41df7b  [BEAM-4300] Fix ValidatesRunner tests in Python: run with 
same mechanism as ITs (#5372)
d41df7b is described below

commit d41df7b17f49569d37a7cdcc618408daba55
Author: Pablo 
AuthorDate: Wed May 16 17:07:36 2018 -0700

[BEAM-4300] Fix ValidatesRunner tests in Python: run with same mechanism as 
ITs (#5372)

* ValidatesRunner and ITs run with same mechanism in Python. Also receive 
test type and GCP arguments.
---
 ...stCommit_Python_ValidatesRunner_Dataflow.groovy | 12 +++-
 .../runners/dataflow/internal/dependency.py|  2 +-
 sdks/python/build.gradle   | 13 -
 sdks/python/run_postcommit.sh  | 21 +--
 sdks/python/run_validatesrunner.sh | 64 --
 5 files changed, 37 insertions(+), 75 deletions(-)

diff --git 
a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy 
b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
index 392a935..4461106 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
@@ -20,9 +20,11 @@ import common_job_properties
 
 // This job runs the suite of Python ValidatesRunner tests against the
 // Dataflow runner.
-job('beam_PostCommit_Python_ValidatesRunner_Dataflow') {
+job('beam_PostCommit_Py_VR_Dataflow') {
   description('Runs Python ValidatesRunner suite on the Dataflow runner.')
 
+  previousNames('beam_PostCommit_Python_ValidatesRunner_Dataflow')
+
   // Set common parameters.
   common_job_properties.setTopLevelMainJobProperties(delegate)
 
@@ -35,8 +37,12 @@ job('beam_PostCommit_Python_ValidatesRunner_Dataflow') {
   'Google Cloud Dataflow Runner Python ValidatesRunner Tests',
   'Run Python Dataflow ValidatesRunner')
 
-  // Execute shell command to test Python SDK.
+  // Execute gradle task to test Python SDK.
   steps {
-shell('cd ' + common_job_properties.checkoutDir + ' && bash 
sdks/python/run_validatesrunner.sh')
+gradle {
+  rootBuildScriptDir(common_job_properties.checkoutDir)
+  tasks(':beam-sdks-python:validatesRunnerTests')
+  common_job_properties.setGradleSwitches(delegate)
+}
   }
 }
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py 
b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
index 289e078..2307371 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
@@ -256,7 +256,7 @@ def _populate_requirements_cache(requirements_file, 
cache_dir):
   # the requirements file and will not download package dependencies.
   cmd_args = [
   _get_python_executable(), '-m', 'pip', 'download', '--dest', cache_dir,
-  '-r', requirements_file,
+  '-r', requirements_file, '--exists-action', 'i',
   # Download from PyPI source distributions.
   '--no-binary', ':all:']
   logging.info('Executing command: %s', cmd_args)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index df5a098..aa7d736 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -187,11 +187,20 @@ task localWordCount(dependsOn: 'installGcpTest') {
   }
 }
 
+task postCommitVRTests(dependsOn: 'installGcpTest') {
+  doLast {
+exec {
+  executable 'sh'
+  args '-c', ". ${envdir}/bin/activate && ./run_postcommit.sh IT"
+}
+  }
+}
+
 task validatesRunnerTests(dependsOn: 'installGcpTest') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && ./run_postcommit.sh"
+  args '-c', ". ${envdir}/bin/activate && ./run_postcommit.sh 
ValidatesRunner"
 }
   }
 }
@@ -209,5 +218,5 @@ task postCommit() {
   dependsOn "preCommit"
   dependsOn "localWordCount"
   dependsOn "hdfsIntegrationTest"
-  dependsOn "validatesRunnerTests"
+  dependsOn "postCommitVRTests"
 }
diff --git a/sdks/python/run_postcommit.sh b/sdks/python/run_postcommit.sh
index 582c432..d26a1c9 100755
--- a/sdks/python/run_postcommit.sh
+++ b/sdks/python/run_postcommit.sh
@@ -19,11 +19,17 @@
 # This script will be run by Jenkins as a post commit test. In order to run
 # locally make the following changes:
 #
-# LOCAL_PATH   -> Path of tox and virtualenv if you have them already 
installed.
 # GCS_LOCATION -> Temporary location to use for service tests.
 # PROJECT  -> Project name to use for service jobs.
 #
-# Execute from the root of the repository: sdks/python/run_postcommit.sh
+
+if [ -z "$1" ]; then
+  printf "Usage: \n$> ./run_postcommit.sh  [gcp_location] 
[gcp_project]"
+  printf "\n\ttest_type: ValidatesRunner or IT"
+  printf 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1673

[Apache Jenkins Server]

See 


Changes:

[github] [BEAM-4300] Fix ValidatesRunner tests in Python: run with same 
mechanism

--
Started by GitHub push by pabloem
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d41df7b17f49569d37a7cdcc618408daba55 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d41df7b17f49569d37a7cdcc618408daba55
Commit message: "[BEAM-4300] Fix ValidatesRunner tests in Python: run with same 
mechanism as ITs (#5372)"
 > git rev-list --no-walk c3c2ffdce7a4da2cf65f47ff8cb01f30f423170a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6917648266644930611.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user git...@alasdairhodge.co.uk

[jira] [Work logged] (BEAM-4199) [SQL] Add a DLQ support for Pubsub tables

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4199?focusedWorklogId=102732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102732
 ]

ASF GitHub Bot logged work on BEAM-4199:


Author: ASF GitHub Bot
Created on: 17/May/18 00:00
Start Date: 17/May/18 00:00
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5380: [BEAM-4199][SQL] Add 
support for DLQ to PubsubJsonTable
URL: https://github.com/apache/beam/pull/5380#issuecomment-389702508
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102732)
Time Spent: 2h 10m  (was: 2h)

> [SQL] Add a DLQ support for Pubsub tables
> -
>
> Key: BEAM-4199
> URL: https://issues.apache.org/jira/browse/BEAM-4199
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently we crash the pipeline if there's any error processing the message 
> from the pubsub, including if it has incorrect JSON format, like missing 
> fields etc.
> Correct solution would be for the user to specify a way to handle the errors, 
> and ideally point to a dead-letter-queue where Beam should send the messages 
> it could not process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102730
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:58
Start Date: 16/May/18 23:58
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188805559
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
 
 Review comment:
   yes, as I don't see other ways to spoof remote path.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102730)
Time Spent: 13h 50m  (was: 13h 40m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 13h 50m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102728
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:57
Start Date: 16/May/18 23:57
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188805479
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
+  with open(to_path, 'w') as f:
 f.write('Package content.')
-  return tarball_path
+  return to_path
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_download', 

[jira] [Commented] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[Alex Amato (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478254#comment-16478254
 ] 

Alex Amato commented on BEAM-3926:
--

Hi Etienne, I saw your PR for the metrics pusher 
([https://github.com/apache/beam/pull/4548/files])

Its true that the dataflow engine today handles pushing metrics to different 
places inside of its service.

Although, it might be appropriate to have metrics pusher push metrics to the 
dataflow service. It seems like an appropriate use of the layer there. However, 
perhaps your design assumes metrics are already aggregated before pushing. 
Dataflow expects workers to push metrics (local value for the worker) to the 
service, which aggregates them together.

Metrics pusher relies on a metrics container to exist on a cloud hosted engine 
to collected these already aggregated metrics? Then it pushes to where ever 
appropriate correct? If this is the case, then you're right that metrics pusher 
would need to be implemented in the Dataflow service, ideally accounting for 
the options/sinks you have specified.

Though, perhaps a design is possible to send the pre aggregated metrics back to 
a worker (by querying them from the service) and then use the same 
MetricsPusher.

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102724
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:54
Start Date: 16/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188805008
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
+  with open(to_path, 'w') as f:
 f.write('Package content.')
-  return tarball_path
+  return to_path
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_download', 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102723
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:53
Start Date: 16/May/18 23:53
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188804835
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -130,19 +99,43 @@ def _download_file(self, from_url, to_path):
   except Exception:
 logging.info('Failed to download Artifact from %s', from_url)
 raise
+elif from_url.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_url.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_url, to_path)
+  elif to_path.startswith('gs://'):
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102723)
Time Spent: 13h 20m  (was: 13h 10m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 13h 20m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102722
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:52
Start Date: 16/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188804700
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -78,39 +78,8 @@ class Stager(object):
   """Stager identifies and copies the appropriate artifacts to the staging
   location."""
 
-  def _copy_file(self, from_path, to_path):
-"""Copies a local file to a GCS file or vice versa."""
-logging.info('file copy from %s to %s.', from_path, to_path)
-if from_path.startswith('gs://') or to_path.startswith('gs://'):
-  from apache_beam.io.gcp import gcsio
-  if from_path.startswith('gs://') and to_path.startswith('gs://'):
-# Both files are GCS files so copy.
-gcsio.GcsIO().copy(from_path, to_path)
-  elif to_path.startswith('gs://'):
-# Only target is a GCS file, read local file and upload.
-with open(from_path, 'rb') as f:
-  with gcsio.GcsIO().open(to_path, mode='wb') as g:
-pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
-for chunk in iter(pfun, ''):
-  g.write(chunk)
-  else:
-# Source is a GCS file but target is local file.
-with gcsio.GcsIO().open(from_path, mode='rb') as g:
-  with open(to_path, 'wb') as f:
-pfun = functools.partial(g.read, gcsio.DEFAULT_READ_BUFFER_SIZE)
-for chunk in iter(pfun, ''):
-  f.write(chunk)
-else:
-  # Branch used only for unit tests and integration tests.
-  # In such environments GCS support is not available.
-  if not os.path.isdir(os.path.dirname(to_path)):
-logging.info(
-'Created folder (since we have not done yet, and any errors '
-'will follow): %s ', os.path.dirname(to_path))
-os.mkdir(os.path.dirname(to_path))
-  shutil.copyfile(from_path, to_path)
-
-  def _download_file(self, from_url, to_path):
+  @staticmethod
+  def _download_file(from_url, to_path):
 
 Review comment:
   As it also support http:// and gs:// I would prefer calling it from_url.
   Note: from can not be used.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102722)
Time Spent: 13h 10m  (was: 13h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102721
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:50
Start Date: 16/May/18 23:50
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188804467
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -746,17 +745,32 @@ def translate_scalar_counter_float(accumulator, 
metric_update_proto):
 metric_update_proto.floatingPoint = accumulator.value
 
 
-class _ParameterizedStager(Stager):
-  def __init__(self, stage_artifact):
-super(_ParameterizedStager, self).__init__()
-self.stage_artifact_method = stage_artifact
+class _LegacyDataflowStager(Stager):
+  # TODO(silviuc): Staged files should have a job specific prefix.
 
 Review comment:
   Removed it


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102721)
Time Spent: 13h  (was: 12h 50m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102720
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:50
Start Date: 16/May/18 23:50
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188804353
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -746,17 +745,32 @@ def translate_scalar_counter_float(accumulator, 
metric_update_proto):
 metric_update_proto.floatingPoint = accumulator.value
 
 
-class _ParameterizedStager(Stager):
-  def __init__(self, stage_artifact):
-super(_ParameterizedStager, self).__init__()
-self.stage_artifact_method = stage_artifact
+class _LegacyDataflowStager(Stager):
+  # TODO(silviuc): Staged files should have a job specific prefix.
+  # To prevent several jobs in the same project stomping on each other due to a
+  # shared staging location.
+  def __init__(self, dataflow_application_client):
+super(_LegacyDataflowStager, self).__init__()
+self.stage_artifact_method = dataflow_application_client._gcs_file_copy
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102720)
Time Spent: 12h 50m  (was: 12h 40m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4198) Automatically infer JSON schema from Pubsub messages

[Teng Peng (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478251#comment-16478251
 ] 

Teng Peng commented on BEAM-4198:
-

Quiet interesting. I am working on it.

> Automatically infer JSON schema from Pubsub messages
> 
>
> Key: BEAM-4198
> URL: https://issues.apache.org/jira/browse/BEAM-4198
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Anton Kedin
>Priority: Major
>
> JsonToRow transform allows JSON String->Row conversion but requires users to 
> know and specify the correct schema upfront. It would be great to be able to 
> infer the schema from a message sample.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102716
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188798067
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
+  with open(to_path, 'w') as f:
 f.write('Package content.')
-  return tarball_path
+  return to_path
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_download', 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102719
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188799057
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
+  with open(to_path, 'w') as f:
 f.write('Package content.')
-  return tarball_path
+  return to_path
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_download', 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102718
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188802535
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
+  return path.startswith('/tmp/remote/')
+
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+with mock.patch(
+'.'.join([
+self.__module__, TestStager.__name__,
+TestStager._is_remote_path.__name__
+]), is_remote_path):
+  self.assertEqual([sdk_filename],
+   self.stager.stage_job_resources(
+   options, staging_location=staging_dir))
 
   def test_sdk_location_http(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'http://storage.googleapis.com/my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-def file_download(_, to_folder):
-  tarball_path = os.path.join(to_folder, 'sdk-tarball')
-  with open(tarball_path, 'w') as f:
+def file_download(dummy_self, _, to_path):
 
 Review comment:
   Would it be simpler to implement desirable default behavior in TestStager()? 
Also why do we have dummy_self in the picture here?


This is an automated message from the 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102714
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188795976
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -746,17 +745,32 @@ def translate_scalar_counter_float(accumulator, 
metric_update_proto):
 metric_update_proto.floatingPoint = accumulator.value
 
 
-class _ParameterizedStager(Stager):
-  def __init__(self, stage_artifact):
-super(_ParameterizedStager, self).__init__()
-self.stage_artifact_method = stage_artifact
+class _LegacyDataflowStager(Stager):
+  # TODO(silviuc): Staged files should have a job specific prefix.
 
 Review comment:
   Can you please check if this TODO is still relevant and remove it if not?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102714)
Time Spent: 12h 10m  (was: 12h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102713
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188801882
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager_test.py
 ##
 @@ -420,134 +375,167 @@ def 
test_sdk_location_local_directory_not_present(self):
 sdk_location = 'nosuchdir'
 with self.assertRaises(RuntimeError) as cm:
   options = PipelineOptions()
-  options.view_as(GoogleCloudOptions).staging_location = staging_dir
   self.update_options(options)
   options.view_as(SetupOptions).sdk_location = sdk_location
 
-  dependency.stage_job_resources(options)
+  self.stager.stage_job_resources(options, staging_location=staging_dir)
 self.assertEqual(
 'The file "%s" cannot be found. Its '
 'location was specified by the --sdk_location command-line option.' %
-sdk_location,
-cm.exception.args[0])
+sdk_location, cm.exception.args[0])
 
-  def test_sdk_location_gcs_source_file(self):
+  def test_sdk_location_remote_source_file(self):
 staging_dir = self.make_temp_dir()
 sdk_location = 'gs://my-gcs-bucket/tarball.tar.gz'
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [names.DATAFLOW_SDK_TARBALL_FILE],
-  dependency.stage_job_resources(options))
-
-  def test_sdk_location_gcs_wheel_file(self):
+with mock.patch('.'.join([
+self.__module__, TestStager.__name__, 
TestStager.stage_artifact.__name__
+])):
+  with mock.patch('.'.join([
+  self.__module__, TestStager.__name__,
+  TestStager._download_file.__name__
+  ])):
+self.assertEqual([names.DATAFLOW_SDK_TARBALL_FILE],
+ self.stager.stage_job_resources(
+ options, staging_location=staging_dir))
+
+  def test_sdk_location_remote_wheel_file(self):
 staging_dir = self.make_temp_dir()
 sdk_filename = 'apache_beam-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl'
-sdk_location = 'gs://my-gcs-bucket/' + sdk_filename
+sdk_location = '/tmp/remote/my-bucket/' + sdk_filename
 
 options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
 self.update_options(options)
 options.view_as(SetupOptions).sdk_location = sdk_location
 
-with mock.patch('apache_beam.runners.dataflow.internal.'
-'dependency._dependency_file_copy'):
-  self.assertEqual(
-  [sdk_filename],
-  dependency.stage_job_resources(options))
+# We can not rely on actual remote file systems paths hence making
+# '/tmp/remote/' a new remote path.
+def is_remote_path(dummy_self, path):
 
 Review comment:
   Are you sure we need to do this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102713)
Time Spent: 12h 10m  (was: 12h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102717
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188796273
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -78,39 +78,8 @@ class Stager(object):
   """Stager identifies and copies the appropriate artifacts to the staging
   location."""
 
-  def _copy_file(self, from_path, to_path):
-"""Copies a local file to a GCS file or vice versa."""
-logging.info('file copy from %s to %s.', from_path, to_path)
-if from_path.startswith('gs://') or to_path.startswith('gs://'):
-  from apache_beam.io.gcp import gcsio
-  if from_path.startswith('gs://') and to_path.startswith('gs://'):
-# Both files are GCS files so copy.
-gcsio.GcsIO().copy(from_path, to_path)
-  elif to_path.startswith('gs://'):
-# Only target is a GCS file, read local file and upload.
-with open(from_path, 'rb') as f:
-  with gcsio.GcsIO().open(to_path, mode='wb') as g:
-pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
-for chunk in iter(pfun, ''):
-  g.write(chunk)
-  else:
-# Source is a GCS file but target is local file.
-with gcsio.GcsIO().open(from_path, mode='rb') as g:
-  with open(to_path, 'wb') as f:
-pfun = functools.partial(g.read, gcsio.DEFAULT_READ_BUFFER_SIZE)
-for chunk in iter(pfun, ''):
-  f.write(chunk)
-else:
-  # Branch used only for unit tests and integration tests.
-  # In such environments GCS support is not available.
-  if not os.path.isdir(os.path.dirname(to_path)):
-logging.info(
-'Created folder (since we have not done yet, and any errors '
-'will follow): %s ', os.path.dirname(to_path))
-os.mkdir(os.path.dirname(to_path))
-  shutil.copyfile(from_path, to_path)
-
-  def _download_file(self, from_url, to_path):
+  @staticmethod
+  def _download_file(from_url, to_path):
 
 Review comment:
   Consider calling first argument `from_path` or `from`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102717)
Time Spent: 12.5h  (was: 12h 20m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 12.5h
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102715
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188796621
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -130,19 +99,43 @@ def _download_file(self, from_url, to_path):
   except Exception:
 logging.info('Failed to download Artifact from %s', from_url)
 raise
+elif from_url.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_url.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_url, to_path)
+  elif to_path.startswith('gs://'):
 
 Review comment:
   Echoing my comment from previous interation: we only use this method for 
downloads into local folder. So the part that uploads to GCS would be dead 
code, let's remove it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102715)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=102712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102712
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 16/May/18 23:46
Start Date: 16/May/18 23:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188794959
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -746,17 +745,32 @@ def translate_scalar_counter_float(accumulator, 
metric_update_proto):
 metric_update_proto.floatingPoint = accumulator.value
 
 
-class _ParameterizedStager(Stager):
-  def __init__(self, stage_artifact):
-super(_ParameterizedStager, self).__init__()
-self.stage_artifact_method = stage_artifact
+class _LegacyDataflowStager(Stager):
+  # TODO(silviuc): Staged files should have a job specific prefix.
+  # To prevent several jobs in the same project stomping on each other due to a
+  # shared staging location.
+  def __init__(self, dataflow_application_client):
+super(_LegacyDataflowStager, self).__init__()
+self.stage_artifact_method = dataflow_application_client._gcs_file_copy
 
 Review comment:
   The composition would be more transparent if we store the reference to the 
dataflow client in the constructor, and call 
`self.dataflow_application_client._gcs_file_copy(...)` in stage_artifact.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102712)
Time Spent: 12h  (was: 11h 50m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3581) [SQL] Support for Non-ASCII chars is flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3581?focusedWorklogId=102711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102711
 ]

ASF GitHub Bot logged work on BEAM-3581:


Author: ASF GitHub Bot
Created on: 16/May/18 23:45
Start Date: 16/May/18 23:45
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5391: [BEAM-3581] [SQL] Use 
saffron.properties for charset
URL: https://github.com/apache/beam/pull/5391#issuecomment-389700168
 
 
   LGTM
   I think that correct jira is BEAM-3733


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102711)
Time Spent: 1h 20m  (was: 1h 10m)

> [SQL] Support for Non-ASCII chars is flaky
> --
>
> Key: BEAM-3581
> URL: https://issues.apache.org/jira/browse/BEAM-3581
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Beam SQL overrides default charset that Calcite uses and sets it to UTF16. It 
> is done via system properties.
> Problem is that we do this only when it hasn't been set yet. So if system 
> property has been set to ISO-8859-1 (Calcite's default), then test runs will 
> fail when trying to encode characters not supported in that encoding.
> Solution:
>  - because it's a system property, we don't want to force override it;
>  - for the same reason we cannot set it for a specific query execution;
>  - we can expose a static method on BeamSql to override these properties if 
> explicitly requested;
>  - affected tests will explicitly override it;
>  - otherwise behavior will stay unchanged and we will respect defaults and 
> user settings;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478240#comment-16478240
 ] 

Scott Wegner commented on BEAM-4328:


It's suspicious that the parallel build is broken for {{-PisRelease}}. When 
attempting a release build without specifying {{--no-parallel}}, I get all 
sorts of [weird zlib issues with during 
compilation|https://scans.gradle.com/s/pb5cbitnjpptw/failure?openFailures=WzBd=WzIse31d#top=0]:
 

{quote}
Error snapshotting jar [beam-runners-core-construction-java-2.5.0.jar]
> Unexpected end of ZLIB input stream

org.gradle.api.UncheckedIOException: Failed to capture snapshot of input files 
for task ':beam-runners-core-java:compileJava' property 'classpath' during 
up-to-date check.

Caused by: org.gradle.api.UncheckedIOException: Error snapshotting jar 
[beam-runners-core-construction-java-2.5.0.jar]

Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
{quote}

This makes me think there's some dependency declaration missing on the release 
build. I'll investigate this tomorrow further.

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478240#comment-16478240
 ] 

Scott Wegner edited comment on BEAM-4328 at 5/16/18 11:35 PM:
--

It's suspicious that the parallel build is broken for {{\-PisRelease}}. When 
attempting a release build without specifying {{--no-parallel}}, I get all 
sorts of [weird zlib issues with during 
compilation|https://scans.gradle.com/s/pb5cbitnjpptw/failure?openFailures=WzBd=WzIse31d#top=0]:
 

{quote}
Error snapshotting jar [beam-runners-core-construction-java-2.5.0.jar]
> Unexpected end of ZLIB input stream

org.gradle.api.UncheckedIOException: Failed to capture snapshot of input files 
for task ':beam-runners-core-java:compileJava' property 'classpath' during 
up-to-date check.

Caused by: org.gradle.api.UncheckedIOException: Error snapshotting jar 
[beam-runners-core-construction-java-2.5.0.jar]

Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
{quote}

This makes me think there's some dependency declaration missing on the release 
build. I'll investigate this tomorrow further.


was (Author: swegner):
It's suspicious that the parallel build is broken for {{-PisRelease}}. When 
attempting a release build without specifying {{--no-parallel}}, I get all 
sorts of [weird zlib issues with during 
compilation|https://scans.gradle.com/s/pb5cbitnjpptw/failure?openFailures=WzBd=WzIse31d#top=0]:
 

{quote}
Error snapshotting jar [beam-runners-core-construction-java-2.5.0.jar]
> Unexpected end of ZLIB input stream

org.gradle.api.UncheckedIOException: Failed to capture snapshot of input files 
for task ':beam-runners-core-java:compileJava' property 'classpath' during 
up-to-date check.

Caused by: org.gradle.api.UncheckedIOException: Error snapshotting jar 
[beam-runners-core-construction-java-2.5.0.jar]

Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
{quote}

This makes me think there's some dependency declaration missing on the release 
build. I'll investigate this tomorrow further.

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3581) [SQL] Support for Non-ASCII chars is flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3581?focusedWorklogId=102708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102708
 ]

ASF GitHub Bot logged work on BEAM-3581:


Author: ASF GitHub Bot
Created on: 16/May/18 23:35
Start Date: 16/May/18 23:35
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5391: [BEAM-3581] [SQL] 
Use saffron.properties for charset
URL: https://github.com/apache/beam/pull/5391#issuecomment-389698404
 
 
   @kennknowles @akedin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102708)
Time Spent: 1h 10m  (was: 1h)

> [SQL] Support for Non-ASCII chars is flaky
> --
>
> Key: BEAM-3581
> URL: https://issues.apache.org/jira/browse/BEAM-3581
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Beam SQL overrides default charset that Calcite uses and sets it to UTF16. It 
> is done via system properties.
> Problem is that we do this only when it hasn't been set yet. So if system 
> property has been set to ISO-8859-1 (Calcite's default), then test runs will 
> fail when trying to encode characters not supported in that encoding.
> Solution:
>  - because it's a system property, we don't want to force override it;
>  - for the same reason we cannot set it for a specific query execution;
>  - we can expose a static method on BeamSql to override these properties if 
> explicitly requested;
>  - affected tests will explicitly override it;
>  - otherwise behavior will stay unchanged and we will respect defaults and 
> user settings;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3581) [SQL] Support for Non-ASCII chars is flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3581?focusedWorklogId=102707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102707
 ]

ASF GitHub Bot logged work on BEAM-3581:


Author: ASF GitHub Bot
Created on: 16/May/18 23:34
Start Date: 16/May/18 23:34
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #5391: [BEAM-3581] 
[SQL] Use saffron.properties for charset
URL: https://github.com/apache/beam/pull/5391
 
 
   Hit the charset issues again for things that load calcite not via maven or 
gradle. This should fix it once and for all by adding the config file to the 
jar.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [X] Write a pull request description that is detailed enough to 
understand:
  - [X] What the pull request does
  - [X] Why it does it
  - [X] How it does it
  - [X] Why this approach
- [X] Each commit in the pull request should have a meaningful subject line 
and body.
- [X] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102707)
Time Spent: 1h  (was: 50m)

> [SQL] Support for Non-ASCII chars is flaky
> --
>
> Key: BEAM-3581
> URL: https://issues.apache.org/jira/browse/BEAM-3581
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Beam SQL overrides default charset that Calcite uses and sets it to UTF16. It 
> is done via system properties.
> Problem is that we do this only when it hasn't been set yet. So if system 
> property has been set to ISO-8859-1 (Calcite's default), then test runs will 
> fail when trying to encode characters not supported in that encoding.
> Solution:
>  - because it's a system property, we don't want to force override it;
>  - for the same reason we cannot set it for a specific query execution;
>  - we can expose a static method on BeamSql to override these properties if 
> explicitly requested;
>  - affected tests will explicitly override it;
>  - otherwise behavior will stay unchanged and we will respect defaults and 
> user settings;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4199) [SQL] Add a DLQ support for Pubsub tables

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4199?focusedWorklogId=102704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102704
 ]

ASF GitHub Bot logged work on BEAM-4199:


Author: ASF GitHub Bot
Created on: 16/May/18 23:20
Start Date: 16/May/18 23:20
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5380: [BEAM-4199][SQL] Add 
support for DLQ to PubsubJsonTable
URL: https://github.com/apache/beam/pull/5380#issuecomment-389695979
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102704)
Time Spent: 1h 50m  (was: 1h 40m)

> [SQL] Add a DLQ support for Pubsub tables
> -
>
> Key: BEAM-4199
> URL: https://issues.apache.org/jira/browse/BEAM-4199
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Currently we crash the pipeline if there's any error processing the message 
> from the pubsub, including if it has incorrect JSON format, like missing 
> fields etc.
> Correct solution would be for the user to specify a way to handle the errors, 
> and ideally point to a dead-letter-queue where Beam should send the messages 
> it could not process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4328:
---
Summary: gradle release build failed on task 
:beam-sdks-java-io-google-cloud-platform:test  (was: gradle release build 
failed to task :beam-sdks-java-io-google-cloud-platform:test)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4199) [SQL] Add a DLQ support for Pubsub tables

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4199?focusedWorklogId=102705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102705
 ]

ASF GitHub Bot logged work on BEAM-4199:


Author: ASF GitHub Bot
Created on: 16/May/18 23:20
Start Date: 16/May/18 23:20
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5380: [BEAM-4199][SQL] Add 
support for DLQ to PubsubJsonTable
URL: https://github.com/apache/beam/pull/5380#issuecomment-389686277
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102705)
Time Spent: 2h  (was: 1h 50m)

> [SQL] Add a DLQ support for Pubsub tables
> -
>
> Key: BEAM-4199
> URL: https://issues.apache.org/jira/browse/BEAM-4199
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Currently we crash the pipeline if there's any error processing the message 
> from the pubsub, including if it has incorrect JSON format, like missing 
> fields etc.
> Correct solution would be for the user to specify a way to handle the errors, 
> and ideally point to a dead-letter-queue where Beam should send the messages 
> it could not process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4328) gradle release build failed to task :beam-sdks-java-io-google-cloud-platform:test

[Boyuan Zhang (JIRA)]

Boyuan Zhang created BEAM-4328:
--

 Summary: gradle release build failed to task 
:beam-sdks-java-io-google-cloud-platform:test
 Key: BEAM-4328
 URL: https://issues.apache.org/jira/browse/BEAM-4328
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Boyuan Zhang
Assignee: Luke Cwik


Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
build 

--no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.

Gradle scan: 
[https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]

 

Running ./gradlew clean && ./gradlew  
:beam-sdks-java-io-google-cloud-platform:test -PisRelease

--no-parallel succeed: 
[https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4327) Enforce ErrorProne analysis in the java harness project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4327:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-harness}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-harness:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/harness}} project.
# In {{sdks/java/harness/build.gradle}}, add {{failOnWarning: true}} to the 
call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
# In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in the java harness project
> ---
>
> Key: BEAM-4327
> URL: https://issues.apache.org/jira/browse/BEAM-4327
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-harness}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-harness:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/harness}} project.
> # In {{sdks/java/harness/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions 

[jira] [Created] (BEAM-4327) Enforce ErrorProne analysis in the java harness project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4327:
--

 Summary: Enforce ErrorProne analysis in the java harness project
 Key: BEAM-4327
 URL: https://issues.apache.org/jira/browse/BEAM-4327
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-harness
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
# In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4326) Enforce ErrorProne analysis in the fn-execution project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4326:
---
Component/s: (was: dsl-sql)
 sdk-java-harness

> Enforce ErrorProne analysis in the fn-execution project
> ---
>
> Key: BEAM-4326
> URL: https://issues.apache.org/jira/browse/BEAM-4326
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
> # In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4326) Enforce ErrorProne analysis in the fn-execution project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4326:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
# In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
# In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in the fn-execution project
> ---
>
> Key: BEAM-4326
> URL: https://issues.apache.org/jira/browse/BEAM-4326
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
> # In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> 

[jira] [Created] (BEAM-4326) Enforce ErrorProne analysis in the fn-execution project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4326:
--

 Summary: Enforce ErrorProne analysis in the fn-execution project
 Key: BEAM-4326
 URL: https://issues.apache.org/jira/browse/BEAM-4326
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
# In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4325) Enforce ErrorProne analysis in the SQL project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4325:
---
Summary: Enforce ErrorProne analysis in the SQL project  (was: Enforce 
ErrorProne analysis in sorter SQL project)

> Enforce ErrorProne analysis in the SQL project
> --
>
> Key: BEAM-4325
> URL: https://issues.apache.org/jira/browse/BEAM-4325
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
> # In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} 
> to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4325) Enforce ErrorProne analysis in sorter SQL project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4325:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
# In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sorter}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sorter:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sorter}} project.
# In {{sdks/java/extensions/sorter/build.gradle}}, add {{failOnWarning: true}} 
to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in sorter SQL project
> -
>
> Key: BEAM-4325
> URL: https://issues.apache.org/jira/browse/BEAM-4325
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sql}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sql:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sql}} project.
> # In {{sdks/java/extensions/sql/build.gradle}}, add {{failOnWarning: true}} 
> to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> 

[jira] [Created] (BEAM-4325) Enforce ErrorProne analysis in sorter SQL project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4325:
--

 Summary: Enforce ErrorProne analysis in sorter SQL project
 Key: BEAM-4325
 URL: https://issues.apache.org/jira/browse/BEAM-4325
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sorter}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sorter:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sorter}} project.
# In {{sdks/java/extensions/sorter/build.gradle}}, add {{failOnWarning: true}} 
to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4325) Enforce ErrorProne analysis in sorter SQL project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4325:
---
Component/s: (was: sdk-java-core)
 dsl-sql

> Enforce ErrorProne analysis in sorter SQL project
> -
>
> Key: BEAM-4325
> URL: https://issues.apache.org/jira/browse/BEAM-4325
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sorter}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sorter:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sorter}} 
> project.
> # In {{sdks/java/extensions/sorter/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4324) Enforce ErrorProne analysis in sorter extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4324:
---
Component/s: (was: sdk-java-sketching)
 sdk-java-core

> Enforce ErrorProne analysis in sorter extensions project
> 
>
> Key: BEAM-4324
> URL: https://issues.apache.org/jira/browse/BEAM-4324
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
> project.
> # In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4324) Enforce ErrorProne analysis in sorter extensions project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4324:
--

 Summary: Enforce ErrorProne analysis in sorter extensions project
 Key: BEAM-4324
 URL: https://issues.apache.org/jira/browse/BEAM-4324
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-sketching
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
project.
# In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4324) Enforce ErrorProne analysis in sorter extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4324:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sorter}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sorter:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sorter}} project.
# In {{sdks/java/extensions/sorter/build.gradle}}, add {{failOnWarning: true}} 
to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
project.
# In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in sorter extensions project
> 
>
> Key: BEAM-4324
> URL: https://issues.apache.org/jira/browse/BEAM-4324
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sorter}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sorter:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sorter}} 
> project.
> # In {{sdks/java/extensions/sorter/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter 

[jira] [Updated] (BEAM-4323) Enforce ErrorProne analysis in sketching extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4323:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
project.
# In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-protobuf}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-protobuf:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/protobuf}} 
project.
# In {{sdks/java/extensions/protobuf/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in sketching extensions project
> ---
>
> Key: BEAM-4323
> URL: https://issues.apache.org/jira/browse/BEAM-4323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-sketching
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-sketching}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-sketching:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/sketching}} 
> project.
> # In {{sdks/java/extensions/sketching/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> 

[jira] [Created] (BEAM-4323) Enforce ErrorProne analysis in sketching extensions project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4323:
--

 Summary: Enforce ErrorProne analysis in sketching extensions 
project
 Key: BEAM-4323
 URL: https://issues.apache.org/jira/browse/BEAM-4323
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-protobuf}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-protobuf:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/protobuf}} 
project.
# In {{sdks/java/extensions/protobuf/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4323) Enforce ErrorProne analysis in sketching extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4323:
---
Component/s: (was: sdk-java-core)
 sdk-java-sketching

> Enforce ErrorProne analysis in sketching extensions project
> ---
>
> Key: BEAM-4323
> URL: https://issues.apache.org/jira/browse/BEAM-4323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-sketching
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-protobuf}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-protobuf:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/protobuf}} 
> project.
> # In {{sdks/java/extensions/protobuf/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4322) Enforce ErrorProne analysis in protobuf extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4322:
---
Component/s: (was: sdk-java-join-library)
 sdk-java-core

> Enforce ErrorProne analysis in protobuf extensions project
> --
>
> Key: BEAM-4322
> URL: https://issues.apache.org/jira/browse/BEAM-4322
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-join-library}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-join-library:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/join-library}} 
> project.
> # In {{sdks/java/extensions/join-library/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4322) Enforce ErrorProne analysis in protobuf extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4322:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-protobuf}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-protobuf:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/protobuf}} 
project.
# In {{sdks/java/extensions/protobuf/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-join-library}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-join-library:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/join-library}} 
project.
# In {{sdks/java/extensions/join-library/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in protobuf extensions project
> --
>
> Key: BEAM-4322
> URL: https://issues.apache.org/jira/browse/BEAM-4322
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-protobuf}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-protobuf:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/protobuf}} 
> project.
> # In {{sdks/java/extensions/protobuf/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> 

[jira] [Created] (BEAM-4322) Enforce ErrorProne analysis in protobuf extensions project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4322:
--

 Summary: Enforce ErrorProne analysis in protobuf extensions project
 Key: BEAM-4322
 URL: https://issues.apache.org/jira/browse/BEAM-4322
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-join-library
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-join-library}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-join-library:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/join-library}} 
project.
# In {{sdks/java/extensions/join-library/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4321) Enforce ErrorProne analysis in join-library extensions project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4321:
--

 Summary: Enforce ErrorProne analysis in join-library extensions 
project
 Key: BEAM-4321
 URL: https://issues.apache.org/jira/browse/BEAM-4321
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-json-jackson}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-json-jackson:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/jackson}} project.
# In {{sdks/java/extensions/jackson/build.gradle}}, add {{failOnWarning: true}} 
to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4321) Enforce ErrorProne analysis in join-library extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4321:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-join-library}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-join-library:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/join-library}} 
project.
# In {{sdks/java/extensions/join-library/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-json-jackson}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-json-jackson:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/jackson}} project.
# In {{sdks/java/extensions/jackson/build.gradle}}, add {{failOnWarning: true}} 
to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in join-library extensions project
> --
>
> Key: BEAM-4321
> URL: https://issues.apache.org/jira/browse/BEAM-4321
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-join-library
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-join-library}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-join-library:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/join-library}} 
> project.
> # In {{sdks/java/extensions/join-library/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> 

[jira] [Updated] (BEAM-4321) Enforce ErrorProne analysis in join-library extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4321:
---
Component/s: (was: sdk-java-core)
 sdk-java-join-library

> Enforce ErrorProne analysis in join-library extensions project
> --
>
> Key: BEAM-4321
> URL: https://issues.apache.org/jira/browse/BEAM-4321
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-join-library
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-json-jackson}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-json-jackson:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/jackson}} 
> project.
> # In {{sdks/java/extensions/jackson/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4320) Enforce ErrorProne analysis in jackson extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4320:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-extensions-json-jackson}}. Additional context discussed on the 
[dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-extensions-json-jackson:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/extensions/jackson}} project.
# In {{sdks/java/extensions/jackson/build.gradle}}, add {{failOnWarning: true}} 
to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-build-tools}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-build-tools:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/build-tools}} project.
# In {{sdks/java/build-tools/build.gradle}}, add {{failOnWarning: true}} to the 
call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in jackson extensions project
> -
>
> Key: BEAM-4320
> URL: https://issues.apache.org/jira/browse/BEAM-4320
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-extensions-json-jackson}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-extensions-json-jackson:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/extensions/jackson}} 
> project.
> # In {{sdks/java/extensions/jackson/build.gradle}}, add {{failOnWarning: 
> true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is 

[jira] [Updated] (BEAM-4319) Enforce ErrorProne analysis in build-tools project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4319:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-build-tools}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-build-tools:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/build-tools}} project.
# In {{sdks/java/build-tools/build.gradle}}, add {{failOnWarning: true}} to the 
call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-spark}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-spark:assemble}}
# Fix each ErrorProne warning from the {{runners/spark}} project.
# In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in build-tools project
> --
>
> Key: BEAM-4319
> URL: https://issues.apache.org/jira/browse/BEAM-4319
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-build-tools}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-build-tools:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/build-tools}} project.
> # In {{sdks/java/build-tools/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:

[jira] [Created] (BEAM-4320) Enforce ErrorProne analysis in jackson extensions project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4320:
--

 Summary: Enforce ErrorProne analysis in jackson extensions project
 Key: BEAM-4320
 URL: https://issues.apache.org/jira/browse/BEAM-4320
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-sdks-java-build-tools}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-sdks-java-build-tools:assemble}}
# Fix each ErrorProne warning from the {{sdks/java/build-tools}} project.
# In {{sdks/java/build-tools/build.gradle}}, add {{failOnWarning: true}} to the 
call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4320) Enforce ErrorProne analysis in jackson extensions project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4320:
---
Component/s: (was: build-system)
 sdk-java-core

> Enforce ErrorProne analysis in jackson extensions project
> -
>
> Key: BEAM-4320
> URL: https://issues.apache.org/jira/browse/BEAM-4320
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-build-tools}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-build-tools:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/build-tools}} project.
> # In {{sdks/java/build-tools/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4319) Enforce ErrorProne analysis in build-tools project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4319:
---
Component/s: (was: runner-spark)
 build-system

> Enforce ErrorProne analysis in build-tools project
> --
>
> Key: BEAM-4319
> URL: https://issues.apache.org/jira/browse/BEAM-4319
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-spark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-spark:assemble}}
> # Fix each ErrorProne warning from the {{runners/spark}} project.
> # In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4318) Enforce ErrorProne analysis in Spark runner project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4318:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-spark}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-spark:assemble}}
# Fix each ErrorProne warning from the {{runners/spark}} project.
# In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
# Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
project.
# In {{runners/google-cloud-dataflow-java/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in Spark runner project
> ---
>
> Key: BEAM-4318
> URL: https://issues.apache.org/jira/browse/BEAM-4318
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-spark}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-spark:assemble}}
> # Fix each ErrorProne warning from the {{runners/spark}} project.
> # In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with 

[jira] [Created] (BEAM-4319) Enforce ErrorProne analysis in build-tools project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4319:
--

 Summary: Enforce ErrorProne analysis in build-tools project
 Key: BEAM-4319
 URL: https://issues.apache.org/jira/browse/BEAM-4319
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-spark}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-spark:assemble}}
# Fix each ErrorProne warning from the {{runners/spark}} project.
# In {{runners/spark/build.gradle}}, add {{failOnWarning: true}} to the call 
the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4318) Enforce ErrorProne analysis in Spark runner project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4318:
---
Component/s: (was: runner-dataflow)
 runner-spark

> Enforce ErrorProne analysis in Spark runner project
> ---
>
> Key: BEAM-4318
> URL: https://issues.apache.org/jira/browse/BEAM-4318
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4318) Enforce ErrorProne analysis in Spark runner project

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4318:
--

 Summary: Enforce ErrorProne analysis in Spark runner project
 Key: BEAM-4318
 URL: https://issues.apache.org/jira/browse/BEAM-4318
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Reporter: Scott Wegner


Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
# Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
project.
# In {{runners/google-cloud-dataflow-java/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4314) Enforce ErrorProne analysis in fn-execution project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4314:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-java-fn-execution}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-java-fn-execution:assemble}}
# Fix each ErrorProne warning from the {{runners/java-fn-execution}} project.
# In {{runners/java-fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-runners-java-fn-execution}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-runners-java-fn-execution}}
# Fix each ErrorProne warning from the {{runners/java-fn-execution}} project.
# In {{runners/java-fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in fn-execution project
> ---
>
> Key: BEAM-4314
> URL: https://issues.apache.org/jira/browse/BEAM-4314
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-java-fn-execution}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-java-fn-execution:assemble}}
> # Fix each ErrorProne warning from the {{runners/java-fn-execution}} project.
> # In {{runners/java-fn-execution/build.gradle}}, add {{failOnWarning: true}} 
> to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> 

[jira] [Updated] (BEAM-4315) Enforce ErrorProne analysis in local-artifact-service project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4315:
---
Description: 
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-local-artifact-service-java}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-local-artifact-service-java:assemble}}
# Fix each ErrorProne warning from the {{runners/local-artifact-service-java}} 
project.
# In {{runners/local-artifact-service-java/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com

  was:
Java ErrorProne static analysis was [recently 
enabled|https://github.com/apache/beam/pull/5161] in the Gradle build process, 
but only as warnings. ErrorProne errors are generally useful and easy to fix. 
Some work was done to [make sdks-java-core 
ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add enforcement. 
This task is clean ErrorProne warnings and add enforcement in 
{{beam-local-artifact-service-java}}. Additional context discussed on the [dev 
list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].

Fixing this issue will involve:

# Follow instructions in the [Contribution 
Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
environment.
# Run the following command to compile and run ErrorProne analysis on the 
project: {{./gradlew :beam-local-artifact-service-java}}
# Fix each ErrorProne warning from the {{runners/local-artifact-service-java}} 
project.
# In {{runners/local-artifact-service-java/build.gradle}}, add {{failOnWarning: 
true}} to the call the {{applyJavaNature()}} 
([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).


This starter issue is sponsored by [~swegner]. Feel free to [reach 
out|https://beam.apache.org/community/contact-us/] with questions or code 
review:

* JIRA: [~swegner]
* GitHub: [@swegner|https://github.com/swegner]
* Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
* Email: swegner at google dot com


> Enforce ErrorProne analysis in local-artifact-service project
> -
>
> Key: BEAM-4315
> URL: https://issues.apache.org/jira/browse/BEAM-4315
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-local-artifact-service-java}}. Additional context discussed on the 
> [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-local-artifact-service-java:assemble}}
> # Fix each ErrorProne warning from the 
> {{runners/local-artifact-service-java}} project.
> # In {{runners/local-artifact-service-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> 

[jira] [Updated] (BEAM-4308) Enforce ErrorProne analysis in runners-core project

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4308:
---
Summary: Enforce ErrorProne analysis in runners-core project  (was: CLONE - 
Enforce ErrorProne analysis in runners-core project)

> Enforce ErrorProne analysis in runners-core project
> ---
>
> Key: BEAM-4308
> URL: https://issues.apache.org/jira/browse/BEAM-4308
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-core-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-core-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/core}} project.
> # In {{runners/core/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)