Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #353

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a Portable TransformEvaluatorRegistry

[tgroh] Consistent Parameter Order in Evaluator Factories

--
[...truncated 18.13 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 26, 2018 5:42:34 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 26, 2018 5:42:34 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0526054230-7d0bc86d/output/results/staging/
May 26, 2018 5:42:34 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash -HMU01OF9AT0mGPlYeRwLQ> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0526054230-7d0bc86d/output/results/staging/pipeline--HMU01OF9AT0mGPlYeRwLQ.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 26, 2018 5:42:35 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_22_42_34-10417657797012504459?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-25_22_42_34-10417657797012504459

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 26, 2018 5:42:35 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_22_42_34-10417657797012504459
May 26, 2018 5:42:35 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-25_22_42_34-10417657797012504459 with 1 
expected assertions.
May 26, 2018 5:42:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-26T05:42:34.931Z: Autoscaling is enabled for job 
2018-05-25_22_42_34-10417657797012504459. The number of 

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=106116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106116
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 26/May/18 03:52
Start Date: 26/May/18 03:52
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5490: [BEAM-4077] 
Refactor schemas and fields for simplicity and clarity
URL: https://github.com/apache/beam/pull/5490#issuecomment-392234002
 
 
   OK, I rebuilt the commit history and removed RowSqlTypes. I kept some of it 
in CalciteUtils. I think we need to have SQL types independent of Beam types 
unless Beam adopts the standard types. But I think that doesn't fit perfectly - 
SQL has many types that are like compound types in Beam.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106116)
Time Spent: 1h 10m  (was: 1h)

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #352

[Apache Jenkins Server]

See 


Changes:

[katarzyna.kucharczyk] Replaced created datetime to jenkins build id in 
kubernetes namespace.

--
[...truncated 18.08 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 26, 2018 3:24:19 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 26, 2018 3:24:19 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0526032416-f602189/output/results/staging/
May 26, 2018 3:24:19 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash oPsyAR6GoAjy_N1kiYrKHQ> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0526032416-f602189/output/results/staging/pipeline-oPsyAR6GoAjy_N1kiYrKHQ.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 26, 2018 3:24:21 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_20_24_20-2030771955770548121?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-25_20_24_20-2030771955770548121

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 26, 2018 3:24:21 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_20_24_20-2030771955770548121
May 26, 2018 3:24:21 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-25_20_24_20-2030771955770548121 with 1 
expected assertions.
May 26, 2018 3:24:34 AM 

[beam] 01/01: Merge pull request #5477: Add a Portable TransformEvaluatorRegistry

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 707fb29ffd9f6aaad8b736792ff1b476f20de28b
Merge: d2394ca 80f8a23
Author: Thomas Groh 
AuthorDate: Fri May 25 18:17:45 2018 -0700

Merge pull request #5477: Add a Portable TransformEvaluatorRegistry

 .../direct/portable/FlattenEvaluatorFactory.java   |  2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java |  2 +-
 .../portable/GroupByKeyOnlyEvaluatorFactory.java   |  4 +--
 .../direct/portable/ImpulseEvaluatorFactory.java   |  2 +-
 .../portable/TransformEvaluatorRegistry.java   | 33 ++
 .../portable/FlattenEvaluatorFactoryTest.java  |  2 +-
 .../GroupByKeyOnlyEvaluatorFactoryTest.java|  2 +-
 .../portable/ImpulseEvaluatorFactoryTest.java  |  2 +-
 8 files changed, 41 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (d2394ca -> 707fb29)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d2394ca  Merge pull request #5452: [BEAM-4371] Namespace shares the 
same name between Jenkins Performance test jobs
 add f571cd5  Add a Portable TransformEvaluatorRegistry
 add 80f8a23  Consistent Parameter Order in Evaluator Factories
 new 707fb29  Merge pull request #5477: Add a Portable 
TransformEvaluatorRegistry

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../direct/portable/FlattenEvaluatorFactory.java   |  2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java |  2 +-
 .../portable/GroupByKeyOnlyEvaluatorFactory.java   |  4 +--
 .../direct/portable/ImpulseEvaluatorFactory.java   |  2 +-
 .../portable/TransformEvaluatorRegistry.java   | 33 ++
 .../portable/FlattenEvaluatorFactoryTest.java  |  2 +-
 .../GroupByKeyOnlyEvaluatorFactoryTest.java|  2 +-
 .../portable/ImpulseEvaluatorFactoryTest.java  |  2 +-
 8 files changed, 41 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #351

[Apache Jenkins Server]

See 


Changes:

[tgroh] Make ImmutableExecutableStage Public

[lcwik] [BEAM-4411] Update Jackson version to latest to pick up security fixes.

[kenn] Implement UNNEST operator for SQL

--
[...truncated 18.11 MB...]
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 26, 2018 1:09:22 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 26, 2018 1:09:22 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0526010918-acc5122a/output/results/staging/
May 26, 2018 1:09:22 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash Qn4l5fslnPO696chCQHfsw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0526010918-acc5122a/output/results/staging/pipeline-Qn4l5fslnPO696chCQHfsw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 26, 2018 1:09:23 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_18_09_22-18215969196897215788?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-25_18_09_22-18215969196897215788

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 26, 2018 1:09:23 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_18_09_22-18215969196897215788
May 26, 2018 1:09:23 AM 

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106098
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191034888
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
 
 Review comment:
   s/attempts/maxAttempts


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106098)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106099
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191035404
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
+  throws InterruptedException {
+ArrayList errorList = new ArrayList();
+int countAttempts = 0;
+
+while (countAttempts < attempts) {
+  try {
+function.run();
+return;
+  } catch (Exception e) {
+LOG.warn(
+"Attempt #{} of {} threw exception: {}", countAttempts + 1, 
attempts, e.getMessage());
+errorList.add(e);
+countAttempts++;
+if (countAttempts == attempts) {
+  throw RetryException.composeErrors(errorList);
 
 Review comment:
   I think just throwing the last exception will be cleaner (and log previous 
once). We can also get rid of "RetryException".


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106099)
Time Spent: 4h 20m  (was: 4h 10m)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106095=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106095
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191035244
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
 
 Review comment:
   Can we pass a lambda instead of introducing a new interface RetryFunction ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106095)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106094
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191035269
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
 
 Review comment:
   Set default value for maxAttemps and delay ?
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106094)
Time Spent: 4h  (was: 3h 50m)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106096
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191034899
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
+  throws InterruptedException {
+ArrayList errorList = new ArrayList();
+int countAttempts = 0;
 
 Review comment:
   s/countAttempts/attempts


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106096)
Time Spent: 4h 10m  (was: 4h)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106093
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191035287
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
+  throws InterruptedException {
+ArrayList errorList = new ArrayList();
+int countAttempts = 0;
+
+while (countAttempts < attempts) {
+  try {
+function.run();
+return;
+  } catch (Exception e) {
+LOG.warn(
+"Attempt #{} of {} threw exception: {}", countAttempts + 1, 
attempts, e.getMessage());
+errorList.add(e);
+countAttempts++;
+if (countAttempts == attempts) {
+  throw RetryException.composeErrors(errorList);
+} else {
+  Thread.sleep(delay);
 
 Review comment:
   We should use exponential backoff in case the service is overloaded.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106093)
Time Spent: 3h 50m  (was: 3h 40m)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3949) IOIT's setup() and teardown() db connection attempt sometimes fail resulting in test flakiness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3949?focusedWorklogId=106097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106097
 ]

ASF GitHub Bot logged work on BEAM-3949:


Author: ASF GitHub Bot
Created on: 26/May/18 01:08
Start Date: 26/May/18 01:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5434: [BEAM-3949] IOIT's setup() and teardown() db connection attempt 
sometimes fail resulting in test flakiness
URL: https://github.com/apache/beam/pull/5434#discussion_r191035038
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
 ##
 @@ -36,4 +42,55 @@ public static String getHashForRecordCount(int recordCount, 
Map
 }
 return hash;
   }
+
+  public static void retry(RetryFunction function, int attempts, long delay)
 
 Review comment:
   executeWithRetry ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106097)
Time Spent: 4h 10m  (was: 4h)

> IOIT's setup() and teardown() db connection attempt sometimes fail resulting 
> in test flakiness
> --
>
> Key: BEAM-3949
> URL: https://issues.apache.org/jira/browse/BEAM-3949
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> setup() and teardown() methods sometimes have trouble connecting database in 
> Performance tests. It results in test flakiness. 
> Example logs: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/65/console]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106089
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 26/May/18 00:57
Start Date: 26/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191034939
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
 ##
 @@ -6,31 +6,33 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
  * http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
  */
-package org.apache.beam.runners.reference.testing;
 
-import io.grpc.ManagedChannel;
-import io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+package org.apache.beam.fn.harness.control;
 
-/**
- * A {@link org.apache.beam.sdk.fn.channel.ManagedChannelFactory} that uses 
in-process channels.
- *
- * The channel builder uses {@link ApiServiceDescriptor#getUrl()} as the 
unique in-process name.
- */
-public class InProcessManagedChannelFactory extends ManagedChannelFactory {
+import io.grpc.ClientInterceptor;
+import io.grpc.Metadata;
+import io.grpc.Metadata.Key;
+import io.grpc.stub.MetadataUtils;
+
+/** A {@link ClientInterceptor} that attaches a provided SDK Harness ID to 
outgoing messages. */
+public class AddHarnessIdInterceptor {
+  private static final Key ID_KEY = Key.of("worker_id", 
Metadata.ASCII_STRING_MARSHALLER);
 
-  @Override
-  public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
-return 
InProcessChannelBuilder.forName(apiServiceDescriptor.getUrl()).build();
+  public static ClientInterceptor create(String harnessId) {
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106089)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106090
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 26/May/18 00:57
Start Date: 26/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191034956
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
 ##
 @@ -6,31 +6,33 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
  * http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
  */
-package org.apache.beam.runners.reference.testing;
 
-import io.grpc.ManagedChannel;
-import io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+package org.apache.beam.fn.harness.control;
 
-/**
- * A {@link org.apache.beam.sdk.fn.channel.ManagedChannelFactory} that uses 
in-process channels.
- *
- * The channel builder uses {@link ApiServiceDescriptor#getUrl()} as the 
unique in-process name.
- */
-public class InProcessManagedChannelFactory extends ManagedChannelFactory {
+import io.grpc.ClientInterceptor;
+import io.grpc.Metadata;
+import io.grpc.Metadata.Key;
+import io.grpc.stub.MetadataUtils;
+
+/** A {@link ClientInterceptor} that attaches a provided SDK Harness ID to 
outgoing messages. */
+public class AddHarnessIdInterceptor {
 
 Review comment:
   I find this much more readable as a factory method than as an inline key and 
`MetadataUtils` call - it tells us exactly the thing it does.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106090)
Time Spent: 2h 50m  (was: 2h 40m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106086
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 26/May/18 00:57
Start Date: 26/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191034902
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
 ##
 @@ -115,43 +124,46 @@ public static void main(PipelineOptions options,
   }
 
   public static void main(
+  String id,
   PipelineOptions options,
   Endpoints.ApiServiceDescriptor loggingApiServiceDescriptor,
   Endpoints.ApiServiceDescriptor controlApiServiceDescriptor,
   ManagedChannelFactory channelFactory,
   StreamObserverFactory streamObserverFactory) {
 IdGenerator idGenerator = IdGenerators.decrementingLongs();
-try (BeamFnLoggingClient logging = new BeamFnLoggingClient(
-options,
-loggingApiServiceDescriptor,
-channelFactory::forDescriptor)) {
+try (BeamFnLoggingClient logging =
 
 Review comment:
   No. This is done to close the client automatically at the completion of the 
`try` block, and the client must exist to send intercepted LOG messages to the 
logging server.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106086)
Time Spent: 2h 20m  (was: 2h 10m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106088
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 26/May/18 00:57
Start Date: 26/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191034893
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
 ##
 @@ -115,43 +124,46 @@ public static void main(PipelineOptions options,
   }
 
   public static void main(
+  String id,
   PipelineOptions options,
   Endpoints.ApiServiceDescriptor loggingApiServiceDescriptor,
   Endpoints.ApiServiceDescriptor controlApiServiceDescriptor,
   ManagedChannelFactory channelFactory,
   StreamObserverFactory streamObserverFactory) {
 IdGenerator idGenerator = IdGenerators.decrementingLongs();
-try (BeamFnLoggingClient logging = new BeamFnLoggingClient(
-options,
-loggingApiServiceDescriptor,
-channelFactory::forDescriptor)) {
+try (BeamFnLoggingClient logging =
+new BeamFnLoggingClient(
+options, loggingApiServiceDescriptor, 
channelFactory::forDescriptor)) {
 
   LOG.info("Fn Harness started");
-  EnumMap Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106085
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 26/May/18 00:57
Start Date: 26/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191034868
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
 ##
 @@ -80,23 +83,28 @@ public static void main(String[] args) throws Exception {
 System.out.format("Control location %s%n", 
System.getenv(CONTROL_API_SERVICE_DESCRIPTOR));
 System.out.format("Pipeline options %s%n", 
System.getenv(PIPELINE_OPTIONS));
 
-ObjectMapper objectMapper = new ObjectMapper().registerModules(
-ObjectMapper.findModules(ReflectHelpers.findClassLoader()));
-PipelineOptions options = objectMapper.readValue(
-System.getenv(PIPELINE_OPTIONS), PipelineOptions.class);
+String id = System.getenv(HARNESS_ID);
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106085)
Time Spent: 2h 10m  (was: 2h)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106087
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 26/May/18 00:57
Start Date: 26/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191034964
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java
 ##
 @@ -69,21 +72,23 @@
   private final CompletableFuture onFinish;
 
   public BeamFnControlClient(
-  Endpoints.ApiServiceDescriptor apiServiceDescriptor,
-  Function channelFactory,
+  String id,
+  ApiServiceDescriptor apiServiceDescriptor,
+  ManagedChannelFactory channelFactory,
   BiFunction<
-  StreamObserverClientFactory,
-  StreamObserver,
-  StreamObserver>
+  StreamObserverClientFactory,
 
 Review comment:
   This is autoformatted.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106087)
Time Spent: 2.5h  (was: 2h 20m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #558

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4317) Enforce ErrorProne analysis in reference runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4317?focusedWorklogId=106082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106082
 ]

ASF GitHub Bot logged work on BEAM-4317:


Author: ASF GitHub Bot
Created on: 26/May/18 00:45
Start Date: 26/May/18 00:45
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5492: [BEAM-4317] Enforce 
ErrorProne analysis in runners/reference/java
URL: https://github.com/apache/beam/pull/5492#issuecomment-392221872
 
 
   Maven probably should also include pom changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106082)
Time Spent: 20m  (was: 10m)

> Enforce ErrorProne analysis in reference runner project
> ---
>
> Key: BEAM-4317
> URL: https://issues.apache.org/jira/browse/BEAM-4317
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-reference-java}} and {{beam-runners-reference-job-server}}. 
> Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-reference-java:assemble 
> :beam-runners-reference-job-server:assemble}}
> # Fix each ErrorProne warning from the {{runners/reference/}} directory.
> # In {{runners/reference/java/build.gradle}} and 
> {{runners/reference/job-server/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4371) Namespace shares the same name between Jenkins Performance test jobs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4371?focusedWorklogId=106079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106079
 ]

ASF GitHub Bot logged work on BEAM-4371:


Author: ASF GitHub Bot
Created on: 26/May/18 00:40
Start Date: 26/May/18 00:40
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #5452: [BEAM-4371] 
Namespace shares the same name between Jenkins Performance test jobs
URL: https://github.com/apache/beam/pull/5452#issuecomment-392221327
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106079)
Time Spent: 10h 10m  (was: 10h)

> Namespace shares the same name between Jenkins Performance test jobs
> 
>
> Key: BEAM-4371
> URL: https://issues.apache.org/jira/browse/BEAM-4371
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> In _common_job_properties.groovy_ is defined _getKubernetesNamespace(def 
> testName)_ which runs only after _seed job_.
> Expected behaviour is to create a new namespace with every test run.
> The failure was observed after retrying JdbcIOIT job. The first failure did 
> not deleted namespace. Next jobs failed because of:
> {code}
> + kubectl 
> --kubeconfig=/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_JDBC/config-jdbcioit-1526904136153
>  create namespace jdbcioit-1526904136153 Error from server (AlreadyExists): 
> namespaces "jdbcioit-1526904136153" already exists
> {code}
> Here are examples of jobs on Jenkins:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/601/console]
> and
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/602/console]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (1b7be6e -> d2394ca)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1b7be6e  Merge pull request #5428: [BEAM-4167] Implement UNNEST 
operator for SQL
 add 31d03b6  Replaced created datetime to jenkins build id in kubernetes 
namespace.
 new d2394ca  Merge pull request #5452: [BEAM-4371] Namespace shares the 
same name between Jenkins Performance test jobs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/common_job_properties.groovy | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Work logged] (BEAM-4371) Namespace shares the same name between Jenkins Performance test jobs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4371?focusedWorklogId=106080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106080
 ]

ASF GitHub Bot logged work on BEAM-4371:


Author: ASF GitHub Bot
Created on: 26/May/18 00:40
Start Date: 26/May/18 00:40
Worklog Time Spent: 10m 
  Work Description: chamikaramj closed pull request #5452: [BEAM-4371] 
Namespace shares the same name between Jenkins Performance test jobs
URL: https://github.com/apache/beam/pull/5452
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/common_job_properties.groovy 
b/.test-infra/jenkins/common_job_properties.groovy
index f07ac59e08d..52e1dd86a49 100644
--- a/.test-infra/jenkins/common_job_properties.groovy
+++ b/.test-infra/jenkins/common_job_properties.groovy
@@ -290,8 +290,9 @@ class common_job_properties {
 }
   }
 
+  // Namespace must contain lower case alphanumeric characters or '-'
   static String getKubernetesNamespace(def testName) {
-return "${testName}-${new Date().getTime()}"
+return "${testName}-\${BUILD_ID}"
   }
 
   static String getKubeconfigLocationForNamespace(def namespace) {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106080)
Time Spent: 10h 20m  (was: 10h 10m)

> Namespace shares the same name between Jenkins Performance test jobs
> 
>
> Key: BEAM-4371
> URL: https://issues.apache.org/jira/browse/BEAM-4371
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> In _common_job_properties.groovy_ is defined _getKubernetesNamespace(def 
> testName)_ which runs only after _seed job_.
> Expected behaviour is to create a new namespace with every test run.
> The failure was observed after retrying JdbcIOIT job. The first failure did 
> not deleted namespace. Next jobs failed because of:
> {code}
> + kubectl 
> --kubeconfig=/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_JDBC/config-jdbcioit-1526904136153
>  create namespace jdbcioit-1526904136153 Error from server (AlreadyExists): 
> namespaces "jdbcioit-1526904136153" already exists
> {code}
> Here are examples of jobs on Jenkins:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/601/console]
> and
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/602/console]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5452: [BEAM-4371] Namespace shares the same name between Jenkins Performance test jobs

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d2394ca9298feec34e618bb2828dae1ce153fa2d
Merge: 1b7be6e 31d03b6
Author: Chamikara Jayalath 
AuthorDate: Fri May 25 17:40:34 2018 -0700

Merge pull request #5452: [BEAM-4371] Namespace shares the same name 
between Jenkins Performance test jobs

 .test-infra/jenkins/common_job_properties.groovy | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=106078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106078
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 26/May/18 00:35
Start Date: 26/May/18 00:35
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #5370: [BEAM-2885] Move the 
ReferenceRunner Job Service, Artifact Services to the DirectRunner module
URL: https://github.com/apache/beam/pull/5370#issuecomment-392220906
 
 
   (And actually just taking a dependency causes issues due to shadowing, but 
copying doesn't seem like the right solution either.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106078)
Time Spent: 4.5h  (was: 4h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=106076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106076
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 26/May/18 00:29
Start Date: 26/May/18 00:29
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #5370: [BEAM-2885] Move the 
ReferenceRunner Job Service, Artifact Services to the DirectRunner module
URL: https://github.com/apache/beam/pull/5370#issuecomment-392220354
 
 
   In porting the Flink runner to master, I found it uses the LocalFilesystem 
artifact server (which was moved in this PR). This is because it does its own 
staging (after creating the job) based on what was staged through the staging 
API. Taking a dependency on the DirectRunner is ugly; is that preferable to 
reverting these changes? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106076)
Time Spent: 4h 20m  (was: 4h 10m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #214

[Apache Jenkins Server]

See 


Changes:

[tgroh] Make ImmutableExecutableStage Public

[lcwik] [BEAM-4411] Update Jackson version to latest to pick up security fixes.

[kenn] Implement UNNEST operator for SQL

--
[...truncated 361.49 KB...]
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn.processElement(AvroIOIT.java:132)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #220

[Apache Jenkins Server]

See 


Changes:

[tgroh] Make ImmutableExecutableStage Public

[lcwik] [BEAM-4411] Update Jackson version to latest to pick up security fixes.

[kenn] Implement UNNEST operator for SQL

--
[...truncated 291.31 KB...]
java.lang.RuntimeException: com.mongodb.MongoTimeoutException: Timed out 
after 3 ms while waiting for a server that matches 
ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster 
state is {type=UNKNOWN, servers=[{address=104.154.118.62:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:275)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:197)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:181)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:160)
at 
com.google.cloud.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:77)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=104.154.118.62:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.SocketTimeoutException: connect timed out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #311

[Apache Jenkins Server]

See 


Changes:

[tgroh] Make ImmutableExecutableStage Public

[lcwik] [BEAM-4411] Update Jackson version to latest to pick up security fixes.

[kenn] Implement UNNEST operator for SQL

--
[...truncated 113.86 KB...]
SLF4J: Found binding in 
[jar:file:/home/jenkins/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/jenkins/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]

Gradle Test Executor 1 finished executing tests.

> Task :beam-sdks-java-io-hadoop-input-format:integrationTest

org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT > classMethod 
FAILED
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.createTable(DatabaseTestHelper.java:46)
at 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT.setUp(HadoopInputFormatIOIT.java:98)

Caused by:
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.(PGStream.java:61)
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 10 more

org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT > classMethod 
FAILED
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.deleteTable(DatabaseTestHelper.java:57)
at 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT.tearDown(HadoopInputFormatIOIT.java:125)

Caused by:
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.(PGStream.java:61)
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 10 more

2 tests completed, 2 failed
Finished generating test XML results (0.013 secs) into: 

Generating HTML test report...
Finished generating test html results (0.027 secs) into: 

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=106066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106066
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 25/May/18 23:42
Start Date: 25/May/18 23:42
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5467: [BEAM-4328]: 
beam-sdks-java-io-google-cloud-platform:test failure
URL: https://github.com/apache/beam/pull/5467#issuecomment-392214682
 
 
   It seems like, only when I tying to run: ./gradlew build -PisRelease, 
beam-sdks-java-core-jar doesn't get repackaged dependencies.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106066)
Time Spent: 3h 20m  (was: 3h 10m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[Boyuan Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491402#comment-16491402
 ] 

Boyuan Zhang commented on BEAM-4328:


Ran ./gradlew build -PisRelease --no-parallel against master head and checked 
produced beam-sdks-java-core-2.5.0.jar, there is no repackaged things as 
expected.

More details: when running ./gradlew 
:beam-sdks-java-io-google-cloud-platform:test -PisRelease, tests passed and 
java-core -jar has repackaged things.

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4406) Increment major version used by Dataflow runner for portability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4406?focusedWorklogId=106056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106056
 ]

ASF GitHub Bot logged work on BEAM-4406:


Author: ASF GitHub Bot
Created on: 25/May/18 23:26
Start Date: 25/May/18 23:26
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #5486: [BEAM-4406] Updating 
portable Dataflow major version numbers
URL: https://github.com/apache/beam/pull/5486#issuecomment-392212666
 
 
   Seems no new failures were introduced, but there are pre-existing failures 
from a few days ago.
   
   I think this CL can wait until the ValidatesRunner tests are fixed if we 
want to be sure it succeeds.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106056)
Time Spent: 0.5h  (was: 20m)

> Increment major version used by Dataflow runner for portability
> ---
>
> Key: BEAM-4406
> URL: https://issues.apache.org/jira/browse/BEAM-4406
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The major version for portable pipelines in Java and Python is set to 1. I'd 
> like to set them to 7 so they have parity with the major version of native 
> pipelines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #350

[Apache Jenkins Server]

See 


--
[...truncated 18.09 MB...]
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 25, 2018 10:54:56 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 25, 2018 10:54:56 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525225452-1f29f4e2/output/results/staging/
May 25, 2018 10:54:56 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash 6k_jKhNdgPvBrYBBtdSQYQ> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525225452-1f29f4e2/output/results/staging/pipeline-6k_jKhNdgPvBrYBBtdSQYQ.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 10:54:58 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_15_54_57-3947313671853722850?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-25_15_54_57-3947313671853722850

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 10:54:58 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_15_54_57-3947313671853722850
May 25, 2018 10:54:58 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-25_15_54_57-3947313671853722850 with 1 
expected assertions.
May 25, 2018 10:55:07 PM 

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=106051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106051
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 25/May/18 22:56
Start Date: 25/May/18 22:56
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5490: [BEAM-4077] 
Refactor schemas and fields for simplicity and clarity
URL: https://github.com/apache/beam/pull/5490#issuecomment-392208122
 
 
   Ah, the calcite <-> Beam mapping was a bit funky since it mixed up type 
names and types. I will need to spend a little more time on that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106051)
Time Spent: 1h  (was: 50m)

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?focusedWorklogId=106049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106049
 ]

ASF GitHub Bot logged work on BEAM-3813:


Author: ASF GitHub Bot
Created on: 25/May/18 22:55
Start Date: 25/May/18 22:55
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5491: [BEAM-3813] Support 
(de)serialization of S3 encryption options via JSON
URL: https://github.com/apache/beam/pull/5491#issuecomment-392207913
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106049)
Time Spent: 2h 20m  (was: 2h 10m)

> Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> We should enable AWS S3 users to use encryption when reading or writing by 
> provide encryption keys or using server side encryption via an algorithm, or 
> a key management system (KMS)..
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?focusedWorklogId=106050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106050
 ]

ASF GitHub Bot logged work on BEAM-3813:


Author: ASF GitHub Bot
Created on: 25/May/18 22:55
Start Date: 25/May/18 22:55
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5491: [BEAM-3813] Support 
(de)serialization of S3 encryption options via JSON
URL: https://github.com/apache/beam/pull/5491#issuecomment-392207913
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106050)
Time Spent: 2.5h  (was: 2h 20m)

> Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We should enable AWS S3 users to use encryption when reading or writing by 
> provide encryption keys or using server side encryption via an algorithm, or 
> a key management system (KMS)..
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106031
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191017457
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
 ##
 @@ -6,31 +6,33 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
  * http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
  */
-package org.apache.beam.runners.reference.testing;
 
-import io.grpc.ManagedChannel;
-import io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+package org.apache.beam.fn.harness.control;
 
-/**
- * A {@link org.apache.beam.sdk.fn.channel.ManagedChannelFactory} that uses 
in-process channels.
- *
- * The channel builder uses {@link ApiServiceDescriptor#getUrl()} as the 
unique in-process name.
- */
-public class InProcessManagedChannelFactory extends ManagedChannelFactory {
+import io.grpc.ClientInterceptor;
+import io.grpc.Metadata;
+import io.grpc.Metadata.Key;
+import io.grpc.stub.MetadataUtils;
+
+/** A {@link ClientInterceptor} that attaches a provided SDK Harness ID to 
outgoing messages. */
+public class AddHarnessIdInterceptor {
+  private static final Key ID_KEY = Key.of("worker_id", 
Metadata.ASCII_STRING_MARSHALLER);
 
-  @Override
-  public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
-return 
InProcessChannelBuilder.forName(apiServiceDescriptor.getUrl()).build();
+  public static ClientInterceptor create(String harnessId) {
 
 Review comment:
   Shall we have a null check for harnessId?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106031)
Time Spent: 1h 10m  (was: 1h)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106037
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191012929
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
 ##
 @@ -70,13 +92,42 @@ public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
*/
   private static class Default extends ManagedChannelFactory {
 @Override
-public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
+public ManagedChannel forDescriptor(
+ApiServiceDescriptor apiServiceDescriptor, List 
interceptors) {
   return ManagedChannelBuilder.forTarget(apiServiceDescriptor.getUrl())
   .usePlaintext(true)
   // Set the message size to max value here. The actual size is 
governed by the
   // buffer size in the layers above.
   .maxInboundMessageSize(Integer.MAX_VALUE)
+  .intercept(interceptors)
   .build();
 }
   }
+
+  private static class InterceptedManagedChannelFactory extends 
ManagedChannelFactory {
+private final ManagedChannelFactory channelFactory;
+private final List interceptors;
+
+private InterceptedManagedChannelFactory(
+ManagedChannelFactory managedChannelFactory, List 
interceptors) {
+  this.channelFactory = managedChannelFactory;
+  this.interceptors = interceptors;
+}
+
+@Override
+protected ManagedChannel forDescriptorOnly(ApiServiceDescriptor 
apiServiceDescriptor) {
+  return forDescriptor(apiServiceDescriptor, interceptors);
+}
+
+@Override
+protected ManagedChannel forDescriptor(
+ApiServiceDescriptor apiServiceDescriptor, List 
interceptors) {
+  return channelFactory.forDescriptor(apiServiceDescriptor, interceptors);
+}
+
+@Override
+public ManagedChannelFactory withInterceptors(List 
interceptors) {
+  return new InterceptedManagedChannelFactory(channelFactory, 
interceptors);
 
 Review comment:
   I suppose this method is for Factory chaining.
   Should we pass "this" in that case?
   
   I would suggest to remove chaining in this manner and provide a builder to 
do the chaining.
   Something like this:
   
`ManagedChannelFactoryBuilder.builder(channelFactory).withInterceptors(interceptors)`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106037)
Time Spent: 1h 40m  (was: 1.5h)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106038
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191015522
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
 ##
 @@ -115,43 +124,46 @@ public static void main(PipelineOptions options,
   }
 
   public static void main(
+  String id,
   PipelineOptions options,
   Endpoints.ApiServiceDescriptor loggingApiServiceDescriptor,
   Endpoints.ApiServiceDescriptor controlApiServiceDescriptor,
   ManagedChannelFactory channelFactory,
   StreamObserverFactory streamObserverFactory) {
 IdGenerator idGenerator = IdGenerators.decrementingLongs();
-try (BeamFnLoggingClient logging = new BeamFnLoggingClient(
-options,
-loggingApiServiceDescriptor,
-channelFactory::forDescriptor)) {
+try (BeamFnLoggingClient logging =
+new BeamFnLoggingClient(
+options, loggingApiServiceDescriptor, 
channelFactory::forDescriptor)) {
 
   LOG.info("Fn Harness started");
-  EnumMap Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106039
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191017692
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java
 ##
 @@ -69,21 +72,23 @@
   private final CompletableFuture onFinish;
 
   public BeamFnControlClient(
-  Endpoints.ApiServiceDescriptor apiServiceDescriptor,
-  Function channelFactory,
+  String id,
+  ApiServiceDescriptor apiServiceDescriptor,
+  ManagedChannelFactory channelFactory,
   BiFunction<
-  StreamObserverClientFactory,
-  StreamObserver,
-  StreamObserver>
+  StreamObserverClientFactory,
 
 Review comment:
   Reformat?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106039)
Time Spent: 2h  (was: 1h 50m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106034
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191013612
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
 ##
 @@ -41,7 +42,24 @@ public static ManagedChannelFactory createEpoll() {
 return new Epoll();
   }
 
-  public abstract ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor);
+  public final ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
+return forDescriptorOnly(apiServiceDescriptor);
+  }
+
+  protected ManagedChannel forDescriptorOnly(ApiServiceDescriptor descriptor) {
 
 Review comment:
   Based on the previous comment, we can remove this method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106034)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106032
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191015783
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
 ##
 @@ -115,43 +124,46 @@ public static void main(PipelineOptions options,
   }
 
   public static void main(
+  String id,
   PipelineOptions options,
   Endpoints.ApiServiceDescriptor loggingApiServiceDescriptor,
   Endpoints.ApiServiceDescriptor controlApiServiceDescriptor,
   ManagedChannelFactory channelFactory,
   StreamObserverFactory streamObserverFactory) {
 IdGenerator idGenerator = IdGenerators.decrementingLongs();
-try (BeamFnLoggingClient logging = new BeamFnLoggingClient(
-options,
-loggingApiServiceDescriptor,
-channelFactory::forDescriptor)) {
+try (BeamFnLoggingClient logging =
 
 Review comment:
   logging is not used. Shall we remove it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106032)
Time Spent: 1h 20m  (was: 1h 10m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106033
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191017563
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
 ##
 @@ -6,31 +6,33 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
  * http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
  */
-package org.apache.beam.runners.reference.testing;
 
-import io.grpc.ManagedChannel;
-import io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+package org.apache.beam.fn.harness.control;
 
-/**
- * A {@link org.apache.beam.sdk.fn.channel.ManagedChannelFactory} that uses 
in-process channels.
- *
- * The channel builder uses {@link ApiServiceDescriptor#getUrl()} as the 
unique in-process name.
- */
-public class InProcessManagedChannelFactory extends ManagedChannelFactory {
+import io.grpc.ClientInterceptor;
+import io.grpc.Metadata;
+import io.grpc.Metadata.Key;
+import io.grpc.stub.MetadataUtils;
+
+/** A {@link ClientInterceptor} that attaches a provided SDK Harness ID to 
outgoing messages. */
+public class AddHarnessIdInterceptor {
 
 Review comment:
   Not sure if we need a repository of interceptors. Just putting it as a 
thought.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106033)
Time Spent: 1.5h  (was: 1h 20m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106029=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106029
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191014342
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
 ##
 @@ -70,13 +92,42 @@ public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
*/
   private static class Default extends ManagedChannelFactory {
 @Override
-public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
+public ManagedChannel forDescriptor(
+ApiServiceDescriptor apiServiceDescriptor, List 
interceptors) {
   return ManagedChannelBuilder.forTarget(apiServiceDescriptor.getUrl())
   .usePlaintext(true)
   // Set the message size to max value here. The actual size is 
governed by the
   // buffer size in the layers above.
   .maxInboundMessageSize(Integer.MAX_VALUE)
+  .intercept(interceptors)
   .build();
 }
   }
+
+  private static class InterceptedManagedChannelFactory extends 
ManagedChannelFactory {
 
 Review comment:
   What is the purpose of this class?
   Based on method `ManagedChannelFactory.forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor, List interceptors)`
   All implementations of ManagedChannelFactory should support interception and 
if not, they should throw exception.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106029)
Time Spent: 50m  (was: 40m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106035=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106035
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191015296
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
 ##
 @@ -80,23 +83,28 @@ public static void main(String[] args) throws Exception {
 System.out.format("Control location %s%n", 
System.getenv(CONTROL_API_SERVICE_DESCRIPTOR));
 System.out.format("Pipeline options %s%n", 
System.getenv(PIPELINE_OPTIONS));
 
-ObjectMapper objectMapper = new ObjectMapper().registerModules(
-ObjectMapper.findModules(ReflectHelpers.findClassLoader()));
-PipelineOptions options = objectMapper.readValue(
-System.getenv(PIPELINE_OPTIONS), PipelineOptions.class);
+String id = System.getenv(HARNESS_ID);
 
 Review comment:
   Shall we also log this parameter as other parameters.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106035)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106030
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191010767
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
 ##
 @@ -41,7 +42,24 @@ public static ManagedChannelFactory createEpoll() {
 return new Epoll();
   }
 
-  public abstract ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor);
+  public final ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
+return forDescriptorOnly(apiServiceDescriptor);
 
 Review comment:
   Shall we get away with this indirection?
   Implementer can always overwrite this public method (if we drop the final).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106030)
Time Spent: 1h  (was: 50m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106036
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191014723
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
 ##
 @@ -70,13 +92,42 @@ public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
*/
   private static class Default extends ManagedChannelFactory {
 @Override
-public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
+public ManagedChannel forDescriptor(
 
 Review comment:
   As this method requires more than just descriptor, shall we rename it to 
something like `createChannel` or `getChannel`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106036)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=106028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106028
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 25/May/18 21:59
Start Date: 25/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5456: 
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#discussion_r191012317
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
 ##
 @@ -70,13 +92,42 @@ public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
*/
   private static class Default extends ManagedChannelFactory {
 @Override
-public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
+public ManagedChannel forDescriptor(
+ApiServiceDescriptor apiServiceDescriptor, List 
interceptors) {
   return ManagedChannelBuilder.forTarget(apiServiceDescriptor.getUrl())
   .usePlaintext(true)
   // Set the message size to max value here. The actual size is 
governed by the
   // buffer size in the layers above.
   .maxInboundMessageSize(Integer.MAX_VALUE)
+  .intercept(interceptors)
   .build();
 }
   }
+
+  private static class InterceptedManagedChannelFactory extends 
ManagedChannelFactory {
+private final ManagedChannelFactory channelFactory;
+private final List interceptors;
+
+private InterceptedManagedChannelFactory(
+ManagedChannelFactory managedChannelFactory, List 
interceptors) {
+  this.channelFactory = managedChannelFactory;
+  this.interceptors = interceptors;
+}
+
+@Override
+protected ManagedChannel forDescriptorOnly(ApiServiceDescriptor 
apiServiceDescriptor) {
+  return forDescriptor(apiServiceDescriptor, interceptors);
+}
+
+@Override
+protected ManagedChannel forDescriptor(
+ApiServiceDescriptor apiServiceDescriptor, List 
interceptors) {
+  return channelFactory.forDescriptor(apiServiceDescriptor, interceptors);
 
 Review comment:
   The behavior in not well defined.
   Though this is an intercepted factory, the original interceptors 
(interceptors from constructor) are not applied.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106028)
Time Spent: 40m  (was: 0.5h)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4317) Enforce ErrorProne analysis in reference runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4317?focusedWorklogId=106025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106025
 ]

ASF GitHub Bot logged work on BEAM-4317:


Author: ASF GitHub Bot
Created on: 25/May/18 21:45
Start Date: 25/May/18 21:45
Worklog Time Spent: 10m 
  Work Description: iemejia opened a new pull request #5492: [BEAM-4317] 
Enforce ErrorProne analysis in runners/referebce/java
URL: https://github.com/apache/beam/pull/5492
 
 
   R: @tgroh  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106025)
Time Spent: 10m
Remaining Estimate: 0h

> Enforce ErrorProne analysis in reference runner project
> ---
>
> Key: BEAM-4317
> URL: https://issues.apache.org/jira/browse/BEAM-4317
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-reference-java}} and {{beam-runners-reference-job-server}}. 
> Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-reference-java:assemble 
> :beam-runners-reference-job-server:assemble}}
> # Fix each ErrorProne warning from the {{runners/reference/}} directory.
> # In {{runners/reference/java/build.gradle}} and 
> {{runners/reference/job-server/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4317) Enforce ErrorProne analysis in reference runner project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4317:
--

Assignee: Ismaël Mejía

> Enforce ErrorProne analysis in reference runner project
> ---
>
> Key: BEAM-4317
> URL: https://issues.apache.org/jira/browse/BEAM-4317
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-reference-java}} and {{beam-runners-reference-job-server}}. 
> Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-reference-java:assemble 
> :beam-runners-reference-job-server:assemble}}
> # Fix each ErrorProne warning from the {{runners/reference/}} directory.
> # In {{runners/reference/java/build.gradle}} and 
> {{runners/reference/job-server/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?focusedWorklogId=106020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106020
 ]

ASF GitHub Bot logged work on BEAM-3813:


Author: ASF GitHub Bot
Created on: 25/May/18 21:27
Start Date: 25/May/18 21:27
Worklog Time Spent: 10m 
  Work Description: iemejia opened a new pull request #5491: [BEAM-3813] 
Support (de)serialization of S3 encryption options via JSON
URL: https://github.com/apache/beam/pull/5491
 
 
   Following your remarks, I reused the same JIRA ticket since this is related. 
PTAL
   R: @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106020)
Time Spent: 2h 10m  (was: 2h)

> Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We should enable AWS S3 users to use encryption when reading or writing by 
> provide encryption keys or using server side encryption via an algorithm, or 
> a key management system (KMS)..
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #5111

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #557

[Apache Jenkins Server]

See 


Changes:

[kenn] Implement UNNEST operator for SQL

--
[...truncated 17.72 MB...]

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 25, 2018 9:04:14 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_14_04_13-8205296970256443908?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-25_14_04_13-8205296970256443908

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 25, 2018 9:04:14 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_14_04_13-8205296970256443908
May 25, 2018 9:04:14 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-25_14_04_13-8205296970256443908 with 0 
expected assertions.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:13.429Z: Autoscaling is enabled for job 
2018-05-25_14_04_13-8205296970256443908. The number of workers will be between 
1 and 1000.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:13.473Z: Autoscaling was automatically enabled for 
job 2018-05-25_14_04_13-8205296970256443908.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:16.038Z: Checking required Cloud APIs are enabled.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:16.290Z: Checking permissions granted to controller 
Service Account.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:21.483Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:21.996Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.271Z: Expanding GroupByKey operations into 
optimizable parts.
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.321Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.609Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.651Z: Elided trivial flatten 
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.706Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.762Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.812Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 25, 2018 9:04:26 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T21:04:22.864Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to 

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=106013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106013
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 25/May/18 21:05
Start Date: 25/May/18 21:05
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #5490: 
[BEAM-4077] Refactor schemas and fields for simplicity and clarity
URL: https://github.com/apache/beam/pull/5490
 
 
   This refactor makes the difference between TypeName (which is not a type) 
and FieldType (which is a type) more explicit and makes it easier and quicker 
to construct types. It also makes the 95% case of non-null types a bit pithier.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106013)
Time Spent: 40m  (was: 0.5h)

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=106014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106014
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 25/May/18 21:05
Start Date: 25/May/18 21:05
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5490: [BEAM-4077] 
Refactor schemas and fields for simplicity and clarity
URL: https://github.com/apache/beam/pull/5490#issuecomment-392187220
 
 
   R: @akedin 
   
   I think many of these cleanups are suggestions from you. I think there's 
more that could be done, too, and I am happy to spend a few more minutes. Like 
the `withCollectionType` methods and all that are not actually needed any more.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106014)
Time Spent: 50m  (was: 40m)

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=106012=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106012
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 25/May/18 21:02
Start Date: 25/May/18 21:02
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5488: [BEAM-4077] 
Schema fields are non-null by default
URL: https://github.com/apache/beam/pull/5488
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index a9d2f333d12..3a2fa6d8c65 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -73,65 +73,64 @@ public Builder addField(Field field) {
   return this;
 }
 
-public Builder addByteField(String name, boolean nullable) {
-  fields.add(Field.of(name, TypeName.BYTE.type()).withNullable(nullable));
+public Builder addByteField(String name) {
+  fields.add(Field.of(name, FieldType.BYTE));
   return this;
 }
 
-public Builder addInt16Field(String name, boolean nullable) {
-  fields.add(Field.of(name, TypeName.INT16.type()).withNullable(nullable));
+public Builder addInt16Field(String name) {
+  fields.add(Field.of(name, FieldType.INT16));
   return this;
 }
 
-public Builder addInt32Field(String name, boolean nullable) {
-  fields.add(Field.of(name, TypeName.INT32.type()).withNullable(nullable));
+public Builder addInt32Field(String name) {
+  fields.add(Field.of(name, FieldType.INT32));
   return this;
 }
 
-public Builder addInt64Field(String name, boolean nullable) {
-  fields.add(Field.of(name, TypeName.INT64.type()).withNullable(nullable));
+public Builder addInt64Field(String name) {
+  fields.add(Field.of(name, FieldType.INT64));
   return this;
 }
 
-public Builder addDecimalField(String name, boolean nullable) {
-  fields.add(Field.of(name, 
TypeName.DECIMAL.type()).withNullable(nullable));
+public Builder addDecimalField(String name) {
+  fields.add(Field.of(name, FieldType.DECIMAL));
   return this;
 }
 
-public Builder addFloatField(String name, boolean nullable) {
-  fields.add(Field.of(name, TypeName.FLOAT.type()).withNullable(nullable));
+public Builder addFloatField(String name) {
+  fields.add(Field.of(name, FieldType.FLOAT));
   return this;
 }
 
-public Builder addDoubleField(String name, boolean nullable) {
-  fields.add(Field.of(name, 
TypeName.DOUBLE.type()).withNullable(nullable));
+public Builder addDoubleField(String name) {
+  fields.add(Field.of(name, FieldType.DOUBLE));
   return this;
 }
 
-public Builder addStringField(String name, boolean nullable) {
-  fields.add(Field.of(name, 
TypeName.STRING.type()).withNullable(nullable));
+public Builder addStringField(String name) {
+  fields.add(Field.of(name, FieldType.STRING));
   return this;
 }
 
-public Builder addDateTimeField(String name, boolean nullable) {
-  fields.add(Field.of(name, 
TypeName.DATETIME.type()).withNullable(nullable));
+public Builder addDateTimeField(String name) {
+  fields.add(Field.of(name, FieldType.DATETIME));
   return this;
 }
 
-public Builder addBooleanField(String name, boolean nullable) {
-  fields.add(Field.of(name, 
TypeName.BOOLEAN.type()).withNullable(nullable));
+public Builder addBooleanField(String name) {
+  fields.add(Field.of(name, FieldType.BOOLEAN));
   return this;
 }
 
 public Builder addArrayField(String name, FieldType collectionElementType) 
{
   fields.add(
-  Field.of(name, 
TypeName.ARRAY.type().withCollectionElementType(collectionElementType)));
+  Field.of(name, FieldType.array(collectionElementType)));
   return this;
 }
 
-public Builder addRowField(String name, Schema fieldSchema, boolean 
nullable) {
-  fields.add(Field.of(name, TypeName.ROW.type().withRowSchema(fieldSchema))
-  .withNullable(nullable));
+public Builder addRowField(String name, Schema fieldSchema) {
+  fields.add(Field.of(name, FieldType.row(fieldSchema)));
   return this;
 }
 
@@ -186,8 +185,13 @@ public int hashCode() {
 return fields;
   }
 
-  /**
-   * An enumerated list of supported types.
+  /** An enumerated list of type constructors.
+   *
+   * 
+   *   Atomic types are built from type constructors that take no 

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=106011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-106011
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 25/May/18 21:02
Start Date: 25/May/18 21:02
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5488: [BEAM-4077] 
Schema fields are non-null by default
URL: https://github.com/apache/beam/pull/5488#issuecomment-392186549
 
 
   Actually this needs some deeper work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 106011)
Time Spent: 20m  (was: 10m)

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5110

[Apache Jenkins Server]

See 


Changes:

[kenn] Implement UNNEST operator for SQL

--
[...truncated 1.05 MB...]
test_compatibility (apache_beam.typehints.typehints_test.DictHintTestCase) ... 
ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_key_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #349

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4308] Enforce ErrorProne analysis in runners-core-java

--
[...truncated 18.00 MB...]
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 25, 2018 8:42:03 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 25, 2018 8:42:03 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525204200-e318b26c/output/results/staging/
May 25, 2018 8:42:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash a4AKjWWef-uZqG1fwRw2aw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525204200-e318b26c/output/results/staging/pipeline-a4AKjWWef-uZqG1fwRw2aw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 8:42:05 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_13_42_04-414989599226294784?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-25_13_42_04-414989599226294784

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 8:42:05 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_13_42_04-414989599226294784
May 25, 2018 8:42:05 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-25_13_42_04-414989599226294784 with 

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #555

[Apache Jenkins Server]

See 

[beam] branch master updated (795f064 -> 1b7be6e)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 795f064  Make ImmutableExecutableStage Public
 add b7ea0ed  Implement UNNEST operator for SQL
 new 1b7be6e  Merge pull request #5428: [BEAM-4167] Implement UNNEST 
operator for SQL

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../interpreter/BeamSqlExpressionExecutor.java |   4 +-
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  13 +-
 .../operator/BeamSqlCaseExpression.java|  12 +-
 .../operator/BeamSqlCastExpression.java|  29 ++--
 ...n.java => BeamSqlCorrelVariableExpression.java} |  30 ++--
 .../operator/BeamSqlDefaultExpression.java |   4 +-
 .../interpreter/operator/BeamSqlDotExpression.java |   8 +-
 .../interpreter/operator/BeamSqlExpression.java|   9 +-
 .../operator/BeamSqlInputRefExpression.java|   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java |   9 +-
 .../interpreter/operator/BeamSqlUdfExpression.java |   6 +-
 .../operator/BeamSqlWindowEndExpression.java   |   4 +-
 .../operator/BeamSqlWindowExpression.java  |   6 +-
 .../operator/BeamSqlWindowStartExpression.java |   4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java|  10 +-
 .../operator/array/BeamSqlArrayExpression.java |   6 +-
 .../operator/array/BeamSqlArrayItemExpression.java |   8 +-
 .../collection/BeamSqlCardinalityExpression.java   |   8 +-
 .../collection/BeamSqlSingleElementExpression.java |   6 +-
 .../comparison/BeamSqlCompareExpression.java   |   8 +-
 .../comparison/BeamSqlIsNotNullExpression.java |   6 +-
 .../comparison/BeamSqlIsNullExpression.java|   6 +-
 .../date/BeamSqlCurrentDateExpression.java |   4 +-
 .../date/BeamSqlCurrentTimeExpression.java |   4 +-
 .../date/BeamSqlCurrentTimestampExpression.java|   4 +-
 .../operator/date/BeamSqlDateCeilExpression.java   |   6 +-
 .../operator/date/BeamSqlDateFloorExpression.java  |   6 +-
 .../date/BeamSqlDatetimeMinusExpression.java   |   5 +-
 .../date/BeamSqlDatetimePlusExpression.java|  20 ++-
 .../operator/date/BeamSqlExtractExpression.java|   6 +-
 .../date/BeamSqlIntervalMultiplyExpression.java|  12 +-
 .../BeamSqlTimestampMinusIntervalExpression.java   |   8 +-
 .../BeamSqlTimestampMinusTimestampExpression.java  |  10 +-
 .../operator/logical/BeamSqlAndExpression.java |   6 +-
 .../operator/logical/BeamSqlNotExpression.java |   6 +-
 .../operator/logical/BeamSqlOrExpression.java  |   6 +-
 .../operator/map/BeamSqlMapExpression.java |   8 +-
 .../operator/map/BeamSqlMapItemExpression.java |   9 +-
 .../operator/math/BeamSqlMathBinaryExpression.java |   8 +-
 .../operator/math/BeamSqlMathUnaryExpression.java  |   6 +-
 .../operator/math/BeamSqlPiExpression.java |   4 +-
 .../operator/math/BeamSqlRandExpression.java   |   6 +-
 .../math/BeamSqlRandIntegerExpression.java |   9 +-
 .../reinterpret/BeamSqlReinterpretExpression.java  |   7 +-
 .../operator/row/BeamSqlFieldAccessExpression.java |   6 +-
 .../string/BeamSqlCharLengthExpression.java|   6 +-
 .../operator/string/BeamSqlConcatExpression.java   |   8 +-
 .../operator/string/BeamSqlInitCapExpression.java  |   6 +-
 .../operator/string/BeamSqlLowerExpression.java|   6 +-
 .../operator/string/BeamSqlOverlayExpression.java  |  12 +-
 .../operator/string/BeamSqlPositionExpression.java |  17 +--
 .../string/BeamSqlSubstringExpression.java |  17 +--
 .../operator/string/BeamSqlTrimExpression.java |  13 +-
 .../operator/string/BeamSqlUpperExpression.java|   6 +-
 .../sql/impl/planner/BeamQueryPlanner.java |   1 +
 .../extensions/sql/impl/planner/BeamRuleSets.java  |   4 +
 .../{BeamFilterRel.java => BeamUncollectRel.java}  |  54 ---
 .../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 158 +
 .../{BeamUnionRule.java => BeamUncollectRule.java} |  31 ++--
 .../extensions/sql/impl/rule/BeamUnnestRule.java   |  58 
 .../impl/transform/BeamAggregationTransforms.java  |   7 +-
 .../sql/impl/transform/BeamSqlFilterFn.java|   3 +-
 .../sql/impl/transform/BeamSqlProjectFn.java   |   3 +-
 ...mSqlArrayTest.java => BeamSqlDslArrayTest.java} |  92 +++-
 .../operator/BeamNullExperssionTest.java   |   9 +-
 .../operator/BeamSqlAndOrExpressionTest.java   |  13 +-
 .../operator/BeamSqlCaseExpressionTest.java|  13 +-
 .../operator/BeamSqlCastExpressionTest.java|  41 --
 .../operator/BeamSqlCompareExpressionTest.java |  25 ++--
 .../operator/BeamSqlDotExpressionTest.java |   7 +-
 .../operator/BeamSqlInputRefExpressionTest.java|  

[beam] 01/01: Merge pull request #5428: [BEAM-4167] Implement UNNEST operator for SQL

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1b7be6e3040c48547caa73b670242af43dd5ec80
Merge: 795f064 b7ea0ed
Author: Kenn Knowles 
AuthorDate: Fri May 25 13:21:09 2018 -0700

Merge pull request #5428: [BEAM-4167] Implement UNNEST operator for SQL

 .../interpreter/BeamSqlExpressionExecutor.java |   4 +-
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  13 +-
 .../operator/BeamSqlCaseExpression.java|  12 +-
 .../operator/BeamSqlCastExpression.java|  29 ++--
 ...n.java => BeamSqlCorrelVariableExpression.java} |  30 ++--
 .../operator/BeamSqlDefaultExpression.java |   4 +-
 .../interpreter/operator/BeamSqlDotExpression.java |   8 +-
 .../interpreter/operator/BeamSqlExpression.java|   9 +-
 .../operator/BeamSqlInputRefExpression.java|   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java |   9 +-
 .../interpreter/operator/BeamSqlUdfExpression.java |   6 +-
 .../operator/BeamSqlWindowEndExpression.java   |   4 +-
 .../operator/BeamSqlWindowExpression.java  |   6 +-
 .../operator/BeamSqlWindowStartExpression.java |   4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java|  10 +-
 .../operator/array/BeamSqlArrayExpression.java |   6 +-
 .../operator/array/BeamSqlArrayItemExpression.java |   8 +-
 .../collection/BeamSqlCardinalityExpression.java   |   8 +-
 .../collection/BeamSqlSingleElementExpression.java |   6 +-
 .../comparison/BeamSqlCompareExpression.java   |   8 +-
 .../comparison/BeamSqlIsNotNullExpression.java |   6 +-
 .../comparison/BeamSqlIsNullExpression.java|   6 +-
 .../date/BeamSqlCurrentDateExpression.java |   4 +-
 .../date/BeamSqlCurrentTimeExpression.java |   4 +-
 .../date/BeamSqlCurrentTimestampExpression.java|   4 +-
 .../operator/date/BeamSqlDateCeilExpression.java   |   6 +-
 .../operator/date/BeamSqlDateFloorExpression.java  |   6 +-
 .../date/BeamSqlDatetimeMinusExpression.java   |   5 +-
 .../date/BeamSqlDatetimePlusExpression.java|  20 ++-
 .../operator/date/BeamSqlExtractExpression.java|   6 +-
 .../date/BeamSqlIntervalMultiplyExpression.java|  12 +-
 .../BeamSqlTimestampMinusIntervalExpression.java   |   8 +-
 .../BeamSqlTimestampMinusTimestampExpression.java  |  10 +-
 .../operator/logical/BeamSqlAndExpression.java |   6 +-
 .../operator/logical/BeamSqlNotExpression.java |   6 +-
 .../operator/logical/BeamSqlOrExpression.java  |   6 +-
 .../operator/map/BeamSqlMapExpression.java |   8 +-
 .../operator/map/BeamSqlMapItemExpression.java |   9 +-
 .../operator/math/BeamSqlMathBinaryExpression.java |   8 +-
 .../operator/math/BeamSqlMathUnaryExpression.java  |   6 +-
 .../operator/math/BeamSqlPiExpression.java |   4 +-
 .../operator/math/BeamSqlRandExpression.java   |   6 +-
 .../math/BeamSqlRandIntegerExpression.java |   9 +-
 .../reinterpret/BeamSqlReinterpretExpression.java  |   7 +-
 .../operator/row/BeamSqlFieldAccessExpression.java |   6 +-
 .../string/BeamSqlCharLengthExpression.java|   6 +-
 .../operator/string/BeamSqlConcatExpression.java   |   8 +-
 .../operator/string/BeamSqlInitCapExpression.java  |   6 +-
 .../operator/string/BeamSqlLowerExpression.java|   6 +-
 .../operator/string/BeamSqlOverlayExpression.java  |  12 +-
 .../operator/string/BeamSqlPositionExpression.java |  17 +--
 .../string/BeamSqlSubstringExpression.java |  17 +--
 .../operator/string/BeamSqlTrimExpression.java |  13 +-
 .../operator/string/BeamSqlUpperExpression.java|   6 +-
 .../sql/impl/planner/BeamQueryPlanner.java |   1 +
 .../extensions/sql/impl/planner/BeamRuleSets.java  |   4 +
 .../extensions/sql/impl/rel/BeamUncollectRel.java  |  88 
 .../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 158 +
 .../sql/impl/rule/BeamUncollectRule.java   |  49 +++
 .../extensions/sql/impl/rule/BeamUnnestRule.java   |  58 
 .../impl/transform/BeamAggregationTransforms.java  |   7 +-
 .../sql/impl/transform/BeamSqlFilterFn.java|   3 +-
 .../sql/impl/transform/BeamSqlProjectFn.java   |   3 +-
 ...mSqlArrayTest.java => BeamSqlDslArrayTest.java} |  92 +++-
 .../operator/BeamNullExperssionTest.java   |   9 +-
 .../operator/BeamSqlAndOrExpressionTest.java   |  13 +-
 .../operator/BeamSqlCaseExpressionTest.java|  13 +-
 .../operator/BeamSqlCastExpressionTest.java|  41 --
 .../operator/BeamSqlCompareExpressionTest.java |  25 ++--
 .../operator/BeamSqlDotExpressionTest.java |   7 +-
 .../operator/BeamSqlInputRefExpressionTest.java|  13 +-
 .../interpreter/operator/BeamSqlPrimitiveTest.java |  16 ++-
 .../operator/BeamSqlReinterpretExpressionTest.java |   5 +-
 .../operator/BeamSqlUdfExpressionTest.java |   3 +-
 

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105999
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 25/May/18 20:16
Start Date: 25/May/18 20:16
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5428: [BEAM-4167] 
Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#issuecomment-392168402
 
 
   Thanks! Rewriting and merging.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105999)
Time Spent: 4h 10m  (was: 4h)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (354af94 -> 795f064)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 354af94  [BEAM-4411] Update Jackson version to latest to pick up 
security fixes.
 add 8b70862  Make ImmutableExecutableStage Public
 new 795f064  Make ImmutableExecutableStage Public

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/core/construction/graph/ImmutableExecutableStage.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Make ImmutableExecutableStage Public

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 795f064cbe0d972bee47b4967f6cbdaa15764eee
Merge: 354af94 8b70862
Author: Lukasz Cwik 
AuthorDate: Fri May 25 12:56:21 2018 -0700

Make ImmutableExecutableStage Public

 .../beam/runners/core/construction/graph/ImmutableExecutableStage.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=105991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105991
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 25/May/18 19:50
Start Date: 25/May/18 19:50
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5489: 
[BEAM-4290] proto changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489#discussion_r190993753
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -102,17 +99,24 @@ message ArtifactChunk {
   bytes data = 1;
 }
 
+message PutArtifactMetadata {
+  // (Required) An identifier for artifact staging session.
+  string artifact_staging_id = 1;
+  // (Required) The Artifact metadata.
+  ArtifactMetadata metadata = 2;
+}
+
 // A request to stage an artifact.
 message PutArtifactRequest {
   // (Required)
   oneof content {
-// The Artifact metadata. The first message in a PutArtifact call must 
contain the name
-// of the artifact.
-ArtifactMetadata metadata = 1;
+// The first message in a PutArtifact call must contain this field.
+PutArtifactMetadata metadata = 1;
 
 // A chunk of the artifact. All messages after the first in a PutArtifact 
call must contain a
 // chunk.
 ArtifactChunk data = 2;
+
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105991)
Time Spent: 40m  (was: 0.5h)

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=105992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105992
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 25/May/18 19:50
Start Date: 25/May/18 19:50
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5489: 
[BEAM-4290] proto changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489#discussion_r190994089
 
 

 ##
 File path: model/job-management/src/main/proto/beam_job_api.proto
 ##
 @@ -75,6 +75,11 @@ message PrepareJobResponse {
   // An endpoint which exposes the Beam Artifact Staging API. Artifacts used 
by the job should be
   // staged to this endpoint, and will be available during job execution.
   org.apache.beam.model.pipeline.v1.ApiServiceDescriptor 
artifact_staging_endpoint = 2;
+
+  // (required) Id for the artifact staging. The content of the Id can contain
 
 Review comment:
   Sounds right. Made the changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105992)
Time Spent: 50m  (was: 40m)

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=105990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105990
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 25/May/18 19:45
Start Date: 25/May/18 19:45
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5489: 
[BEAM-4290] proto changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489#discussion_r190993275
 
 

 ##
 File path: model/job-management/src/main/proto/beam_job_api.proto
 ##
 @@ -75,6 +75,11 @@ message PrepareJobResponse {
   // An endpoint which exposes the Beam Artifact Staging API. Artifacts used 
by the job should be
   // staged to this endpoint, and will be available during job execution.
   org.apache.beam.model.pipeline.v1.ApiServiceDescriptor 
artifact_staging_endpoint = 2;
+
+  // (required) Id for the artifact staging. The content of the Id can contain
 
 Review comment:
   I would not mention that it is parsable as that is an implementation detail.
   
   Also, the comment above about the preparationId is now out of date as it is 
not related to artifact staging.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105990)
Time Spent: 0.5h  (was: 20m)

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=105989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105989
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 25/May/18 19:45
Start Date: 25/May/18 19:45
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5489: 
[BEAM-4290] proto changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489#discussion_r190992911
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -102,17 +99,24 @@ message ArtifactChunk {
   bytes data = 1;
 }
 
+message PutArtifactMetadata {
+  // (Required) An identifier for artifact staging session.
+  string artifact_staging_id = 1;
+  // (Required) The Artifact metadata.
+  ArtifactMetadata metadata = 2;
+}
+
 // A request to stage an artifact.
 message PutArtifactRequest {
   // (Required)
   oneof content {
-// The Artifact metadata. The first message in a PutArtifact call must 
contain the name
-// of the artifact.
-ArtifactMetadata metadata = 1;
+// The first message in a PutArtifact call must contain this field.
+PutArtifactMetadata metadata = 1;
 
 // A chunk of the artifact. All messages after the first in a PutArtifact 
call must contain a
 // chunk.
 ArtifactChunk data = 2;
+
 
 Review comment:
   nit: remove whitespace


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105989)
Time Spent: 0.5h  (was: 20m)

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4308) Enforce ErrorProne analysis in runners-core project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4308.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Enforce ErrorProne analysis in runners-core project
> ---
>
> Key: BEAM-4308
> URL: https://issues.apache.org/jira/browse/BEAM-4308
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Scott Wegner
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: errorprone, starter
> Fix For: 2.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-core-java}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-core-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/core}} project.
> # In {{runners/core/build.gradle}}, add {{failOnWarning: true}} to the call 
> the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=105988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105988
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 25/May/18 19:40
Start Date: 25/May/18 19:40
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5489: [BEAM-4290] proto 
changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489#issuecomment-392160244
 
 
   CC: @jkff @herohde @lukecwik @robertwb @bsidhom @axelmagn @tweise @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105988)
Time Spent: 20m  (was: 10m)

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4290?focusedWorklogId=105987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105987
 ]

ASF GitHub Bot logged work on BEAM-4290:


Author: ASF GitHub Bot
Created on: 25/May/18 19:37
Start Date: 25/May/18 19:37
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #5489: [BEAM-4290] 
proto changes to support artifact_staging_id
URL: https://github.com/apache/beam/pull/5489
 
 
   Updating the proto to pass an artifact_staging_id to artifact staging 
service.
   
   Reference document: 
https://docs.google.com/document/d/12zNk3O2nhTB8Zmxw5U78qXrvlk5r42X8tqF248IDlpI/edit
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105987)
Time Spent: 10m
Remaining Estimate: 0h

> ArtifactStagingService that stages to a distributed filesystem
> --
>
> Key: BEAM-4290
> URL: https://issues.apache.org/jira/browse/BEAM-4290
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Using the job's staging directory from PipelineOptions.
> Physical layout on the distributed filesystem is TBD but it should allow for 
> arbitrary filenames and ideally for eventually avoiding uploading artifacts 
> that are already there.
> Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4411) Upgrade Jackson

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4411.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (b807847 -> 354af94)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b807847  Merge pull request #5484: [BEAM-4308] Enforce ErrorProne 
analysis in runners-core-java
 add 08bda41  [BEAM-4411] Update Jackson version to latest to pick up 
security fixes.
 add 354af94  [BEAM-4411] Update Jackson version to latest to pick up 
security fixes.

No new revisions were added by this update.

Summary of changes:
 build_rules.gradle   | 5 ++---
 pom.xml  | 5 ++---
 runners/spark/build.gradle   | 2 +-
 sdks/java/io/elasticsearch/build.gradle  | 2 +-
 sdks/java/io/hadoop-input-format/pom.xml | 1 -
 sdks/java/javadoc/build.gradle   | 4 ++--
 6 files changed, 8 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105986
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 19:34
Start Date: 25/May/18 19:34
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5485: [BEAM-4411] Update 
Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build_rules.gradle b/build_rules.gradle
index 0d00d1f3dd5..c55f1c57243 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -179,7 +179,7 @@ def netty_version = "4.1.8.Final"
 def grpc_google_common_protos = "0.1.9"
 def hamcrest_version = "1.3"
 def hadoop_version = "2.7.3"
-def jackson_version = "2.8.9"
+def jackson_version = "2.9.5"
 def spark_version = "2.3.0"
 def pubsub_grpc_version = "0.1.18"
 def apex_core_version = "3.7.0"
@@ -187,7 +187,6 @@ def apex_malhar_version = "3.4.0"
 def postgres_version = "9.4.1212.jre7"
 def jaxb_api_version = "2.2.12"
 def kafka_version = "1.0.0"
-def jackson_datatype_joda_version = "2.4.0"
 def quickcheck_version = "0.8"
 
 // A map of maps containing common libraries used per language. To use:
@@ -268,7 +267,7 @@ ext.library = [
 jackson_databind: 
"com.fasterxml.jackson.core:jackson-databind:$jackson_version",
 jackson_dataformat_cbor: 
"com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:$jackson_version",
 jackson_dataformat_yaml: 
"com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:$jackson_version",
-jackson_datatype_joda: 
"com.fasterxml.jackson.datatype:jackson-datatype-joda:$jackson_datatype_joda_version",
+jackson_datatype_joda: 
"com.fasterxml.jackson.datatype:jackson-datatype-joda:$jackson_version",
 jackson_module_scala: 
"com.fasterxml.jackson.module:jackson-module-scala_2.11:$jackson_version",
 jaxb_api: "javax.xml.bind:jaxb-api:$jaxb_api_version",
 joda_time: "joda-time:joda-time:2.4",
diff --git a/pom.xml b/pom.xml
index 920f6ff7ce5..8331d5627bb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -136,7 +136,7 @@
 -->
 2.7.3
 1.3
-2.8.9
+2.9.5
 3.0.1
 1.3.9-1
 2.4
@@ -185,7 +185,6 @@
 nothing
 0.20.0
 2.2.12
-2.4.0
 
 
 kubectl
@@ -1489,7 +1488,7 @@
   
 com.fasterxml.jackson.datatype
 jackson-datatype-joda
-${jackson-datatype-joda-version}
+${jackson.version}
   
 
   
diff --git a/runners/spark/build.gradle b/runners/spark/build.gradle
index b236173479e..56042cc0304 100644
--- a/runners/spark/build.gradle
+++ b/runners/spark/build.gradle
@@ -63,7 +63,7 @@ dependencies {
   shadow library.java.slf4j_api
   shadow library.java.joda_time
   shadow "io.dropwizard.metrics:metrics-core:3.1.2"
-  shadow "com.fasterxml.jackson.module:jackson-module-scala_2.11:2.8.9"
+  shadow library.java.jackson_module_scala
   provided library.java.spark_core
   provided library.java.spark_streaming
   provided library.java.spark_network_common
diff --git a/sdks/java/io/elasticsearch/build.gradle 
b/sdks/java/io/elasticsearch/build.gradle
index 27b68dfb792..7b5ff9f2c91 100644
--- a/sdks/java/io/elasticsearch/build.gradle
+++ b/sdks/java/io/elasticsearch/build.gradle
@@ -28,7 +28,7 @@ dependencies {
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.jackson_databind
   shadow library.java.findbugs_jsr305
-  shadow "com.fasterxml.jackson.core:jackson-annotations:2.8.9"
+  shadow library.java.jackson_annotations
   shadow "org.elasticsearch.client:elasticsearch-rest-client:5.6.3"
   shadow "org.apache.httpcomponents:httpasyncclient:4.1.2"
   shadow "org.apache.httpcomponents:httpcore-nio:4.4.5"
diff --git a/sdks/java/io/hadoop-input-format/pom.xml 
b/sdks/java/io/hadoop-input-format/pom.xml
index 1fba06d303f..846228dc009 100644
--- a/sdks/java/io/hadoop-input-format/pom.xml
+++ b/sdks/java/io/hadoop-input-format/pom.xml
@@ -96,7 +96,6 @@
 
   com.fasterxml.jackson.module
   jackson-module-scala_2.11
-  2.8.9
   runtime
 
   
diff --git a/sdks/java/javadoc/build.gradle b/sdks/java/javadoc/build.gradle
index 70fc711792c..e6f7c4dc006 100644
--- a/sdks/java/javadoc/build.gradle
+++ b/sdks/java/javadoc/build.gradle
@@ -101,8 +101,8 @@ task aggregateJavadoc(type: Javadoc) {
 linksOffline 
'https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/',
 'bq-docs'
 linksOffline 
'http://googlecloudplatform.github.io/google-cloud-java/0.8.0/apidocs/index.html',
 

[jira] [Comment Edited] (BEAM-4076) Schema followups

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438551#comment-16438551
 ] 

Kenneth Knowles edited comment on BEAM-4076 at 5/25/18 6:56 PM:


Filed these by browsing the big schema PR for potential followups. Please add 
any that you can think of that I didn't find or that have come up since then. 
[~kedin] [~reuvenlax] [~apilloud]


was (Author: kenn):
Filed these by browsing the big schema PR for potential followups. Please add 
any that you can think of that I didn't find or that have come up since then. 
[~kedin] [~reuvenlax] [~apilloud]]

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4077) Refactor builder field nullability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?focusedWorklogId=105982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105982
 ]

ASF GitHub Bot logged work on BEAM-4077:


Author: ASF GitHub Bot
Created on: 25/May/18 18:55
Start Date: 25/May/18 18:55
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #5488: 
[BEAM-4077] Schema fields are non-null by default
URL: https://github.com/apache/beam/pull/5488
 
 
   Before this change, every field was explicitly called out as nullable or 
not. It was inconvenient. This makes them non-null by default.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105982)
Time Spent: 10m
Remaining Estimate: 0h

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #554

[Apache Jenkins Server]

See 


--
[...truncated 17.66 MB...]
May 25, 2018 6:42:48 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <115879 bytes, hash OKT-Dbn6vi3luY-Ei29xyg> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0525184245-9e72e4ca/output/results/staging/pipeline-OKT-Dbn6vi3luY-Ei29xyg.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 25, 2018 6:42:50 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_11_42_49-10445865842264941750?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-25_11_42_49-10445865842264941750

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 25, 2018 6:42:50 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-25_11_42_49-10445865842264941750
May 25, 2018 6:42:50 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-25_11_42_49-10445865842264941750 with 0 
expected assertions.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:49.422Z: Autoscaling is enabled for job 
2018-05-25_11_42_49-10445865842264941750. The number of workers will be between 
1 and 1000.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:49.464Z: Autoscaling was automatically enabled for 
job 2018-05-25_11_42_49-10445865842264941750.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:51.928Z: Checking required Cloud APIs are enabled.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:52.248Z: Checking permissions granted to controller 
Service Account.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:57.647Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.064Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.336Z: Expanding GroupByKey operations into 
optimizable parts.
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.378Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.654Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.691Z: Elided trivial flatten 
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.738Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.767Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 25, 2018 6:43:06 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-25T18:42:58.805Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 25, 2018 6:43:06 PM 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #348

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4307] Enforce ErrorProne analysis in

--
[...truncated 18.06 MB...]
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 25, 2018 6:32:53 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 25, 2018 6:32:53 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525183249-9baa13ef/output/results/staging/
May 25, 2018 6:32:53 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash 0TBEjGSx74BBJOwXMN0s3g> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0525183249-9baa13ef/output/results/staging/pipeline-0TBEjGSx74BBJOwXMN0s3g.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 6:32:54 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-25_11_32_53-16943753811765757669?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-25_11_32_53-16943753811765757669

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 25, 2018 6:32:54 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 

[jira] [Work logged] (BEAM-4084) Find remaining uses of rowType and RowType, etc, and make them Schema as appropriate

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4084?focusedWorklogId=105977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105977
 ]

ASF GitHub Bot logged work on BEAM-4084:


Author: ASF GitHub Bot
Created on: 25/May/18 18:34
Start Date: 25/May/18 18:34
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5487: [BEAM-4084] Finish 
RowType -> Schema rename
URL: https://github.com/apache/beam/pull/5487#issuecomment-39214
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105977)
Time Spent: 20m  (was: 10m)

> Find remaining uses of rowType and RowType, etc, and make them Schema as 
> appropriate
> 
>
> Key: BEAM-4084
> URL: https://issues.apache.org/jira/browse/BEAM-4084
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4084) Find remaining uses of rowType and RowType, etc, and make them Schema as appropriate

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4084?focusedWorklogId=105976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105976
 ]

ASF GitHub Bot logged work on BEAM-4084:


Author: ASF GitHub Bot
Created on: 25/May/18 18:30
Start Date: 25/May/18 18:30
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #5487: 
[BEAM-4084] Finish RowType -> Schema rename
URL: https://github.com/apache/beam/pull/5487
 
 
   When RowType was moved to SDK core and renamed Schema there were a few 
holdout methods and classes. I think I got them all. Checked with grep. Calcite 
uses the term `rowType` and all the remaining occurrences are overrides of 
Calcite methods.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   R: @reuvenlax 
   CC: @akedin @apilloud 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105976)
Time Spent: 10m
Remaining Estimate: 0h

> Find remaining uses of rowType and RowType, etc, and make them Schema as 
> appropriate
> 
>
> Key: BEAM-4084
> URL: https://issues.apache.org/jira/browse/BEAM-4084
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4084) Find remaining uses of rowType and RowType, etc, and make them Schema as appropriate

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-4084:
-

Assignee: Kenneth Knowles

> Find remaining uses of rowType and RowType, etc, and make them Schema as 
> appropriate
> 
>
> Key: BEAM-4084
> URL: https://issues.apache.org/jira/browse/BEAM-4084
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #213

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4307] Enforce ErrorProne analysis in

[iemejia] [BEAM-4308] Enforce ErrorProne analysis in runners-core-java

[ammons] Say that merging windows are not supported in Distinct.java.

--
[...truncated 491.77 KB...]
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #219

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4307] Enforce ErrorProne analysis in

[iemejia] [BEAM-4308] Enforce ErrorProne analysis in runners-core-java

[ammons] Say that merging windows are not supported in Distinct.java.

--
[...truncated 199.27 KB...]
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=104.197.132.148:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 

[jira] [Assigned] (BEAM-4077) Refactor builder field nullability

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-4077:
-

Assignee: Kenneth Knowles

> Refactor builder field nullability
> --
>
> Key: BEAM-4077
> URL: https://issues.apache.org/jira/browse/BEAM-4077
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> Currently the Schema builder methods take a boolean for nullability. It would 
> be more standard to have separate builder methods. At this point the builder 
> might as well just take the Field spec since it does not add concision.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #214

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-4370) Cannot write to bigquery partitions using DynamicDestination or SerializableFunction<>

[Peter Brumblay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Brumblay closed BEAM-4370.

   Resolution: Not A Problem
Fix Version/s: Not applicable

> Cannot write to bigquery partitions using DynamicDestination or 
> SerializableFunction<>
> --
>
> Key: BEAM-4370
> URL: https://issues.apache.org/jira/browse/BEAM-4370
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Peter Brumblay
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: Not applicable
>
>
> When trying to use BigQueryIO.writeTableRows() with 
> DynamicDestinations or
> SerializableFunction, the 
> operation fails with an error message claiming that decorators are not 
> allowed:
> "Invalid table ID $. Table IDs must be alphanumeric (plus 
> underscores) and must be at most 1024 characters long. Also, Table decorators 
> cannot be used."
> This should work according to ...
>  # 
> [https://stackoverflow.com/questions/47351578/create-dynamic-side-outputs-in-apache-beam-dataflow#comment81668927_47351578]
>  and [https://stackoverflow.com/a/43505535]
>  # 
> [https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/]
> See the following for a complete example demonstrating the error with both 
> scenarios:
> [https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotWriteToPartitionedTable.java]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4370) Cannot write to bigquery partitions using DynamicDestination or SerializableFunction<>

[Peter Brumblay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491095#comment-16491095
 ] 

Peter Brumblay commented on BEAM-4370:
--

Ok. I have figured it  out through a lot of pain and suffering. This is NOT a 
bug in BigQueryIO.

The problem:
If you include a *malformed* table decorator, you get this error message from 
the BIGQUERY REST API:

"Invalid table ID \"merging_partitions$2018-03-03\". Table IDs must be 
alphanumeric (plus underscores) and must be at most 1024 characters long. Also, 
Table decorators cannot be used."

Note 2 things
1. The table decorator I'm trying to use is NOT in the correct format. it 
should NOT be $-MM-DD. It SHOULD be $MMDD.
2. I get an error message which is COMPLETELY misleading. Table decorators CAN 
and SHOULD be used when working with partitions. 

This is a working example: 
[https://gist.github.com/pbrumblay/b49bf08e0b652319120dfb1c861f0936]

Closing.

 

> Cannot write to bigquery partitions using DynamicDestination or 
> SerializableFunction<>
> --
>
> Key: BEAM-4370
> URL: https://issues.apache.org/jira/browse/BEAM-4370
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Peter Brumblay
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: Not applicable
>
>
> When trying to use BigQueryIO.writeTableRows() with 
> DynamicDestinations or
> SerializableFunction, the 
> operation fails with an error message claiming that decorators are not 
> allowed:
> "Invalid table ID $. Table IDs must be alphanumeric (plus 
> underscores) and must be at most 1024 characters long. Also, Table decorators 
> cannot be used."
> This should work according to ...
>  # 
> [https://stackoverflow.com/questions/47351578/create-dynamic-side-outputs-in-apache-beam-dataflow#comment81668927_47351578]
>  and [https://stackoverflow.com/a/43505535]
>  # 
> [https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/]
> See the following for a complete example demonstrating the error with both 
> scenarios:
> [https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotWriteToPartitionedTable.java]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #310

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-4307] Enforce ErrorProne analysis in

[iemejia] [BEAM-4308] Enforce ErrorProne analysis in runners-core-java

[ammons] Say that merging windows are not supported in Distinct.java.

--
[...truncated 106.00 KB...]
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 6,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:compileJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-hadoop-input-format:compileJava' 
is ea3d80da0ff8871be19663f7049ffa85
Skipping task ':beam-sdks-java-io-hadoop-input-format:compileJava' as it is 
up-to-date.
:beam-sdks-java-io-hadoop-input-format:compileJava (Thread[Task worker for ':' 
Thread 15,5,main]) completed. Took 0.027 secs.
:beam-sdks-java-io-hadoop-input-format:classes (Thread[Task worker for ':' 
Thread 15,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:classes UP-TO-DATE
Skipping task ':beam-sdks-java-io-hadoop-input-format:classes' as it has no 
actions.
:beam-sdks-java-io-hadoop-input-format:classes (Thread[Task worker for ':' 
Thread 15,5,main]) completed. Took 0.0 secs.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' is 
6a2507576764a0a599e98c0f3a45d6a4
Caching disabled for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar': Caching 
has not been enabled for the task
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 6,5,main]) completed. Took 0.016 secs.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 6,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
1fff1b8bb08220ac011b01c8a6549817
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 6,5,main]) completed. Took 0.049 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 6,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 6,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 6,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
bb7a74337b69a2e0e8dea7bfb4168ec4
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 6,5,main]) completed. Took 0.03 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 6,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
3b44f2213773048cf7c532f181760527
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 6,5,main]) completed. Took 0.054 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 6,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 6,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 6,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
35b893aa0c0aa77d63bbe1f0778fbb0a
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.

[jira] [Work logged] (BEAM-4406) Increment major version used by Dataflow runner for portability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4406?focusedWorklogId=105974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105974
 ]

ASF GitHub Bot logged work on BEAM-4406:


Author: ASF GitHub Bot
Created on: 25/May/18 18:04
Start Date: 25/May/18 18:04
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5486: [BEAM-4406] Updating 
portable Dataflow major version numbers
URL: https://github.com/apache/beam/pull/5486#issuecomment-392136504
 
 
   Run Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105974)
Time Spent: 20m  (was: 10m)

> Increment major version used by Dataflow runner for portability
> ---
>
> Key: BEAM-4406
> URL: https://issues.apache.org/jira/browse/BEAM-4406
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The major version for portable pipelines in Java and Python is set to 1. I'd 
> like to set them to 7 so they have parity with the major version of native 
> pipelines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105975
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 18:04
Start Date: 25/May/18 18:04
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5485: 
[BEAM-4411] Update Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#discussion_r190969772
 
 

 ##
 File path: sdks/java/io/hadoop-input-format/pom.xml
 ##
 @@ -96,7 +96,6 @@
 
   com.fasterxml.jackson.module
   jackson-module-scala_2.11
-  2.8.9
   runtime
 
 Review comment:
   Cool. SGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105975)
Time Spent: 1h 20m  (was: 1h 10m)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105972
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 18:03
Start Date: 25/May/18 18:03
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5485: [BEAM-4411] Update 
Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#issuecomment-392136175
 
 
   Run Apex ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105972)
Time Spent: 1h 10m  (was: 1h)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105969
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 18:02
Start Date: 25/May/18 18:02
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5485: 
[BEAM-4411] Update Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#discussion_r190969051
 
 

 ##
 File path: sdks/java/io/hadoop-input-format/pom.xml
 ##
 @@ -96,7 +96,6 @@
 
   com.fasterxml.jackson.module
   jackson-module-scala_2.11
-  2.8.9
   runtime
 
 Review comment:
   It is inherited from the root pom.xml here:
   
https://github.com/apache/beam/blob/b807847455e5fb1b2fbdd5285a2f696960ab754f/pom.xml#L1275
   
   Anytime someone adds it explicitly, it overrides the root but in this case 
it is a duplicate of what was defined in the root pom.xml.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105969)
Time Spent: 40m  (was: 0.5h)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105970
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 18:02
Start Date: 25/May/18 18:02
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5485: [BEAM-4411] Update 
Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#issuecomment-392136096
 
 
   Run Spark ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105970)
Time Spent: 50m  (was: 40m)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105971
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 18:02
Start Date: 25/May/18 18:02
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5485: [BEAM-4411] Update 
Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#issuecomment-392136139
 
 
   Run Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105971)
Time Spent: 1h  (was: 50m)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105968
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 18:01
Start Date: 25/May/18 18:01
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5485: 
[BEAM-4411] Update Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#discussion_r190969051
 
 

 ##
 File path: sdks/java/io/hadoop-input-format/pom.xml
 ##
 @@ -96,7 +96,6 @@
 
   com.fasterxml.jackson.module
   jackson-module-scala_2.11
-  2.8.9
   runtime
 
 Review comment:
   It is inherited from the root pom.xml here:
   
https://github.com/apache/beam/blob/b807847455e5fb1b2fbdd5285a2f696960ab754f/pom.xml#L1275


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105968)
Time Spent: 0.5h  (was: 20m)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4411) Upgrade Jackson

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4411?focusedWorklogId=105964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105964
 ]

ASF GitHub Bot logged work on BEAM-4411:


Author: ASF GitHub Bot
Created on: 25/May/18 17:49
Start Date: 25/May/18 17:49
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5485: 
[BEAM-4411] Update Jackson version to latest to pick up security fixes.
URL: https://github.com/apache/beam/pull/5485#discussion_r190966001
 
 

 ##
 File path: sdks/java/io/hadoop-input-format/pom.xml
 ##
 @@ -96,7 +96,6 @@
 
   com.fasterxml.jackson.module
   jackson-module-scala_2.11
-  2.8.9
   runtime
 
 Review comment:
   Is the version here somehow 'inherited' from the top level pom? Or why don't 
we have it here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105964)
Time Spent: 20m  (was: 10m)

> Upgrade Jackson
> ---
>
> Key: BEAM-4411
> URL: https://issues.apache.org/jira/browse/BEAM-4411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Update Jackson library to latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4406) Increment major version used by Dataflow runner for portability

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4406?focusedWorklogId=105963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105963
 ]

ASF GitHub Bot logged work on BEAM-4406:


Author: ASF GitHub Bot
Created on: 25/May/18 17:43
Start Date: 25/May/18 17:43
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #5486: [BEAM-4406] 
Updating portable Dataflow major version numbers
URL: https://github.com/apache/beam/pull/5486
 
 
   Updating the major version numbers used for portable Dataflow pipelines in 
the Java and Python SDKs to match the version for native pipelines.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105963)
Time Spent: 10m
Remaining Estimate: 0h

> Increment major version used by Dataflow runner for portability
> ---
>
> Key: BEAM-4406
> URL: https://issues.apache.org/jira/browse/BEAM-4406
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The major version for portable pipelines in Java and Python is set to 1. I'd 
> like to set them to 7 so they have parity with the major version of native 
> pipelines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4400) Integrate Jenkins Job Cacher Plugin for improved build caching

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491028#comment-16491028
 ] 

Luke Cwik commented on BEAM-4400:
-

How do we ensure that tasks correctly label their inputs and outputs so that 
the task cache correctly says that a task doesn't need to run?

> Integrate Jenkins Job Cacher Plugin for improved build caching
> --
>
> Key: BEAM-4400
> URL: https://issues.apache.org/jira/browse/BEAM-4400
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Scott Wegner
>Priority: Minor
>
> See discussion on 
> [dev|https://lists.apache.org/thread.html/c59289787c665c7732df4095bda0877637eee59e3add4d26a11f6b7f@%3Cdev.beam.apache.org%3E].
>  With the new Gradle build, we are now able to build incrementally, but are 
> not yet taking advantage of it on Jenkins. This would give us the benefit of 
> much faster pre-commits for files that aren't changed. For example, A change 
> in the Python which doesn't touch any Java would not need to re-run Java 
> pre-commits.
> By default, Gradle uses a build cache local to the workspace, and in Jenkins 
> the workspace gets nuked on each build. There is a [Jenkins Job Cacher 
> Plugin|https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin] which 
> solves this exact plugin and integrates with Gradle's build cache support. I 
> believe all we need to do is enable and configure this plugin to realize the 
> benefits of incremental builds.
> To enable the plugin, we likely need to reach out the Apache INFRA team.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)