[jira] [Commented] (BEAM-2980) BagState.isEmpty needs a tighter spec

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488425#comment-16488425
 ] 

Raghu Angadi commented on BEAM-2980:


Is there a pointer to rationale for existence of {{isEmpty()}}? 

> BagState.isEmpty needs a tighter spec
> -
>
> Key: BEAM-2980
> URL: https://issues.apache.org/jira/browse/BEAM-2980
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Daniel Mills
>Priority: Major
>
> Consider the following:
> {code}
> BagState myBag = // empty
> ReadableState isMyBagEmpty = myBag.isEmpty();
> myBag.add(bizzle);
> bool empty = isMyBagEmpty.read();
> {code}
> Should {{empty}} be true or false? We need a consistent answer, across all 
> kinds of state, when snapshots are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #333

[Apache Jenkins Server]

See 


Changes:

[klk] Autoformat sdks/java/extensions/sql

[klk] Add spotless config to global gradle file

[klk] Activate spotless for SQL module

--
[...truncated 17.99 MB...]
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 24, 2018 5:01:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 24, 2018 5:01:47 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524050142-7d26c29f/output/results/staging/
May 24, 2018 5:01:47 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash HfF9Xp8MuI2tMrGID6KaWg> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524050142-7d26c29f/output/results/staging/pipeline-HfF9Xp8MuI2tMrGID6KaWg.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 24, 2018 5:01:48 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #332

[Apache Jenkins Server]

See 


Changes:

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 18.01 MB...]
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 24, 2018 2:55:05 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 24, 2018 2:55:05 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524025500-8f305c81/output/results/staging/
May 24, 2018 2:55:05 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash SHq4GJySDcL1H-u00k9GJA> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524025500-8f305c81/output/results/staging/pipeline-SHq4GJySDcL1H-u00k9GJA.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 24, 2018 2:55:07 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #101

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105427=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105427
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 24/May/18 02:16
Start Date: 24/May/18 02:16
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5428: [BEAM-4167] 
Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#issuecomment-391565591
 
 
   Autoformat and permanent enforcement are now submitted prior to this PR and 
I have rebased it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105427)
Time Spent: 4h  (was: 3h 50m)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (e8ca1e1 -> d110f6b)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e8ca1e1  Merge pull request #5459: Use WindowOnlyContext in 
ReduceFnContextFactory
 add 5b64f73  Autoformat sdks/java/extensions/sql
 add ca4dc98  Add spotless config to global gradle file
 add b99d3e9  Activate spotless for SQL module
 add d110f6b  Merge pull request #5458: Establish and enforce exact Java 
format in sdks/java/extensions/sql

No new revisions were added by this update.

Summary of changes:
 build_rules.gradle |  14 +-
 sdks/java/extensions/sql/build.gradle  |   2 +-
 .../apache/beam/sdk/extensions/sql/BeamSql.java|  84 ++--
 .../apache/beam/sdk/extensions/sql/BeamSqlCli.java |  22 +-
 .../sdk/extensions/sql/BeamSqlSeekableTable.java   |  22 +-
 .../beam/sdk/extensions/sql/BeamSqlTable.java  |  26 +-
 .../apache/beam/sdk/extensions/sql/BeamSqlUdf.java |  12 +-
 .../beam/sdk/extensions/sql/QueryTransform.java|  58 ++-
 .../beam/sdk/extensions/sql/RowSqlTypes.java   |  64 ++-
 .../beam/sdk/extensions/sql/SchemaHelper.java  |  33 +-
 .../sdk/extensions/sql/example/BeamSqlExample.java |  33 +-
 .../extensions/sql/example/BeamSqlPojoExample.java |  72 ++--
 .../sdk/extensions/sql/example/model/Customer.java |   4 +-
 .../sdk/extensions/sql/example/model/Order.java|   4 +-
 .../extensions/sql/example/model/package-info.java |   4 +-
 .../sdk/extensions/sql/example/package-info.java   |   4 +-
 .../sdk/extensions/sql/impl/BeamCalciteSchema.java |   4 +-
 .../sql/impl/BeamCalciteSchemaFactory.java |  10 +-
 .../sdk/extensions/sql/impl/BeamCalciteTable.java  |   4 +-
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |  12 +-
 .../beam/sdk/extensions/sql/impl/JdbcDriver.java   |  25 +-
 .../interpreter/BeamSqlExpressionExecutor.java |  14 +-
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  76 ++--
 .../operator/BeamSqlCaseExpression.java|  18 +-
 .../operator/BeamSqlCastExpression.java| 100 ++---
 .../operator/BeamSqlDefaultExpression.java |   5 +-
 .../interpreter/operator/BeamSqlDotExpression.java |   7 +-
 .../interpreter/operator/BeamSqlExpression.java|  15 +-
 .../operator/BeamSqlInputRefExpression.java|   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java | 102 +++--
 .../interpreter/operator/BeamSqlUdfExpression.java |  17 +-
 .../operator/BeamSqlWindowEndExpression.java   |   5 +-
 .../operator/BeamSqlWindowExpression.java  |   8 +-
 .../operator/BeamSqlWindowStartExpression.java |   9 +-
 .../sql/impl/interpreter/operator/UdafImpl.java|  12 +-
 .../arithmetic/BeamSqlArithmeticExpression.java|  17 +-
 .../arithmetic/BeamSqlDivideExpression.java|   7 +-
 .../arithmetic/BeamSqlMinusExpression.java |   7 +-
 .../operator/arithmetic/BeamSqlModExpression.java  |   7 +-
 .../arithmetic/BeamSqlMultiplyExpression.java  |   7 +-
 .../operator/arithmetic/BeamSqlPlusExpression.java |   7 +-
 .../operator/arithmetic/package-info.java  |   4 +-
 .../operator/array/BeamSqlArrayExpression.java |  11 +-
 .../operator/array/BeamSqlArrayItemExpression.java |   8 +-
 .../interpreter/operator/array/package-info.java   |   4 +-
 .../collection/BeamSqlCardinalityExpression.java   |   8 +-
 .../collection/BeamSqlSingleElementExpression.java |  12 +-
 .../operator/collection/package-info.java  |   4 +-
 .../comparison/BeamSqlCompareExpression.java   |  62 ++-
 .../comparison/BeamSqlEqualsExpression.java|  10 +-
 .../comparison/BeamSqlGreaterThanExpression.java   |  10 +-
 .../BeamSqlGreaterThanOrEqualsExpression.java  |  10 +-
 .../comparison/BeamSqlIsNotNullExpression.java |   8 +-
 .../comparison/BeamSqlIsNullExpression.java|   8 +-
 .../comparison/BeamSqlLessThanExpression.java  |  10 +-
 .../BeamSqlLessThanOrEqualsExpression.java |  10 +-
 .../comparison/BeamSqlNotEqualsExpression.java |  10 +-
 .../operator/comparison/package-info.java  |   4 +-
 .../date/BeamSqlCurrentDateExpression.java |   6 +-
 .../date/BeamSqlCurrentTimeExpression.java |   6 +-
 .../date/BeamSqlCurrentTimestampExpression.java|   6 +-
 .../operator/date/BeamSqlDateCeilExpression.java   |   9 +-
 .../operator/date/BeamSqlDateFloorExpression.java  |   9 +-
 .../date/BeamSqlDatetimeMinusExpression.java   |  15 +-
 .../date/BeamSqlDatetimePlusExpression.java|  49 ++-
 .../operator/date/BeamSqlExtractExpression.java|  42 +-
 .../date/BeamSqlIntervalMultiplyExpression.java|  35 +-
 .../BeamSqlTimestampMinusIntervalExpression.java   |   6 +-
 .../BeamSqlTimestampMinusTimestampExpression.java  |  21 +-
 .../interpreter/operator/date/TimeUnitUtils.java   |   8 +-
 .../interpreter/operator/date/package-info.java|   4 +-
 

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105417
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 24/May/18 01:12
Start Date: 24/May/18 01:12
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391555771
 
 
   Hey Luck @lukecwik , commit 
https://github.com/apache/beam/pull/5457/commits/a0945b5d7efddb149f43f06e68eaf4f22577b2ed
 addressed exclude all problems. 
   
   Followings are the shadowJar content:
   ```
   META-INF/
   META-INF/MANIFEST.MF
   META-INF/services/
   org/
   org/apache/
   org/apache/beam/
   org/apache/beam/sdk/
   org/apache/beam/sdk/io/
   org/apache/beam/sdk/io/gcp/
   org/apache/beam/sdk/io/gcp/datastore/
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$DeleteKey.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$UpsertFn.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$DeleteKeyFn.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Read$Builder.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreIO.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$DeleteEntity.class
   org/apache/beam/sdk/io/gcp/datastore/AdaptiveThrottler.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Read$ReadFn.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$DatastoreWriterFn.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Read$SplitQueryFn.class
   org/apache/beam/sdk/io/gcp/datastore/AutoValue_DatastoreV1_Read$1.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$DeleteEntityFn.class
   org/apache/beam/sdk/io/gcp/datastore/AutoValue_DatastoreV1_Read$Builder.class
   org/apache/beam/sdk/io/gcp/datastore/AutoValue_DatastoreV1_Read.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$WriteBatcherImpl.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Write.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$V1DatastoreFactory.class
   org/apache/beam/sdk/io/gcp/datastore/MovingAverage.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Read$V1Options.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Mutate.class
   
org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Read$GqlQueryTranslateFn.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$Read.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1$WriteBatcher.class
   org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.class
   org/apache/beam/sdk/io/gcp/bigtable/
   org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig$Builder.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableService$Writer.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableIO_Read.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableService$Reader.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$BigtableWriteException.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableIO_Read$1.class
   
org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl$BigtableReaderImpl.class
   
org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl$BigtableWriterImpl$1.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$BigtableReader.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableIO_Write.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$Read$Builder.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$Write$BigtableWriterFn.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableIO_Read$Builder.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$BigtableSource.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$Write$Builder.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$Read.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableConfig$1.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO$Write.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.class
   
org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl$BigtableWriterImpl.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableConfig$Builder.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableConfig.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableIO_Write$Builder.class
   org/apache/beam/sdk/io/gcp/bigtable/AutoValue_BigtableIO_Write$1.class
   org/apache/beam/sdk/io/gcp/bigtable/BigtableService.class
   org/apache/beam/sdk/io/gcp/bigquery/
   
org/apache/beam/sdk/io/gcp/bigquery/WriteTables$GarbageCollectTemporaryFiles.class
   
org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers$TableSchemaToJsonSchema.class
   org/apache/beam/sdk/io/gcp/bigquery/StreamingInserts.class
   

[jira] [Assigned] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[Boyuan Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyuan Zhang reassigned BEAM-4328:
--

Assignee: Boyuan Zhang  (was: Luke Cwik)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #331

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

--
[...truncated 18.02 MB...]
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 24, 2018 12:38:47 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 24, 2018 12:38:47 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524003841-6d509008/output/results/staging/
May 24, 2018 12:38:47 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash EqLlcOXx18uXnRny_t5QLA> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0524003841-6d509008/output/results/staging/pipeline-EqLlcOXx18uXnRny_t5QLA.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 24, 2018 12:38:48 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 

[jira] [Work logged] (BEAM-4393) Missing copyright notices for shaded packages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?focusedWorklogId=105415=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105415
 ]

ASF GitHub Bot logged work on BEAM-4393:


Author: ASF GitHub Bot
Created on: 24/May/18 00:29
Start Date: 24/May/18 00:29
Worklog Time Spent: 10m 
  Work Description: boyuanzz opened a new pull request #5461: [BEAM-4393] 
:Appending lacked LICENSE of shaded dependencies
URL: https://github.com/apache/beam/pull/5461
 
 
   Appending lacked LICENSE of shaded dependencies:
   
   Lacked LICENSE and source:
   protobuf: https://github.com/google/protobuf/blob/master/LICENSE
   jsr-305: https://github.com/amaembo/jsr-305/blob/master/ri/LICENSE
   janino: https://github.com/janino-compiler/janino/blob/master/LICENSE
   
   r: @lukecwik 
   cc: @apilloud 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105415)
Time Spent: 10m
Remaining Estimate: 0h

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #527

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #205

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 432.58 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
xmlioit0writethenreadall--05231705-4i51-harness-vmdz.c.apache-beam-testing.internal/10.128.0.43
 to 163.202.232.35.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #211

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 286.32 KB...]
java.lang.RuntimeException: com.mongodb.MongoTimeoutException: Timed out 
after 3 ms while waiting for a server that matches 
ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster 
state is {type=UNKNOWN, servers=[{address=35.188.199.191:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:275)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:197)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:181)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:160)
at 
com.google.cloud.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:77)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=35.188.199.191:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.SocketTimeoutException: connect timed out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #206

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 365.55 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05231706-mrdn-harness-rwm0.c.apache-beam-testing.internal/10.128.0.39
 to 229.87.226.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #206

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #213

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 378.35 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05231706-b4gp-harness-0n3f.c.apache-beam-testing.internal/10.128.0.41
 to 22.189.226.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
  

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #302

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 104.44 KB...]
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' is 
aa6c1658657792e74d8e11cce1cae372
Caching disabled for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar': Caching 
has not been enabled for the task
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.018 secs.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 2,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
39ee69ada607930be539303d83f0433c
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 2,5,main]) completed. Took 0.053 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 2,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 2,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 10,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
ee02f8db8f7229826c2e7b143a689843
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.024 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
c202f18cbf4d613cdcdf03300e8eb330
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 10,5,main]) completed. Took 0.043 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
69661b2d869fe9977a3c8e4250f9c6c8
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.026 secs.
:beam-sdks-java-io-hadoop-input-format:compileTestJava (Thread[Task worker for 
':' Thread 10,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-hadoop-input-format:compileTestJava' is 
f8491e8f0242ee02d16b4b0936693654
Skipping task ':beam-sdks-java-io-hadoop-input-format:compileTestJava' as it is 
up-to-date.
:beam-sdks-java-io-hadoop-input-format:compileTestJava (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.36 secs.
:beam-sdks-java-io-hadoop-input-format:testClasses (Thread[Task worker for ':' 
Thread 10,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:testClasses UP-TO-DATE
Skipping task 

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #100

[Apache Jenkins Server]

See 


Changes:

[tgroh] Use WindowOnlyContext in ReduceFnContextFactory

--
[...truncated 186.22 KB...]
"user_name": "assert:odd/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:odd/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s30"
}, 
"serialized_fn": "", 
"user_name": "assert:odd/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s32", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:odd/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s31"
}, 
"serialized_fn": "", 
"user_name": "assert:odd/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s33", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105413
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:47
Start Date: 23/May/18 23:47
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190431619
 
 

 ##
 File path: runners/java-fn-execution/build.gradle
 ##
 @@ -24,6 +24,7 @@ dependencies {
   compile library.java.guava
   compile library.java.findbugs_annotations
   compile project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
 
 Review comment:
   Done.
   
   I can't find any documentation at all about the `provided` dependency scope 
in Gradle - every time I google it it sends me to `compileOnly`. This is a 
result of our `build_rules.gradle`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105413)
Time Spent: 15h 20m  (was: 15h 10m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 15h 20m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4393) Missing copyright notices for shaded packages

[Boyuan Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyuan Zhang reassigned BEAM-4393:
--

Assignee: Boyuan Zhang

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Assignee: Boyuan Zhang
>Priority: Major
> Fix For: 2.5.0
>
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5459: Use WindowOnlyContext in ReduceFnContextFactory

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e8ca1e17851f36f955f2dd497db935f2c14b1114
Merge: e31f27f 773583d
Author: Thomas Groh 
AuthorDate: Wed May 23 16:36:03 2018 -0700

Merge pull request #5459: Use WindowOnlyContext in ReduceFnContextFactory

 .../main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (e31f27f -> e8ca1e1)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e31f27f  Merge pull request #5422 from timrobertson100/BEAM-4347
 add 773583d  Use WindowOnlyContext in ReduceFnContextFactory
 new e8ca1e1  Merge pull request #5459: Use WindowOnlyContext in 
ReduceFnContextFactory

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105410
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 23:29
Start Date: 23/May/18 23:29
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391534219
 
 
   Thanks for getting to the bottom of `exclude(".*")`! I'll rebase on this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105410)
Time Spent: 1h 50m  (was: 1h 40m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105407=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105407
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 23:22
Start Date: 23/May/18 23:22
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391531500
 
 
   Hey Luke @lukecwik , commit 
https://github.com/apache/beam/pull/5457/commits/19bfd82863eb3b9c4bb008677c581fbc457a0f42
 addressed harness and build_rule.gradle comments, please review it.
   I'll take a look at whether build scripts using exclude(".*") and change 
them together. Thanks for your comments!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105407)
Time Spent: 1h 40m  (was: 1.5h)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105406
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:20
Start Date: 23/May/18 23:20
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190427412
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
+PipelineOptionsFactory.create(),
+loggingServer.getApiServiceDescriptor(),
+controlServer.getApiServiceDescriptor(),
+InProcessManagedChannelFactory.create(),
+StreamObserverFactory.direct()));
+executor.submit(
+() -> {
+  try {
+fnHarness.get();
+  } catch (Throwable t) {
+executor.shutdownNow();
+  }
+});
+
+// TODO: find some way to populate the actual ID in FnHarness.main()
 
 Review comment:
   I see what you're saying. It makes sense, but I'm not in favor of providing 
the same value in two different ways.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105403
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:11
Start Date: 23/May/18 23:11
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190425499
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -64,6 +64,7 @@ dependencies {
   shadow library.java.slf4j_api
 
 Review comment:
   I still think we want to use `provided` for `beam-sdks-java-harness` so that 
it is correctly added to the generated maven pom.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105403)
Time Spent: 14h 50m  (was: 14h 40m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 14h 50m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105404
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:11
Start Date: 23/May/18 23:11
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190425600
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java
 ##
 @@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.collect.Iterables.getOnlyElement;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.Iterables;
+import java.io.Serializable;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.InProcessServerFactory;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.Impulse;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.joda.time.Instant;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link RemoteStageEvaluatorFactory}. */
+@RunWith(JUnit4.class)
+public class RemoteStageEvaluatorFactoryTest implements Serializable {
+
+  private transient RemoteStageEvaluatorFactory factory;
+  private transient ExecutorService executor;
+  private transient GrpcFnServer dataServer;
 
 Review comment:
   Got it


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105404)
Time Spent: 14h 50m  (was: 14h 40m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105405=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105405
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:11
Start Date: 23/May/18 23:11
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190425759
 
 

 ##
 File path: runners/java-fn-execution/build.gradle
 ##
 @@ -24,6 +24,7 @@ dependencies {
   compile library.java.guava
   compile library.java.findbugs_annotations
   compile project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
 
 Review comment:
   Please use `provided`
   
   Don't we need a corresponding change to the pom.xml?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105405)
Time Spent: 15h  (was: 14h 50m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 15h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105402
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:06
Start Date: 23/May/18 23:06
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190425270
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -58,12 +58,14 @@ dependencies {
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
   compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
 
 Review comment:
   We have been using `provided` scope in other places and this gets correctly 
translated when producing the maven jar.
   
   Your correct in that we should migrate to `compileOnly` as the provided 
scope but until that happens please stick with `provided` for now. I filed 
https://issues.apache.org/jira/browse/BEAM-4395 for the migration to 
`compileOnly`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105402)
Time Spent: 14h 40m  (was: 14.5h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 14h 40m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4395) Gradle defines the `compileOnly` scope which is the Gradle way of doing `provided`

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-4395:
---

 Summary: Gradle defines the `compileOnly` scope which is the 
Gradle way of doing `provided`
 Key: BEAM-4395
 URL: https://issues.apache.org/jira/browse/BEAM-4395
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Luke Cwik


It would be wise to use the `compileOnly` capability that is part of Gradle 
instead of relying on the `propdeps` and `propdeps-maven` plugins.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=105400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105400
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 23:01
Start Date: 23/May/18 23:01
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190424525
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
+PipelineOptionsFactory.create(),
+loggingServer.getApiServiceDescriptor(),
+controlServer.getApiServiceDescriptor(),
+InProcessManagedChannelFactory.create(),
+StreamObserverFactory.direct()));
+executor.submit(
+() -> {
+  try {
+fnHarness.get();
+  } catch (Throwable t) {
+executor.shutdownNow();
+  }
+});
+
+// TODO: find some way to populate the actual ID in FnHarness.main()
 
 Review comment:
   The point I'm trying to make is that the ApiServiceDescriptor is meant to 
describe who and how your connecting to an endpoint. I can foresee that in the 
future you would specify things like credentials inside this 
ApiServiceDescriptor. Since talking to the service requires a client id, I felt 
as 

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=105398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105398
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 23/May/18 23:00
Start Date: 23/May/18 23:00
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5336: 
[BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r190423680
 
 

 ##
 File path: sdks/python/apache_beam/utils/windowed_value.py
 ##
 @@ -178,33 +182,14 @@ def __repr__(self):
 self.windows,
 self.pane_info)
 
-  def __hash__(self):
-return (hash(self.value) +
-3 * self.timestamp_micros +
-7 * hash(self.windows) +
-11 * hash(self.pane_info))
-
-  # We'd rather implement __eq__, but Cython supports that via __richcmp__
-  # instead.  Fortunately __cmp__ is understood by both (but not by Python 3).
-  def __cmp__(left, right):  # pylint: disable=no-self-argument
-"""Compares left and right for equality.
-
-For performance reasons, doesn't actually impose an ordering
-on unequal values (always returning 1).
-"""
-if type(left) is not type(right):
-  return cmp(type(left), type(right))
+  def _key(self):
+return self.value, self.timestamp_micros, self.windows, self.pane_info
 
-# TODO(robertwb): Avoid the type checks?
-# Returns False (0) if equal, and True (1) if not.
-return not WindowedValue._typed_eq(left, right)
+  def __eq__(self, other):
+return type(self) == type(other) and self._key() == other._key()
 
-  @staticmethod
-  def _typed_eq(left, right):
-return (left.timestamp_micros == right.timestamp_micros
-and left.value == right.value
-and left.windows == right.windows
-and left.pane_info == right.pane_info)
+  def __hash__(self):
+return hash((type(self),) + self._key())
 
 Review comment:
   I think hash(self._key()) would be sufficient.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105398)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=105397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105397
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 23/May/18 23:00
Start Date: 23/May/18 23:00
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5336: 
[BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r190419324
 
 

 ##
 File path: sdks/python/apache_beam/utils/timestamp.py
 ##
 @@ -143,11 +148,35 @@ def __int__(self):
 # Note that the returned value may have lost precision.
 return self.micros // 100
 
-  def __cmp__(self, other):
-# Allow comparisons between Duration and Timestamp values.
+  def __eq__(self, other):
 
 Review comment:
   How about we keep `__eq__` and `__lt__`, remove the rest and decorate the 
class with `@functools.total_ordering`?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105397)
Time Spent: 1h  (was: 50m)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=105395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105395
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 23/May/18 23:00
Start Date: 23/May/18 23:00
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5336: 
[BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r190419116
 
 

 ##
 File path: sdks/python/apache_beam/utils/timestamp.py
 ##
 @@ -143,11 +148,35 @@ def __int__(self):
 # Note that the returned value may have lost precision.
 return self.micros // 100
 
-  def __cmp__(self, other):
-# Allow comparisons between Duration and Timestamp values.
 
 Review comment:
   Let's keep this comment to explain the reason for the casting.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105395)
Time Spent: 40m  (was: 0.5h)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=105396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105396
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 23/May/18 23:00
Start Date: 23/May/18 23:00
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5336: 
[BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r190424036
 
 

 ##
 File path: sdks/python/apache_beam/utils/profiler.py
 ##
 @@ -20,21 +20,19 @@
 For internal use only; no backwards-compatibility guarantees.
 """
 
-import cProfile
+from __future__ import absolute_import
+
+import cProfile  # pylint: disable=bad-python3-import
 
 Review comment:
   For my education, why does py3 lint complain here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105396)
Time Spent: 50m  (was: 40m)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105394
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 22:56
Start Date: 23/May/18 22:56
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391526356
 
 
   It seems like 
https://github.com/apache/beam/blob/4670344e5d9137020e128979a9154ee0794a5653/sdks/java/io/google-cloud-platform/build.gradle#L27
 is broken as well with the wrong shadowClosure and should instead exclude all. 
It is unclear what `.*` is actually applying to (if anything) and the 
`exclude("io/netty")` seems redundant.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105394)
Time Spent: 1.5h  (was: 1h 20m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105391
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422565
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServiceTest.java
 ##
 @@ -6,17 +6,17 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
  * http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: whitespace added to license header


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105391)
Time Spent: 4h  (was: 3h 50m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105384
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422540
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalServiceTest.java
 ##
 @@ -14,9 +15,11 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: whitespace added to license header


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105384)
Time Spent: 3h 20m  (was: 3h 10m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105383
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422545
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerServiceTest.java
 ##
 @@ -14,9 +15,11 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: whitespace added to license header


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105383)
Time Spent: 3h 20m  (was: 3h 10m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105388
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422510
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocationTest.java
 ##
 @@ -14,9 +15,11 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: whitespace added to license header


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105388)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105390
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190421992
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
 ##
 @@ -14,9 +15,11 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: remove added whitespace


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105390)
Time Spent: 3h 50m  (was: 3h 40m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105385
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422159
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
 ##
 @@ -14,10 +14,11 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: remove added whitespace


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105385)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105386
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422404
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -57,12 +57,14 @@ dependencies {
   compile project(path: ":beam-runners-core-java", configuration: "shadow")
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
+  compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api
   shadow library.java.hamcrest_core
   shadow library.java.junit
+  shadow library.java.args4j
 
 Review comment:
   It was the job server that had it which you deleted below.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105386)
Time Spent: 3.5h  (was: 3h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105387
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190421966
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
 ##
 @@ -1,4 +1,5 @@
 /*
+ *
 
 Review comment:
   nit: remove added whitespace


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105387)
Time Spent: 3.5h  (was: 3h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=105389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105389
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 23/May/18 22:51
Start Date: 23/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190422084
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/package-info.java
 ##
 @@ -14,9 +15,9 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   nit: remove added whitespace


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105389)
Time Spent: 3h 40m  (was: 3.5h)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4394) Consider enabling spotless java format throughout codebase

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-4394:
-

 Summary: Consider enabling spotless java format throughout codebase
 Key: BEAM-4394
 URL: https://issues.apache.org/jira/browse/BEAM-4394
 Project: Beam
  Issue Type: New Feature
  Components: build-system
Reporter: Kenneth Knowles


"Spotless" can enforce - and automatically restore - automatic Java formatting. 
Whenever formatting is off, it tells a user the exact command to fix it.

It isn't (just) about code layout, it is about automation. We have pretty 
strict style rules enforced by checkstyle. The most efficient way to fix up a 
file is with autoformat. But if the autoformat hits a bunch of irrelevant 
lines, that is annoying for a reviewer and obscures git blame.

If we enforce autoformat all the time, then it makes sure that autoformatting a 
particular PR has minimal effects and is always safe to do.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105378
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 22:39
Start Date: 23/May/18 22:39
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5457: 
[BEAM-4392] Fix :beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#discussion_r190420382
 
 

 ##
 File path: sdks/java/harness/build.gradle
 ##
 @@ -24,7 +24,16 @@ applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE <<
   // jars.
   {
 dependencies {
-  include('**/*.class')
+  //Directly include all depended projects
+  include(project(":beam-model-pipeline"))
+  include(project(":beam-model-fn-execution"))
+  include(project(":beam-sdks-java-core"))
+  include(project(":beam-sdks-java-fn-execution"))
+  include(project(":beam-sdks-java-extensions-google-cloud-platform-core"))
+  include(project(":beam-runners-core-java"))
+  include(project(":beam-runners-core-construction-java"))
+  //Include all dependencies and transitive dependencies
 
 Review comment:
   nit: add space after `//`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105378)
Time Spent: 1h 20m  (was: 1h 10m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105377
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 22:39
Start Date: 23/May/18 22:39
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5457: 
[BEAM-4392] Fix :beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#discussion_r190420370
 
 

 ##
 File path: sdks/java/harness/build.gradle
 ##
 @@ -24,7 +24,16 @@ applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE <<
   // jars.
   {
 dependencies {
-  include('**/*.class')
+  //Directly include all depended projects
 
 Review comment:
   nit: add space after `//`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105377)
Time Spent: 1h 10m  (was: 1h)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105379
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 22:39
Start Date: 23/May/18 22:39
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5457: 
[BEAM-4392] Fix :beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#discussion_r190420773
 
 

 ##
 File path: sdks/java/harness/build.gradle
 ##
 @@ -24,7 +24,16 @@ applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE <<
   // jars.
   {
 dependencies {
-  include('**/*.class')
+  //Directly include all depended projects
+  include(project(":beam-model-pipeline"))
 
 Review comment:
   We want to use the shadow configuration so we'll need to specify these as 
`path: projectName, configuration: "shadow"`
   
   We should also use a list containing all the projects and loop over it here 
and loop over it down below where we add the compile dependencies. This will 
allow for a person to update the list in one location and have it apply to both 
the shadow plugin and also the dependency set.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105379)
Time Spent: 1h 20m  (was: 1h 10m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105376
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 22:39
Start Date: 23/May/18 22:39
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5457: 
[BEAM-4392] Fix :beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#discussion_r190420339
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -321,7 +321,6 @@ ext.getJavaRelocatedPath = { String suffix ->
 
 ext.DEFAULT_SHADOW_CLOSURE = {
 
 Review comment:
   Add a link to the 
https://github.com/johnrengelman/shadow/blob/98191096a94674245c7b3e63975df9e14f67074e/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/internal/DefaultDependencyFilter.groovy#L123
 with an explanation of how include/exclude matching works.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105376)
Time Spent: 1h  (was: 50m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105373
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 22:36
Start Date: 23/May/18 22:36
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391520130
 
 
   Boyuan and I looked through the source code for the shadow plugin and the 
matching rule was specifically if its explicitly included and not explicitly 
excluded (with no include rules meaning to include all):
   
https://github.com/johnrengelman/shadow/blob/98191096a94674245c7b3e63975df9e14f67074e/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/internal/DefaultDependencyFilter.groovy#L123
   
   So as long as there is one include rule, it disables the default include all 
rule.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105373)
Time Spent: 50m  (was: 40m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #330

[Apache Jenkins Server]

See 


--
[...truncated 18.01 MB...]
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 23, 2018 10:23:55 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 23, 2018 10:23:55 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523222349-240b71e8/output/results/staging/
May 23, 2018 10:23:55 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash nMmn4ni_3wKhn3Q21dBU5Q> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523222349-240b71e8/output/results/staging/pipeline-nMmn4ni_3wKhn3Q21dBU5Q.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 23, 2018 10:23:56 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-23_15_23_55-11859672486562389339?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-23_15_23_55-11859672486562389339

org.apache.beam.sdk.transforms.ViewTest > 

[jira] [Work logged] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?focusedWorklogId=105334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105334
 ]

ASF GitHub Bot logged work on BEAM-3519:


Author: ASF GitHub Bot
Created on: 23/May/18 21:30
Start Date: 23/May/18 21:30
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4727: [BEAM-3519] 
Overrides gRPC dependency for google-cloud-platform and upgrades protobuf
URL: https://github.com/apache/beam/pull/4727#issuecomment-391502872
 
 
   We are actively looking into this but the fix will probably not get into 
next release (2.5.0).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105334)
Time Spent: 3.5h  (was: 3h 20m)

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=105325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105325
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 23/May/18 21:21
Start Date: 23/May/18 21:21
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5443: [BEAM-4357]  Fix 
shadow jar and test it
URL: https://github.com/apache/beam/pull/5443#issuecomment-391500639
 
 
   This PR makes the tests run against the shadow jar. I've been validating 
them with some variant of this: `./gradlew :beam-sdks-java-extensions-sql:test`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105325)
Time Spent: 2h 40m  (was: 2.5h)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105324=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105324
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 21:21
Start Date: 23/May/18 21:21
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391500532
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105324)
Time Spent: 5h 10m  (was: 5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #526

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO

[timrobertson100] [BEAM-4347] Minor changes following PR review

[timrobertson100] [BEAM-4347] Correction of typo following PR review

--
[...truncated 18.52 MB...]
INFO: Uploading <115879 bytes, hash ECLquIjTAqcrvK4hmHYJwA> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0523211123-d90ca70b/output/results/staging/pipeline-ECLquIjTAqcrvK4hmHYJwA.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 23, 2018 9:11:31 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-23_14_11_30-18166739715855869694?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-23_14_11_30-18166739715855869694

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 23, 2018 9:11:31 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-23_14_11_30-18166739715855869694
May 23, 2018 9:11:31 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-23_14_11_30-18166739715855869694 with 0 
expected assertions.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:31.013Z: Autoscaling is enabled for job 
2018-05-23_14_11_30-18166739715855869694. The number of workers will be between 
1 and 1000.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:31.042Z: Autoscaling was automatically enabled for 
job 2018-05-23_14_11_30-18166739715855869694.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:33.397Z: Checking required Cloud APIs are enabled.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:33.515Z: Checking permissions granted to controller 
Service Account.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:37.285Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:37.738Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.004Z: Expanding GroupByKey operations into 
optimizable parts.
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.052Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.322Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.360Z: Elided trivial flatten 
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.411Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.457Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 23, 2018 9:11:46 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T21:11:38.495Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105322
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 21:15
Start Date: 23/May/18 21:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391498774
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105322)
Time Spent: 5h  (was: 4h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105321
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 21:00
Start Date: 23/May/18 21:00
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391494708
 
 
   This passes with my tests in :beam-runners-java-fn-execution:test with 
https://github.com/apache/beam/commit/c17332afef1ba5de86b021ab6af7054c33e0065b
   
   There is a default rule to include all `compile` dependencies into the 
shaded jar, which is what the `exclude(".*")` disables. Without that, the 
`include` rules should be redundant. This increases the risk that we 
accidentally ship things without relocating them. That is something we should 
add tests for.
   
   I have no opinion on which way is right, I'm learning gradle as I go.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105321)
Time Spent: 40m  (was: 0.5h)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105320
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 23/May/18 20:59
Start Date: 23/May/18 20:59
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5428: [BEAM-4167] 
Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#issuecomment-391494400
 
 
   R: @reuvenlax looking for a committer - can you take the second pass?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105320)
Time Spent: 3h 50m  (was: 3h 40m)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4393) Missing copyright notices for shaded packages

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4393:
---
Fix Version/s: 2.5.0

> Missing copyright notices for shaded packages
> -
>
> Key: BEAM-4393
> URL: https://issues.apache.org/jira/browse/BEAM-4393
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Scott Wegner
>Priority: Major
> Fix For: 2.5.0
>
>
> From [discussion on 
> dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
>  it was recently discovered that we are missing copyright notices for 
> packages that we are shading.
> The recommended fix from [~davor] (see linked conversation), is:
> {quote}
>  I'd suggest appending this sentence to the end of the LICENSE file: "A part 
> of several convenience binary distributions of this software is licensed as 
> follows", followed by the full license text (including its copyright, clauses 
> and disclaimer) – for each such case separately. Don't edit the NOTICE file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4393) Missing copyright notices for shaded packages

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4393:
--

 Summary: Missing copyright notices for shaded packages
 Key: BEAM-4393
 URL: https://issues.apache.org/jira/browse/BEAM-4393
 Project: Beam
  Issue Type: Improvement
  Components: build-system, sdk-java-core
Reporter: Scott Wegner


>From [discussion on 
>dev@|https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E],
> it was recently discovered that we are missing copyright notices for packages 
>that we are shading.

The recommended fix from [~davor] (see linked conversation), is:

{quote}
 I'd suggest appending this sentence to the end of the LICENSE file: "A part of 
several convenience binary distributions of this software is licensed as 
follows", followed by the full license text (including its copyright, clauses 
and disclaimer) – for each such case separately. Don't edit the NOTICE file.
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=105317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105317
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 23/May/18 20:45
Start Date: 23/May/18 20:45
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5443: 
[BEAM-4357]  Fix shadow jar and test it
URL: https://github.com/apache/beam/pull/5443#discussion_r190393045
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -332,6 +328,12 @@ ext.DEFAULT_SHADOW_CLOSURE = {
 exclude "com.google.common.util.concurrent.testing.**"
   }
 }
+ext.DEFAULT_SHADOW_CLOSURE = ext.RELOCATE_SHADOW_CLOSURE << {
+  dependencies {
+exclude(".*")
 
 Review comment:
   Try to merge the fixes the Boyuan is working through for the 
sdks/java/harness


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105317)
Time Spent: 2.5h  (was: 2h 20m)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5422 from timrobertson100/BEAM-4347

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e31f27f51c7ca413c7637d71d5c717af6a54281e
Merge: 8345991 585547a
Author: Ismaël Mejía 
AuthorDate: Wed May 23 22:29:31 2018 +0200

Merge pull request #5422 from timrobertson100/BEAM-4347

[BEAM-4347] Enforce ErrorProne analysis in kafka IO

 sdks/java/io/kafka/build.gradle   |  4 +++-
 .../apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java| 12 ++--
 .../main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java   |  1 +
 .../apache/beam/sdk/io/kafka/KafkaUnboundedReader.java| 15 ++-
 .../java/org/apache/beam/sdk/io/kafka/KafkaWriter.java|  2 ++
 .../kafka/CustomTimestampPolicyWithLimitedDelayTest.java  |  2 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java| 12 +++-
 .../apache/beam/sdk/io/kafka/KafkaRecordCoderTest.java|  3 ++-
 8 files changed, 36 insertions(+), 15 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105312
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 20:29
Start Date: 23/May/18 20:29
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391485702
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105312)
Time Spent: 4h 50m  (was: 4h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2980) BagState.isEmpty needs a tighter spec

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487985#comment-16487985
 ] 

Kenneth Knowles commented on BEAM-2980:
---

I can dig that perspective, too. The issue hitting users is probably that 
"current" isn't as obvious to them.

Unfortunately I don't see a good transcript anywhere of the conversation 
between the options of "XYZState is always a changing view of an underlying ref 
cell" versus having a future-like API. It does see that the utility of 
readLater() is tied to a particular implementation strategy wherein local 
changes are buffered separately from whatever is read because of the hint.

TBH I just think futures are simpler. But I see that you'd need to refactor 
bits of the API, for prefetching hint methods to cause futures to be delivered 
elsewhere, versus imperative readLater() calls in the prefetch methods.

> BagState.isEmpty needs a tighter spec
> -
>
> Key: BEAM-2980
> URL: https://issues.apache.org/jira/browse/BEAM-2980
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Daniel Mills
>Priority: Major
>
> Consider the following:
> {code}
> BagState myBag = // empty
> ReadableState isMyBagEmpty = myBag.isEmpty();
> myBag.add(bizzle);
> bool empty = isMyBagEmpty.read();
> {code}
> Should {{empty}} be true or false? We need a consistent answer, across all 
> kinds of state, when snapshots are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4347) Enforce ErrorProne analysis in the kafka IO project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4347.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Enforce ErrorProne analysis in the kafka IO project
> ---
>
> Key: BEAM-4347
> URL: https://issues.apache.org/jira/browse/BEAM-4347
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
> Fix For: 2.5.0
>
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kafka}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kafka:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kafka}} project.
> # In {{sdks/java/io/kafka/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4347) Enforce ErrorProne analysis in the kafka IO project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4347?focusedWorklogId=105315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105315
 ]

ASF GitHub Bot logged work on BEAM-4347:


Author: ASF GitHub Bot
Created on: 23/May/18 20:31
Start Date: 23/May/18 20:31
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5422: [BEAM-4347] Enforce 
ErrorProne analysis in kafka IO
URL: https://github.com/apache/beam/pull/5422#issuecomment-391486339
 
 
   LGTM in postfix mode due to my sloppy fingers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105315)
Time Spent: 5.5h  (was: 5h 20m)

> Enforce ErrorProne analysis in the kafka IO project
> ---
>
> Key: BEAM-4347
> URL: https://issues.apache.org/jira/browse/BEAM-4347
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kafka}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kafka:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kafka}} project.
> # In {{sdks/java/io/kafka/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4347) Enforce ErrorProne analysis in the kafka IO project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4347?focusedWorklogId=105314=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105314
 ]

ASF GitHub Bot logged work on BEAM-4347:


Author: ASF GitHub Bot
Created on: 23/May/18 20:30
Start Date: 23/May/18 20:30
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5422: [BEAM-4347] Enforce 
ErrorProne analysis in kafka IO
URL: https://github.com/apache/beam/pull/5422#issuecomment-391486077
 
 
   Oups I screwed the correct title by double clicking merge by mistake. Thanks 
@timrobertson100 and @rangadi for the proper review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105314)
Time Spent: 5h 20m  (was: 5h 10m)

> Enforce ErrorProne analysis in the kafka IO project
> ---
>
> Key: BEAM-4347
> URL: https://issues.apache.org/jira/browse/BEAM-4347
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kafka}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kafka:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kafka}} project.
> # In {{sdks/java/io/kafka/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4347) Enforce ErrorProne analysis in the kafka IO project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4347?focusedWorklogId=105313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105313
 ]

ASF GitHub Bot logged work on BEAM-4347:


Author: ASF GitHub Bot
Created on: 23/May/18 20:29
Start Date: 23/May/18 20:29
Worklog Time Spent: 10m 
  Work Description: iemejia closed pull request #5422: [BEAM-4347] Enforce 
ErrorProne analysis in kafka IO
URL: https://github.com/apache/beam/pull/5422
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/io/kafka/build.gradle b/sdks/java/io/kafka/build.gradle
index 77bf209157d..2a9f3495782 100644
--- a/sdks/java/io/kafka/build.gradle
+++ b/sdks/java/io/kafka/build.gradle
@@ -17,13 +17,14 @@
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature(enableFindbugs: false)
+applyJavaNature(failOnWarning: true, enableFindbugs: false)
 
 description = "Apache Beam :: SDKs :: Java :: IO :: Kafka"
 ext.summary = "Library to read Kafka topics."
 
 dependencies {
   compile library.java.guava
+  compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.kafka_clients
   shadow library.java.slf4j_api
@@ -37,4 +38,5 @@ dependencies {
   testCompile library.java.hamcrest_core
   testCompile library.java.junit
   testCompile library.java.slf4j_jdk14
+  testCompileOnly library.java.findbugs_annotations
 }
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
index 9ae69da3834..b5d3526d4f1 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
@@ -40,6 +40,7 @@
 import java.util.Map;
 import java.util.Random;
 import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.TimeUnit;
@@ -73,6 +74,7 @@
 import org.apache.kafka.clients.producer.Producer;
 import org.apache.kafka.clients.producer.ProducerConfig;
 import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
 import org.apache.kafka.common.KafkaException;
 import org.apache.kafka.common.TopicPartition;
 import org.apache.kafka.common.serialization.ByteArrayDeserializer;
@@ -262,6 +264,8 @@ public void setup() {
   KafkaExactlyOnceSink.ensureEOSSupport();
 }
 
+// Futures ignored as exceptions will be flushed out in the commitTxn
+@SuppressWarnings("FutureReturnValueIgnored")
 @ProcessElement
 public void processElement(@StateId(NEXT_ID) ValueState nextIdState,
@StateId(MIN_BUFFERED_ID) ValueState 
minBufferedIdState,
@@ -433,18 +437,20 @@ void beginTxn() {
 ProducerSpEL.beginTransaction(producer);
   }
 
-  void sendRecord(TimestampedValue> record, Counter sendCounter) {
+
+  Future sendRecord(TimestampedValue> record, 
Counter sendCounter) {
 try {
   Long timestampMillis = spec.getPublishTimestampFunction() != null
 ? 
spec.getPublishTimestampFunction().getTimestamp(record.getValue(),
   
record.getTimestamp()).getMillis()
 : null;
 
-  producer.send(
+  Future result = producer.send(
   new ProducerRecord<>(
   spec.getTopic(), null, timestampMillis,
   record.getValue().getKey(), record.getValue().getValue()));
   sendCounter.inc();
+  return result;
 } catch (KafkaException e) {
   ProducerSpEL.abortTransaction(producer);
   throw e;
@@ -573,6 +579,8 @@ void commitTxn(long lastRecordId, Counter numTransactions) 
throws IOException {
 
   private final Cache> cache;
 
+  // Exceptions arising from the cache cleanup are ignored
+  @SuppressWarnings("FutureReturnValueIgnored")
   ShardWriterCache() {
 this.cache =
   CacheBuilder.newBuilder()
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 0859233aad1..611073e4d58 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java

[beam] branch master updated (8345991 -> e31f27f)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8345991  Merge pull request #5455: Fix build with Maven (Again!)
 add 41f75b1  [BEAM-4347] Enforce ErrorProne analysis in kafka IO
 add aa25105  [BEAM-4347] Improve ErrorProne analysis in kafka IO
 add 2750999  [BEAM-4347] Minor changes following PR review
 add 585547a  [BEAM-4347] Correction of typo following PR review
 new e31f27f  Merge pull request #5422 from timrobertson100/BEAM-4347

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/kafka/build.gradle   |  4 +++-
 .../apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java| 12 ++--
 .../main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java   |  1 +
 .../apache/beam/sdk/io/kafka/KafkaUnboundedReader.java| 15 ++-
 .../java/org/apache/beam/sdk/io/kafka/KafkaWriter.java|  2 ++
 .../kafka/CustomTimestampPolicyWithLimitedDelayTest.java  |  2 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java| 12 +++-
 .../apache/beam/sdk/io/kafka/KafkaRecordCoderTest.java|  3 ++-
 8 files changed, 36 insertions(+), 15 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=105311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105311
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 23/May/18 20:25
Start Date: 23/May/18 20:25
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5443: [BEAM-4357]  Fix 
shadow jar and test it
URL: https://github.com/apache/beam/pull/5443#issuecomment-391484737
 
 
   Hey @apilloud , could you please explain how you run the build cmd to test 
these gradle changes?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105311)
Time Spent: 2h 20m  (was: 2h 10m)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105310
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 20:22
Start Date: 23/May/18 20:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391483839
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105310)
Time Spent: 4h 40m  (was: 4.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105309
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 20:19
Start Date: 23/May/18 20:19
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391482968
 
 
   Hey @apilloud , I think maybe mine is the more correct one. What do you 
think @lukecwik ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105309)
Time Spent: 0.5h  (was: 20m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-2980) BagState.isEmpty needs a tighter spec

[Ben Chambers (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487964#comment-16487964
 ] 

Ben Chambers edited comment on BEAM-2980 at 5/23/18 8:15 PM:
-

I think `StateFuture` was considered, but intentionally not used because it 
would generally suggest we snapshot the value from the time the future is 
created. The intuition behind `ReadableState` should be it is like a ref-cell. 
As a reference, when you `read` it you get the current value stored in the 
location it references.

This is also why `read()` javadoc indicates the *current* value is returned, 
rather than the value from when the `ReadableState` was created.


was (Author: bchambers):
I think `StateFuture` was considered, but intentionally not used because it 
would generally suggest we snapshot the value from the time the future is 
created. The intuition behind `ReadableState` should be it is like a ref-cell. 
As a reference, when you `read` it you get the current value stored in the 
location it references.

> BagState.isEmpty needs a tighter spec
> -
>
> Key: BEAM-2980
> URL: https://issues.apache.org/jira/browse/BEAM-2980
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Daniel Mills
>Priority: Major
>
> Consider the following:
> {code}
> BagState myBag = // empty
> ReadableState isMyBagEmpty = myBag.isEmpty();
> myBag.add(bizzle);
> bool empty = isMyBagEmpty.read();
> {code}
> Should {{empty}} be true or false? We need a consistent answer, across all 
> kinds of state, when snapshots are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=105307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105307
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 23/May/18 20:14
Start Date: 23/May/18 20:14
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190384255
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
+  repeated double double_values = 2;
+}
+
+// Data associated with a distribution metric.
+// This is based off of the current DistributionData metric
+// This is not a stackdriver or dropwizard compatible
+// style of distribution metric.
+message DistributionData {
+  oneof distribution {
+IntDistributionData int_double_distribution = 1;
+DoubleDistributionData double_distribution_data =2;
+  }
+}
+
+message IntDistributionData {
+  int64 count = 1;
+  int64 sum = 2;
+  int64 min = 3;
+  int64 max = 4;
+}
+
+message DoubleDistributionData {
+  int64 count = 1;
+  double sum = 2;
+  double min = 3;
+  double max = 4;
+}
+
+// General MonitoredState information which contains
+// structured information which does not fit into a typical
+// metric format. For example, a table of important files
+// and metadata which an I/O source is reading.
+// Note: Since MonitoredState is designed to be
+// customizable, and allow engines to aggregate these
 
 Review comment:
   Latest across all shards/bundles? Or Union? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105307)
Time Spent: 2h 10m  (was: 2h)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, 

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105306
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 20:12
Start Date: 23/May/18 20:12
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5457: [BEAM-4392] Fix 
:beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457#issuecomment-391480841
 
 
   
https://github.com/apache/beam/pull/5443/commits/c17332afef1ba5de86b021ab6af7054c33e0065b
 is the version I wrote in in #5443. I don't know which way is right though.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105306)
Time Spent: 20m  (was: 10m)

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #329

[Apache Jenkins Server]

See 


Changes:

[iemejia] Fix build with Maven (Again!)

--
[...truncated 18.04 MB...]
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 23, 2018 8:07:14 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 23, 2018 8:07:14 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523200708-4858810a/output/results/staging/
May 23, 2018 8:07:14 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash BINf6ZeBEX7tFcFVqHM3Yg> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523200708-4858810a/output/results/staging/pipeline-BINf6ZeBEX7tFcFVqHM3Yg.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 23, 2018 8:07:15 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-23_13_07_14-10996759833510424469?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-23_13_07_14-10996759833510424469

[jira] [Commented] (BEAM-2980) BagState.isEmpty needs a tighter spec

[Ben Chambers (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487964#comment-16487964
 ] 

Ben Chambers commented on BEAM-2980:


I think `StateFuture` was considered, but intentionally not used because it 
would generally suggest we snapshot the value from the time the future is 
created. The intuition behind `ReadableState` should be it is like a ref-cell. 
As a reference, when you `read` it you get the current value stored in the 
location it references.

> BagState.isEmpty needs a tighter spec
> -
>
> Key: BEAM-2980
> URL: https://issues.apache.org/jira/browse/BEAM-2980
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Daniel Mills
>Priority: Major
>
> Consider the following:
> {code}
> BagState myBag = // empty
> ReadableState isMyBagEmpty = myBag.isEmpty();
> myBag.add(bizzle);
> bool empty = isMyBagEmpty.read();
> {code}
> Should {{empty}} be true or false? We need a consistent answer, across all 
> kinds of state, when snapshots are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105305
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 20:05
Start Date: 23/May/18 20:05
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391478775
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105305)
Time Spent: 4.5h  (was: 4h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4297) Flink portable runner executable stage operator for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4297?focusedWorklogId=105301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105301
 ]

ASF GitHub Bot logged work on BEAM-4297:


Author: ASF GitHub Bot
Created on: 23/May/18 20:00
Start Date: 23/May/18 20:00
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5407: 
[BEAM-4297] Streaming executable stage translation and operator for portable 
Flink runner.
URL: https://github.com/apache/beam/pull/5407#discussion_r190380409
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -423,8 +432,133 @@ private void translateImpulse(
   String id,
   RunnerApi.Pipeline pipeline,
   StreamingTranslationContext context) {
+// TODO: Fail on stateful DoFns for now.
+// TODO: Support stateful DoFns by inserting group-by-keys where necessary.
+// TODO: Fail on splittable DoFns.
+// TODO: Special-case single outputs to avoid multiplexing PCollections.
+RunnerApi.Components components = pipeline.getComponents();
+RunnerApi.PTransform transform = components.getTransformsOrThrow(id);
+Map outputs = transform.getOutputsMap();
+RehydratedComponents rehydratedComponents =
+RehydratedComponents.forComponents(components);
+
+BiMap outputMap =
+FlinkPipelineTranslatorUtils.createOutputMap(outputs.keySet());
+Map> outputCoders = Maps.newHashMap();
+for (String localOutputName : new TreeMap<>(outputMap.inverse()).values()) 
{
+  String collectionId = outputs.get(localOutputName);
+  Coder windowCoder = (Coder) 
instantiateCoder(collectionId, components);
+  outputCoders.put(localOutputName, windowCoder);
+}
+
+final RunnerApi.ExecutableStagePayload stagePayload;
+try {
+  stagePayload = 
RunnerApi.ExecutableStagePayload.parseFrom(transform.getSpec().getPayload());
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+
+String inputPCollectionId =
+Iterables.getOnlyElement(transform.getInputsMap().values());
 
 Review comment:
   I'm just arguing for pushing most of the manipulation done within 
ExecutableProcessBundleDescriptor into the ExecutableStage payload (minus the 
ApiServiceDescriptor binding) so it doesn't need modification. This would allow 
the ExecutableStage to concretely answer what are the input coders, output 
coders, side input coders, state coders, ... in addition to any other 
information.
   
   Longer term it seems if we had a way for the runner to say whether we need a 
keyed input context or grouped keyed output context makes sense as the runner 
could then say. These are the cases I know of:
   * KV for SplittableDoFn input, StatefulDoFn input, GBK input, 
Multimap side input materialization input, window mapping input
   * KV for GBK output
   Do you know of any others?
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105301)
Time Spent: 2h 50m  (was: 2h 40m)

> Flink portable runner executable stage operator for streaming
> -
>
> Key: BEAM-4297
> URL: https://issues.apache.org/jira/browse/BEAM-4297
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4297) Flink portable runner executable stage operator for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4297?focusedWorklogId=105302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105302
 ]

ASF GitHub Bot logged work on BEAM-4297:


Author: ASF GitHub Bot
Created on: 23/May/18 20:00
Start Date: 23/May/18 20:00
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5407: 
[BEAM-4297] Streaming executable stage translation and operator for portable 
Flink runner.
URL: https://github.com/apache/beam/pull/5407#discussion_r190380409
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -423,8 +432,133 @@ private void translateImpulse(
   String id,
   RunnerApi.Pipeline pipeline,
   StreamingTranslationContext context) {
+// TODO: Fail on stateful DoFns for now.
+// TODO: Support stateful DoFns by inserting group-by-keys where necessary.
+// TODO: Fail on splittable DoFns.
+// TODO: Special-case single outputs to avoid multiplexing PCollections.
+RunnerApi.Components components = pipeline.getComponents();
+RunnerApi.PTransform transform = components.getTransformsOrThrow(id);
+Map outputs = transform.getOutputsMap();
+RehydratedComponents rehydratedComponents =
+RehydratedComponents.forComponents(components);
+
+BiMap outputMap =
+FlinkPipelineTranslatorUtils.createOutputMap(outputs.keySet());
+Map> outputCoders = Maps.newHashMap();
+for (String localOutputName : new TreeMap<>(outputMap.inverse()).values()) 
{
+  String collectionId = outputs.get(localOutputName);
+  Coder windowCoder = (Coder) 
instantiateCoder(collectionId, components);
+  outputCoders.put(localOutputName, windowCoder);
+}
+
+final RunnerApi.ExecutableStagePayload stagePayload;
+try {
+  stagePayload = 
RunnerApi.ExecutableStagePayload.parseFrom(transform.getSpec().getPayload());
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+
+String inputPCollectionId =
+Iterables.getOnlyElement(transform.getInputsMap().values());
 
 Review comment:
   I'm just arguing for pushing most of the manipulation done within 
ExecutableProcessBundleDescriptor into the ExecutableStage payload (minus the 
ApiServiceDescriptor binding) so it doesn't need modification. This would allow 
the ExecutableStage to concretely answer what are the input coders, output 
coders, side input coders, state coders, ... in addition to any other 
information.
   
   Longer term it seems if we had a way for the runner to say whether we need a 
keyed input context or grouped keyed output context makes sense as the runner 
could then say. These are the cases I know of:
   * KV for SplittableDoFn input, StatefulDoFn input, GBK input, 
Multimap side input materialization input, window mapping input
   * KV for GBK output
   
   Do you know of any others?
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105302)
Time Spent: 3h  (was: 2h 50m)

> Flink portable runner executable stage operator for streaming
> -
>
> Key: BEAM-4297
> URL: https://issues.apache.org/jira/browse/BEAM-4297
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105300
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 19:58
Start Date: 23/May/18 19:58
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391476673
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105300)
Time Spent: 4h 20m  (was: 4h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4391) Example of distributed optimization

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487950#comment-16487950
 ] 

Ahmet Altay commented on BEAM-4391:
---

[~joachimvdh] assigned to you. Please consult with [~melap] and [~griscz] to 
find out what would be the best place to put a new example. We would like to 
keep a coherent set of examples across SDKs. It is possible that the ideal 
place for targeted examples could be a separate contrib type of a repository.

> Example of distributed optimization
> ---
>
> Key: BEAM-4391
> URL: https://issues.apache.org/jira/browse/BEAM-4391
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python
>Reporter: Joachim van der Herten
>Assignee: Joachim van der Herten
>Priority: Minor
>
> Currently, we are writing a blogpost on using the Beam Python SDK for solving 
> distributed optimization tasks. It will include an example of a optimization 
> problem with both discrete and continuous parameters, which is then solved 
> using Apache Beam. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4391) Example of distributed optimization

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-4391:
-

Assignee: Joachim van der Herten  (was: Ahmet Altay)

> Example of distributed optimization
> ---
>
> Key: BEAM-4391
> URL: https://issues.apache.org/jira/browse/BEAM-4391
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python
>Reporter: Joachim van der Herten
>Assignee: Joachim van der Herten
>Priority: Minor
>
> Currently, we are writing a blogpost on using the Beam Python SDK for solving 
> distributed optimization tasks. It will include an example of a optimization 
> problem with both discrete and continuous parameters, which is then solved 
> using Apache Beam. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105298
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 23/May/18 19:54
Start Date: 23/May/18 19:54
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5428: [BEAM-4167] 
Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#issuecomment-391475475
 
 
   Agreed. FWIW I've undone the formatting and just left checkstyle broken for 
now. To get a clean history I could put a format commit underneath this PR. I'm 
inclined to do it across the SQL module and then lock down the format.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105298)
Time Spent: 3h 40m  (was: 3.5h)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=105294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105294
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 23/May/18 19:49
Start Date: 23/May/18 19:49
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190376086
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
 
 Review comment:
   `Extrema` -> `ExtremaData` to be consistent with the others.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105294)
Time Spent: 1h 40m  (was: 1.5h)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=105295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105295
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 23/May/18 19:49
Start Date: 23/May/18 19:49
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190376862
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
+  repeated double double_values = 2;
+}
+
+// Data associated with a distribution metric.
+// This is based off of the current DistributionData metric
+// This is not a stackdriver or dropwizard compatible
+// style of distribution metric.
+message DistributionData {
+  oneof distribution {
+IntDistributionData int_double_distribution = 1;
+DoubleDistributionData double_distribution_data =2;
+  }
+}
+
+message IntDistributionData {
+  int64 count = 1;
+  int64 sum = 2;
+  int64 min = 3;
+  int64 max = 4;
+}
+
+message DoubleDistributionData {
+  int64 count = 1;
+  double sum = 2;
+  double min = 3;
+  double max = 4;
+}
+
+// General MonitoredState information which contains
+// structured information which does not fit into a typical
+// metric format. For example, a table of important files
+// and metadata which an I/O source is reading.
+// Note: Since MonitoredState is designed to be
+// customizable, and allow engines to aggregate these
+// metrics in custom ways, there is no enforced aggregation function
+// The main expectation is that an SDK should report
+// all relevant MonitoredState for its PTransform, essentially
+// ‘overwriting’ what it previously sent. This means a RunnerHarness
+// should treat new MonitoredState updates as overwriting old ones. 
+message MonitoringTableData {
+  message MonitoringColumnValue {
+oneof value {
+  int64 int64_value = 1;
+  double double_value = 2;
+  string string_value = 3;
+  google.protobuf.Timestamp timestamp = 4;
+}
+  }
+
+  message MonitoringRow {
+repeated MonitoringColumnValue values = 1;
+  }
+
+ repeated string column_names = 1;
 
 Review comment:
   Mention that the number of column_names must match the number of row_data.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=105297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105297
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 23/May/18 19:49
Start Date: 23/May/18 19:49
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190363113
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
 
 Review comment:
   Could we follow the same pattern as DistributionData and use a oneof with 
IntExtremaData and a DoubleExtremaData?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105297)
Time Spent: 2h  (was: 1h 50m)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=105293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105293
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 23/May/18 19:49
Start Date: 23/May/18 19:49
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190363113
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
 
 Review comment:
   Could we not follow the same pattern as DistributionData and use a oneof 
with IntExtremaData and a DoubleExtremaData?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105293)
Time Spent: 1.5h  (was: 1h 20m)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=105296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105296
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 23/May/18 19:49
Start Date: 23/May/18 19:49
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190376504
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
+  repeated double double_values = 2;
+}
+
+// Data associated with a distribution metric.
+// This is based off of the current DistributionData metric
+// This is not a stackdriver or dropwizard compatible
+// style of distribution metric.
+message DistributionData {
+  oneof distribution {
+IntDistributionData int_double_distribution = 1;
+DoubleDistributionData double_distribution_data =2;
+  }
+}
+
+message IntDistributionData {
+  int64 count = 1;
+  int64 sum = 2;
+  int64 min = 3;
+  int64 max = 4;
+}
+
+message DoubleDistributionData {
+  int64 count = 1;
+  double sum = 2;
+  double min = 3;
+  double max = 4;
+}
+
+// General MonitoredState information which contains
+// structured information which does not fit into a typical
+// metric format. For example, a table of important files
+// and metadata which an I/O source is reading.
+// Note: Since MonitoredState is designed to be
+// customizable, and allow engines to aggregate these
 
 Review comment:
   engines don't customize them as below you mention that the aggregation is 
always just latest from the runners perspective.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105296)
Time Spent: 1h 50m  (was: 1h 40m)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> See [relevant email 
> 

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105292
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 23/May/18 19:48
Start Date: 23/May/18 19:48
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5428: [BEAM-4167] 
Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#issuecomment-391474027
 
 
   I don't disagree with the format. It would just be nice to have it in a 
separate commit. We should probably just run format on everything and then add 
a precommit to verify.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105292)
Time Spent: 3.5h  (was: 3h 20m)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=105267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105267
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 23/May/18 18:58
Start Date: 23/May/18 18:58
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5456: [BEAM-4145] Populate 
the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456#issuecomment-391459842
 
 
   R: @angoenka @youngoli 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105267)
Time Spent: 20m  (was: 10m)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4392?focusedWorklogId=105264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105264
 ]

ASF GitHub Bot logged work on BEAM-4392:


Author: ASF GitHub Bot
Created on: 23/May/18 18:53
Start Date: 23/May/18 18:53
Worklog Time Spent: 10m 
  Work Description: boyuanzz opened a new pull request #5457: [BEAM-4392] 
Fix :beam-runners-java-fn-execution failure
URL: https://github.com/apache/beam/pull/5457
 
 
   Root cause:
   We have wrong shadowClosure configurations in 
beam/sdks/java/harness/build.gradle: 
https://github.com/apache/beam/blob/master/sdks/java/harness/build.gradle#L27, 
which cause produced jar doesn't include expected dependencies.
   
   Solution:
   Change bulid rules of sdks/java/harness to make harness as a uber-jar.
   
   r: @lukecwik @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105264)
Time Spent: 10m
Remaining Estimate: 0h

> Task :beam-runners-java-fn-execution:test gradle build failed
> -
>
> Key: BEAM-4392
> URL: https://issues.apache.org/jira/browse/BEAM-4392
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> build scan: 
> https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105262
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 18:51
Start Date: 23/May/18 18:51
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391457544
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105262)
Time Spent: 4h 10m  (was: 4h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4392) Task :beam-runners-java-fn-execution:test gradle build failed

[Boyuan Zhang (JIRA)]

Boyuan Zhang created BEAM-4392:
--

 Summary: Task :beam-runners-java-fn-execution:test gradle build 
failed
 Key: BEAM-4392
 URL: https://issues.apache.org/jira/browse/BEAM-4392
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Boyuan Zhang
Assignee: Boyuan Zhang


build scan: 
https://www.google.com/url?q=https://scans.gradle.com/s/7ueucex6axz7g=D=AFQjCNGpWIafl8oZttFesg0MVWu7uA8Qmw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=105259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105259
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 18:46
Start Date: 23/May/18 18:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391456168
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105259)
Time Spent: 4h  (was: 3h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?focusedWorklogId=105258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105258
 ]

ASF GitHub Bot logged work on BEAM-4145:


Author: ASF GitHub Bot
Created on: 23/May/18 18:45
Start Date: 23/May/18 18:45
Worklog Time Spent: 10m 
  Work Description: tgroh opened a new pull request #5456: [BEAM-4145] 
Populate the worker_id metadata in the Java SDK Harness
URL: https://github.com/apache/beam/pull/5456
 
 
   The ID is required for any control service with multiple clients, which is 
the general case.
   
   Enable `ManagedChannelFactory` implementations to register interceptors on 
all of the
   created channels.
   
   Add an interceptor which attaches the ID to the metadata. Use this 
interceptor in the 
   Sdk Harness Control Client.
   
   Populate the ID passed via the container contract in the boot go code.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105258)
Time Spent: 10m
Remaining Estimate: 0h

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4297) Flink portable runner executable stage operator for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4297?focusedWorklogId=105256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105256
 ]

ASF GitHub Bot logged work on BEAM-4297:


Author: ASF GitHub Bot
Created on: 23/May/18 18:43
Start Date: 23/May/18 18:43
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5407: 
[BEAM-4297] Streaming executable stage translation and operator for portable 
Flink runner.
URL: https://github.com/apache/beam/pull/5407#discussion_r190358677
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -423,8 +432,133 @@ private void translateImpulse(
   String id,
   RunnerApi.Pipeline pipeline,
   StreamingTranslationContext context) {
+// TODO: Fail on stateful DoFns for now.
+// TODO: Support stateful DoFns by inserting group-by-keys where necessary.
+// TODO: Fail on splittable DoFns.
+// TODO: Special-case single outputs to avoid multiplexing PCollections.
+RunnerApi.Components components = pipeline.getComponents();
+RunnerApi.PTransform transform = components.getTransformsOrThrow(id);
+Map outputs = transform.getOutputsMap();
+RehydratedComponents rehydratedComponents =
+RehydratedComponents.forComponents(components);
+
+BiMap outputMap =
+FlinkPipelineTranslatorUtils.createOutputMap(outputs.keySet());
+Map> outputCoders = Maps.newHashMap();
+for (String localOutputName : new TreeMap<>(outputMap.inverse()).values()) 
{
+  String collectionId = outputs.get(localOutputName);
+  Coder windowCoder = (Coder) 
instantiateCoder(collectionId, components);
+  outputCoders.put(localOutputName, windowCoder);
+}
+
+final RunnerApi.ExecutableStagePayload stagePayload;
+try {
+  stagePayload = 
RunnerApi.ExecutableStagePayload.parseFrom(transform.getSpec().getPayload());
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+
+String inputPCollectionId =
+Iterables.getOnlyElement(transform.getInputsMap().values());
 
 Review comment:
   Unfortunately, Flink needs to have an associated serializer 
(TypeInformation, aka Coder) with each distributed collection. This 
TypeInformation needs to be known at pipeline construction time. It need not 
match the exact coder being used to materialize elements over gRPC, but it does 
need to match the in-memory element type.
   
   We could get around this partially by representing everything as bytes. The 
downside is that each runner-native operation that requires structure (e.g., 
GBK) will require an additional operation to break elements into their 
constituent parts. This step itself also requires knowledge of the coded type, 
so we ultimately run into the same issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 105256)
Time Spent: 2h 40m  (was: 2.5h)

> Flink portable runner executable stage operator for streaming
> -
>
> Key: BEAM-4297
> URL: https://issues.apache.org/jira/browse/BEAM-4297
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4145) Java SDK Harness populates control request headers with worker id

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-4145:
-

Assignee: Thomas Groh  (was: Luke Cwik)

> Java SDK Harness populates control request headers with worker id
> -
>
> Key: BEAM-4145
> URL: https://issues.apache.org/jira/browse/BEAM-4145
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner code needs to be able to identify incoming harness connections by the 
> worker ids that it assigns to them on creation. This is currently done by the 
> go boot code when the harness runs in a docker container. However, in-process 
> harnesses never specify worker ids. This prevents in-process harnesses from 
> being multiplexed by a runner (most likely the ULR and test code).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4279) The Fn Harness doesn't properly handle Flattens which consume a single input multiple times

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-4279:
-

Assignee: Thomas Groh

> The Fn Harness doesn't properly handle Flattens which consume a single input 
> multiple times
> ---
>
> Key: BEAM-4279
> URL: https://issues.apache.org/jira/browse/BEAM-4279
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For a {{Flatten}} which consumes a {{PCollection}} n times, each input 
> element should be present {{n}} times in the output {{PCollection}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)