[jira] [Updated] (BEAM-3446) RedisIO non-prefix read operations

[Vinay varma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay varma updated BEAM-3446:
--
Description: 
Read operation in RedisIO is for prefix based look ups. While this can be used 
for exact key matches as well, the number of operations limits the through put 
of the function.
I suggest exposing current readAll operation as readbyprefix and using more 
simpler operations for readAll functionality.
ex:
{code:java}
String output = jedis.get(element);
if (output != null) {
processContext.output(KV.of(element, output));
}
{code}
instead of:
https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292

  was:
Read operation in RedisIO is for prefix based look ups. While this can be used 
for exact key matches as well, the number of operations limits the through put 
of the function.
I suggest exposing current readAll operation as readbyprefix and using more 
simpler operations for readAll functionality.
ex:
{code:java}
String output = jedis.get(element);
if (output != null) {
processContext.output(KV.of(element, output));
}
{code}



> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Vinay varma
>Assignee: Reuven Lax
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3446) RedisIO non-prefix read operations

[Vinay varma (JIRA)]

Vinay varma created BEAM-3446:
-

 Summary: RedisIO non-prefix read operations
 Key: BEAM-3446
 URL: https://issues.apache.org/jira/browse/BEAM-3446
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: Vinay varma
Assignee: Reuven Lax


Read operation in RedisIO is for prefix based look ups. While this can be used 
for exact key matches as well, the number of operations limits the through put 
of the function.
I suggest exposing current readAll operation as readbyprefix and using more 
simpler operations for readAll functionality.
ex:
{code:java}
String output = jedis.get(element);
if (output != null) {
processContext.output(KV.of(element, output));
}
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #5618

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3861

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #772

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7d240c0bb171af6868f1a6e95196c9dcfc9ac640 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7d240c0bb171af6868f1a6e95196c9dcfc9ac640
Commit message: "Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin;
 > git rev-list 7d240c0bb171af6868f1a6e95196c9dcfc9ac640 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4348363725518436193.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6192784562889100612.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3990135565358217401.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Build failed in Jenkins: beam_PerformanceTests_Spark #1216

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7d240c0bb171af6868f1a6e95196c9dcfc9ac640 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7d240c0bb171af6868f1a6e95196c9dcfc9ac640
Commit message: "Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin;
 > git rev-list 7d240c0bb171af6868f1a6e95196c9dcfc9ac640 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2462114045120393518.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7742392451891100583.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8737989551895204408.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r 

[jira] [Updated] (BEAM-3438) KinesisReaderIT fails due to a missing PipelineOptions property

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3438:
---
Summary: KinesisReaderIT fails due to a missing PipelineOptions property  
(was: KinesisIO IT fail due to an unpredictable comparison and missing 
PipelineOptions property)

> KinesisReaderIT fails due to a missing PipelineOptions property
> ---
>
> Key: BEAM-3438
> URL: https://issues.apache.org/jira/browse/BEAM-3438
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> The KinesisIO IT are failing for two reasons:
> 1. {{KinesisReaderIT}} fails due to a missing property in the 
> {{PipelineOptions}}:
> {code}
> java.lang.IllegalArgumentException: Class interface 
> org.apache.beam.sdk.testing.TestPipelineOptions missing a property named 
> 'project'.
> {code}
> 2. {{KinesisReaderTest.watermarkAdvancesWhenEnoughRecordsReadRecently}} fails 
> due to a data comparison:
> {code}
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3438) KinesisReaderIT fails due to a missing PipelineOptions property

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3438:
---
Description: 
{{KinesisReaderIT}} fails due to a missing property in the {{PipelineOptions}}:

{code}
java.lang.IllegalArgumentException: Class interface 
org.apache.beam.sdk.testing.TestPipelineOptions missing a property named 
'project'.
{code}


  was:
The KinesisIO IT are failing for two reasons:

1. {{KinesisReaderIT}} fails due to a missing property in the 
{{PipelineOptions}}:

{code}
java.lang.IllegalArgumentException: Class interface 
org.apache.beam.sdk.testing.TestPipelineOptions missing a property named 
'project'.
{code}

2. {{KinesisReaderTest.watermarkAdvancesWhenEnoughRecordsReadRecently}} fails 
due to a data comparison:

{code}
org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
was:<[1970-01-01T00:00:01.000]Z>
{code}


> KinesisReaderIT fails due to a missing PipelineOptions property
> ---
>
> Key: BEAM-3438
> URL: https://issues.apache.org/jira/browse/BEAM-3438
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> {{KinesisReaderIT}} fails due to a missing property in the 
> {{PipelineOptions}}:
> {code}
> java.lang.IllegalArgumentException: Class interface 
> org.apache.beam.sdk.testing.TestPipelineOptions missing a property named 
> 'project'.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-3439) Examples and Dataflow runner tests fail due to quota exceeded

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3439.

   Resolution: Fixed
Fix Version/s: Not applicable

Removing the jobs stuck for a while helped.

> Examples and Dataflow runner tests fail due to quota exceeded
> -
>
> Key: BEAM-3439
> URL: https://issues.apache.org/jira/browse/BEAM-3439
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> Jenkins tests are failing due to Dataflow quota exceeded:
> 1. {{Startup of the worker pool in zone us-central1-f failed to bring up any 
> of the desired 1 workers. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded.  
> Limit: 21000.0 in region us-central1.(7a8439dba801f22c): Workflow failed.}}
> 2. {{Project apache-beam-testing has insufficient quota(s) to execute this 
> workflow with 1 instances in region us-central1. Quota summary 
> (required/available): 1/1429 instances, 4/27 CPUs, 430/280 disk GB, 0/1998 
> SSD disk GB, 1/65 instance groups, 1/15 managed instance groups, 1/41 
> instance templates, 1/281 in-use IP addresses.}}
> 3. {{Dataflow quota error for jobs-per-project quota. Project 
> apache-beam-testing is running 50 jobs. Please check the quota usage via 
> Pantheon. If it exceeds the limit, please wait for a workflow to finish or 
> contact dataflow-feedb...@google.com to request an increase in quota. If it 
> does not, contact dataflow-feedb...@google.com.}}
> I think all is related to the disk quota exceeded. I'm investigating why the 
> tests consume so much space.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3416) File is not properly closed in VcfSource when exception is thrown

[Asha Rostamianfar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Asha Rostamianfar updated BEAM-3416:

Summary: File is not properly closed in VcfSource when exception is thrown  
(was: File is not properly close in VcfSource when exception is thrown)

> File is not properly closed in VcfSource when exception is thrown
> -
>
> Key: BEAM-3416
> URL: https://issues.apache.org/jira/browse/BEAM-3416
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Asha Rostamianfar
>Assignee: Asha Rostamianfar
>Priority: Minor
>
> Files are not properly closed when an exception is thrown in VcfSource 
> (apache_beam/io/vcfio.py).
> This is because the file is opened within TextSource (using a 'with' 
> statement), but the exception in VcfSource is thrown outside of that scope.
> This only causes an issue in Windows unit tests where files cannot be deleted 
> if they're used by another process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3423) Distinct.withRepresentativeValueFn throws CoderException "cannot encode null KV"

[huangjianhuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319609#comment-16319609
 ] 

huangjianhuang commented on BEAM-3423:
--

[~kenn] thx. yes, it does work fine without early firings. Maybe the null KVs 
came out because inputs were flush by  early firings and there were no inputs 
when event trigger fried? 

> Distinct.withRepresentativeValueFn throws CoderException "cannot encode null 
> KV" 
> -
>
> Key: BEAM-3423
> URL: https://issues.apache.org/jira/browse/BEAM-3423
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
> Environment: ubuntu16.04, idea, java8
>Reporter: huangjianhuang
>Assignee: Kenneth Knowles
>
> My code as follow:
> {code:java}
> pipeline
> //Read data
> .apply("Read from kafka",
> KafkaIO.read()
> .withBootstrapServers("localhost:9092")
> .withTopic(topic)
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Window. String>>into(FixedWindows.of(Duration.standardSeconds(10)))
> .triggering(AfterWatermark.pastEndOfWindow()
> 
> .withEarlyFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> 
> .discardingFiredPanes().withAllowedLateness(Duration.ZERO))
> //works fine
> //.apply(Distinct.create())
> //ops! -> CoderException: cannot encode a null KV
> .apply(Distinct.withRepresentativeValueFn(new 
> Val()).withRepresentativeType(TypeDescriptors.strings()))
> .apply(MapElements.into(TypeDescriptors.nulls())
> .via(input -> {
> System.out.println(Instant.now());
> System.out.println(input);
> return null;
> }));
> private static class Val implements SerializableFunction String>, String> {
> @Override
> public String apply(KV input) {
> return input.getValue();
> }
> }
> {code}
> Input words to Kafka:
> word1
> //after 10s
> word2
> Then got exceptions as follow:
> {code:java}
> begin
> 2018-01-06T11:18:52.971Z
> KV{null, a}
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.lang.RuntimeException: org.apache.beam.sdk.coders.CoderException: cannot 
> encode a null KV
>   at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
>   at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
>   at com.xiaomi.huyu.processor.dev.EntryPoint.main(EntryPoint.java:37)
> Caused by: java.lang.RuntimeException: 
> org.apache.beam.sdk.coders.CoderException: cannot encode a null KV
>   at 
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:113)
>   at 
> org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:235)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
>   at 
> org.apache.beam.sdk.transforms.Combine$GroupedValues$1.processElement(Combine.java:2149)
> Caused by: org.apache.beam.sdk.coders.CoderException: cannot encode a null KV
>   at org.apache.beam.sdk.coders.KvCoder.encode(KvCoder.java:70)
>   at org.apache.beam.sdk.coders.KvCoder.encode(KvCoder.java:36)
>   at org.apache.beam.sdk.coders.KvCoder.encode(KvCoder.java:73)
>   at 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormatIO_IT #1

[Apache Jenkins Server]

See 


--
[...truncated 19.67 KB...]
2018-01-10 01:34:21,509 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:34:21,515 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:34:43,516 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:34:43,528 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:35:00,376 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:35:00,385 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:35:20,288 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:35:20,301 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:35:48,544 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:35:48,591 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:36:04,232 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:36:04,279 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:36:28,220 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:36:28,236 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:36:52,177 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:36:52,185 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:37:07,400 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:37:07,409 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:37:23,624 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=/tmp/perfkitbenchmarker/runs/dc453292/kubeconfig1 create -f 

2018-01-10 01:37:23,633 dc453292 MainThread beam_integration_benchmark(1/1) 
INFO Retrying exception running IssueRetryableCommand: [Errno 2] No such 
file or directory
2018-01-10 01:37:39,504 

[jira] [Assigned] (BEAM-3190) [SQL] Join Windowing Semantics

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-3190:
-

Assignee: Anton Kedin

> [SQL] Join Windowing Semantics
> --
>
> Key: BEAM-3190
> URL: https://issues.apache.org/jira/browse/BEAM-3190
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Should join implementation reject incorrect windowing strategies?
> Concerns: discarding mode + joins + multiple trigger firings might lead to 
> incorrect results, like missing join/data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3190) [SQL] Join Windowing Semantics

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319536#comment-16319536
 ] 

Anton Kedin commented on BEAM-3190:
---

At the moment SQL joins for unbounded inputs happen per-pane. Which means that 
for discarding mode we're joining deltas, and for accumulating mode we're 
joining full panes each time trigger fires, resulting in duplicates. This means 
it becomes hard to reason about correctness of joins results.

There are few ways to extend joins implementation to mitigate this:
# accumulation. We accumulate inputs in a persistent state and join them at 
each trigger firing:
#* this enables us to join whole streams. At each trigger firing we iterate 
over buffered inputs and emit newly matching records:
#** we need to control state/buffer expiration, so that we don't accumulate 
PCollections forever:
#*** need to configure timeouts when to clear the state. Probably separately 
per input PCollection;
#*** if we know there's a 1-1 or 1-many relation, then we can purge the state 
for one input after the first match;
#*** in case of multiple joins it is unclear how to configure this per-join. 
Currently multiple joins are converted into a tree of nested binary joins;
#** correct behavior for outer joins is unclear:
#*** we need to be able to control when we emit result if there is no matching 
record in one of the inputs;
#*** and we need to be able to control what happens if eventually such matching 
record does appear;
#* potentially Beam state cells can be used for this, but there are 
complications:
#** they are partitioned per key per window;
#** meaning it is unclear how to match records across windows;
#** if we have a single global window, then this becomes feasible:
#*** example is [Nexmark 
Query3|https://github.com/apache/beam/blob/64ff21f35ee2946d38645fb0a51678628e49e62a/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java],
 which does this using GlobalWindows with multiple firings. Join implemented as 
a custom stateful ParDo;
#*** although can be implemented as a custom ParDo for a specific use case, it 
is unclear how to correctly generalize this approach;
#* potentially other mechanism can be use for buffering, not Beam state cells. 
Not investigated at the moment;
# retractions:
#* each stage of the pipeline will issue a retraction if its previous outputs 
are not valid anymore and need to be revoked:
#** for example GBKs and CoGBKs will retract previous results if new input 
arrives;
#** this will work automatically and always produce correct results under the 
hood, if implemented correctly and correct accumulation mode is configured for 
the pipeline;
#** needs more design work;
# support only specific join modes for which we know the behavior:
#* for example, if we can ensure that joins are executed only once per window:
#** we can guarantee that complete windows contents will be joined once;
#** there are known configurations with such properties which we can explicitly 
whitelist and reject everything else:
#*** DefaultTrigger with allowedLateness=0 in any accumulation mode;
#*** AfterWatermark.pastEndOfWindow() with allowedLateness=0 in any 
accumulation mode;
#** unclear how to enforce this for arbitrary windows and triggers;

Next step is to [implement the whitelisted configurations approach|BEAM-3345] 
until we have retractions.

> [SQL] Join Windowing Semantics
> --
>
> Key: BEAM-3190
> URL: https://issues.apache.org/jira/browse/BEAM-3190
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Anton Kedin
>
> Should join implementation reject incorrect windowing strategies?
> Concerns: discarding mode + joins + multiple trigger firings might lead to 
> incorrect results, like missing join/data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Spark #1215

[Apache Jenkins Server]

See 


Changes:

[robertwb] Return state from wait_until_finish.

[ehudm] Cleanup DeprecationWarning for BaseException.message.

[arostami] Fixes unclosed files in VcfSource when exception is thrown.

[lcwik] Add the ability to create an offline repository using

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7d240c0bb171af6868f1a6e95196c9dcfc9ac640 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7d240c0bb171af6868f1a6e95196c9dcfc9ac640
Commit message: "Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin;
 > git rev-list ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins58360172205478112.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5476291196713925447.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins514832380551213781.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

Build failed in Jenkins: beam_PerformanceTests_Python #771

[Apache Jenkins Server]

See 


Changes:

[robertwb] Return state from wait_until_finish.

[ehudm] Cleanup DeprecationWarning for BaseException.message.

[arostami] Fixes unclosed files in VcfSource when exception is thrown.

[lcwik] Add the ability to create an offline repository using

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7d240c0bb171af6868f1a6e95196c9dcfc9ac640 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7d240c0bb171af6868f1a6e95196c9dcfc9ac640
Commit message: "Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin;
 > git rev-list ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7194595307559620807.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4947787913692389774.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7477173540257478425.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

[jira] [Commented] (BEAM-2863) Add support for Side Inputs over the Fn API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319529#comment-16319529
 ] 

ASF GitHub Bot commented on BEAM-2863:
--

lukecwik opened a new pull request #4377: [BEAM-2863] Add the ability to length 
prefix unknown coders
URL: https://github.com/apache/beam/pull/4377
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   Add the ability to length prefix unknown coders using the portable 
representation allowing a Runner to not need to know about all coder 
representations.
   
   This is towards supporting the side inputs over the portability framework 
but can also be used for the data plane.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add support for Side Inputs over the Fn API
> ---
>
> Key: BEAM-2863
> URL: https://issues.apache.org/jira/browse/BEAM-2863
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>  Labels: portability
> Fix For: 2.3.0
>
>
> See:
> * https://s.apache.org/beam-side-inputs-1-pager
> * http://s.apache.org/beam-fn-api-state-api



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5616

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3443) Enable running the build within a sandbox without needing to have network access

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319491#comment-16319491
 ] 

ASF GitHub Bot commented on BEAM-3443:
--

lukecwik closed pull request #4374: [BEAM-3443] Support a sandbox build for 
running demos/hacking on a plane.
URL: https://github.com/apache/beam/pull/4374
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.gitignore b/.gitignore
index 2b9951a88f4..00b57a382f3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,6 +2,9 @@
 # is an input to 'maven-assembly-plugin' that generates source distribution.
 # This is typically in files named 'src.xml' throughout this repository.
 
+# Ignore any offline repositories the user may have created.
+**/offline-repository/**/*
+
 # Ignore files generated by the Gradle build process.
 **/.gradle/**/*
 **/.gogradle/**/*
diff --git a/build.gradle b/build.gradle
index ea19fac7e8d..e34007e5b4e 100644
--- a/build.gradle
+++ b/build.gradle
@@ -152,6 +152,17 @@ ext.library = [
 // fetch and enable plugins.
 buildscript {
   repositories {
+maven { url offlineRepositoryRoot }
+
+// To run gradle in offline mode, one must first invoke
+// 'updateOfflineRepository' to create an offline repo
+// inside the root project directory. See the application
+// of the offline repo plugin within build_rules.gradle
+// for further details.
+if (gradle.startParameter.isOffline()) {
+  return
+}
+
 mavenLocal()
 mavenCentral()
 jcenter()
@@ -159,17 +170,18 @@ buildscript {
 maven { url "http://repo.spring.io/plugins-release; }
   }
   dependencies {
-classpath "net.ltgt.gradle:gradle-apt-plugin:0.12" 
// Enable a Java annotation processor
-classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"   
// Enable proto code generation
-classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE" 
// Enable provided and optional configurations
-classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1" 
// Enable Apache license enforcement
-classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"
// Enable Avro code generation
-classpath "com.diffplug.spotless:spotless-plugin-gradle:3.6.0" 
// Enable a code formatting plugin
-classpath "gradle.plugin.com.github.blindpirate:gogradle:0.7.0"
// Enable Go code compilation
-classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"  
// Enable building Docker containers
-classpath "cz.malohlava:visteg:1.0.3"  
// Enable generating Gradle task dependencies as ".dot" files
-classpath "com.github.jengelman.gradle.plugins:shadow:2.0.1"   
// Enable shading Java dependencies
-classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"  
// Enable the JavaCC parser generator
+classpath "net.ltgt.gradle:gradle-apt-plugin:0.12" 
 // Enable a Java annotation processor
+classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"   
 // Enable proto code generation
+classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE" 
 // Enable provided and optional configurations
+classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1" 
 // Enable Apache license enforcement
+classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"
 // Enable Avro code generation
+classpath "com.diffplug.spotless:spotless-plugin-gradle:3.6.0" 
 // Enable a code formatting plugin
+classpath "gradle.plugin.com.github.blindpirate:gogradle:0.7.0"
 // Enable Go code compilation
+classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"  
 // Enable building Docker containers
+classpath "cz.malohlava:visteg:1.0.3"  
 // Enable generating Gradle task dependencies as 
".dot" files
+classpath "com.github.jengelman.gradle.plugins:shadow:2.0.1"   
 // Enable shading Java dependencies
+classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"  
 // Enable the JavaCC parser generator
+classpath 
"gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.3"
 // Enable creating an offline repository
   }
 }
 
@@ -186,6 +198,9 @@ rat {
   

[beam] branch master updated: Add the ability to create an offline repository using https://github.com/mdietrichstein/gradle-offline-dependencies-plugin

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 7d240c0  Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin
7d240c0 is described below

commit 7d240c0bb171af6868f1a6e95196c9dcfc9ac640
Author: Luke Cwik 
AuthorDate: Tue Jan 2 09:52:27 2018 -0800

Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin
---
 .gitignore |  3 +++
 build.gradle   | 37 ++---
 build_rules.gradle | 36 +++-
 gradle.properties  |  1 +
 4 files changed, 65 insertions(+), 12 deletions(-)

diff --git a/.gitignore b/.gitignore
index 2b9951a..00b57a3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,6 +2,9 @@
 # is an input to 'maven-assembly-plugin' that generates source distribution.
 # This is typically in files named 'src.xml' throughout this repository.
 
+# Ignore any offline repositories the user may have created.
+**/offline-repository/**/*
+
 # Ignore files generated by the Gradle build process.
 **/.gradle/**/*
 **/.gogradle/**/*
diff --git a/build.gradle b/build.gradle
index ea19fac..e34007e 100644
--- a/build.gradle
+++ b/build.gradle
@@ -152,6 +152,17 @@ ext.library = [
 // fetch and enable plugins.
 buildscript {
   repositories {
+maven { url offlineRepositoryRoot }
+
+// To run gradle in offline mode, one must first invoke
+// 'updateOfflineRepository' to create an offline repo
+// inside the root project directory. See the application
+// of the offline repo plugin within build_rules.gradle
+// for further details.
+if (gradle.startParameter.isOffline()) {
+  return
+}
+
 mavenLocal()
 mavenCentral()
 jcenter()
@@ -159,17 +170,18 @@ buildscript {
 maven { url "http://repo.spring.io/plugins-release; }
   }
   dependencies {
-classpath "net.ltgt.gradle:gradle-apt-plugin:0.12" 
// Enable a Java annotation processor
-classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"   
// Enable proto code generation
-classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE" 
// Enable provided and optional configurations
-classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1" 
// Enable Apache license enforcement
-classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"
// Enable Avro code generation
-classpath "com.diffplug.spotless:spotless-plugin-gradle:3.6.0" 
// Enable a code formatting plugin
-classpath "gradle.plugin.com.github.blindpirate:gogradle:0.7.0"
// Enable Go code compilation
-classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"  
// Enable building Docker containers
-classpath "cz.malohlava:visteg:1.0.3"  
// Enable generating Gradle task dependencies as ".dot" files
-classpath "com.github.jengelman.gradle.plugins:shadow:2.0.1"   
// Enable shading Java dependencies
-classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"  
// Enable the JavaCC parser generator
+classpath "net.ltgt.gradle:gradle-apt-plugin:0.12" 
 // Enable a Java annotation processor
+classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"   
 // Enable proto code generation
+classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE" 
 // Enable provided and optional configurations
+classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1" 
 // Enable Apache license enforcement
+classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"
 // Enable Avro code generation
+classpath "com.diffplug.spotless:spotless-plugin-gradle:3.6.0" 
 // Enable a code formatting plugin
+classpath "gradle.plugin.com.github.blindpirate:gogradle:0.7.0"
 // Enable Go code compilation
+classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"  
 // Enable building Docker containers
+classpath "cz.malohlava:visteg:1.0.3"  
 // Enable generating Gradle task dependencies as 
".dot" files
+classpath "com.github.jengelman.gradle.plugins:shadow:2.0.1"   
 // Enable shading Java dependencies
+classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"  
 // Enable 

[jira] [Updated] (BEAM-3345) [SQL] Reject Unsupported Joins

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3345:
--
Issue Type: Task  (was: Bug)

> [SQL] Reject Unsupported Joins
> --
>
> Key: BEAM-3345
> URL: https://issues.apache.org/jira/browse/BEAM-3345
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Related to https://issues.apache.org/jira/browse/BEAM-3190 .
> We don't know what meaningful join semantics is, in presence of multiple 
> trigger firings. I.e. in accumulating mode it will result in multiple 
> duplicated join results, and in discarding mode it will only join deltas, 
> both of these scenarios seem to be super subtle and can lead to 
> misunderstanding and misuse by the end users.
> Until we have a way to implement a correct join semantics (e.g. using 
> retractions) we should reject unsupported window/trigger combinations. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3345) [SQL] Reject Unsupported Joins

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-3345:
-

Assignee: Anton Kedin

> [SQL] Reject Unsupported Joins
> --
>
> Key: BEAM-3345
> URL: https://issues.apache.org/jira/browse/BEAM-3345
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Related to https://issues.apache.org/jira/browse/BEAM-3190 .
> We don't know what meaningful join semantics is, in presence of multiple 
> trigger firings. I.e. in accumulating mode it will result in multiple 
> duplicated join results, and in discarding mode it will only join deltas, 
> both of these scenarios seem to be super subtle and can lead to 
> misunderstanding and misuse by the end users.
> Until we have a way to implement a correct join semantics (e.g. using 
> retractions) we should reject unsupported window/trigger combinations. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3445) _BatchSizeEstimator should not use Metrics

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-3445:
---

 Summary: _BatchSizeEstimator should not use Metrics 
 Key: BEAM-3445
 URL: https://issues.apache.org/jira/browse/BEAM-3445
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Pablo Estrada
Assignee: Pablo Estrada


The _BatchSizeEstimator in apache_beam/transforms/util.py should not use the 
Metrics module to track its information. This should move to a system counter 
that tracks distributions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3444) Fix flake8 detected errors E999 (AST compile error)

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-3444:
-

Assignee: (was: Ahmet Altay)

> Fix flake8 detected errors E999 (AST compile error)
> ---
>
> Key: BEAM-3444
> URL: https://issues.apache.org/jira/browse/BEAM-3444
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: holdenk
>
> Fix flake8 detected errors E999 (AST compile error) so that we can run flake8 
> to catch potential python3 breaking issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3311) Extend BigTableIO to write Iterable of KV

[Anna Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319340#comment-16319340
 ] 

Anna Smith commented on BEAM-3311:
--

Thanks, [~sduskis]!  The approaches suggested seem a bit more complicated than 
writing a new builder method, especially since we have a need to retain 
windowing information.

An implementation for bulk writes using a new builder method for a PTransform 
of an Iterable is suggested here: 
https://github.com/andrisnoko/beam/blob/bulk-write-needs-fix-autovalue/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java

If this seems like a valid approach, we can add tests and remove merge 
conflicts.

> Extend BigTableIO to write Iterable of KV 
> --
>
> Key: BEAM-3311
> URL: https://issues.apache.org/jira/browse/BEAM-3311
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0
>Reporter: Anna Smith
>Assignee: Solomon Duskis
>
> The motivation is to achieve qps as advertised in BigTable in Dataflow 
> streaming mode (ex: 300k qps for 30 node cluster).  Currently we aren't 
> seeing this as the bundle size is small in streaming mode and the requests 
> are overwhelmed by AuthentiationHeader.  For example, in order to achieve qps 
> advertised each payload is recommended to be ~1KB but without batching each 
> payload is 7KB, the majority of which is the authentication header.
> Currently BigTableIO supports DoFn,...> 
> where batching is done per Bundle on flush in finishBundle. We would like to 
> be able to manually batch using a DoFn Iterable>>,...> so we can get around the small Bundle size in 
> streaming.  We have seen some improvements in qps to BigTable when running 
> with Dataflow using this approach.
> Initial thoughts on implementation would be to extend Write in order to have 
> a BulkWrite of Iterable>.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5615

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3444) Fix flake8 detected errors E999 (AST compile error)

[holdenk (JIRA)]

holdenk created BEAM-3444:
-

 Summary: Fix flake8 detected errors E999 (AST compile error)
 Key: BEAM-3444
 URL: https://issues.apache.org/jira/browse/BEAM-3444
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: holdenk
Assignee: Ahmet Altay


Fix flake8 detected errors E999 (AST compile error) so that we can run flake8 
to catch potential python3 breaking issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3416) File is not properly close in VcfSource when exception is thrown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319337#comment-16319337
 ] 

ASF GitHub Bot commented on BEAM-3416:
--

chamikaramj closed pull request #4349: [BEAM-3416] Fixes unclosed files in 
VcfSource when exception is thrown.
URL: https://github.com/apache/beam/pull/4349
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/vcfio.py 
b/sdks/python/apache_beam/io/vcfio.py
index 80f4631e462..a45861ef82a 100644
--- a/sdks/python/apache_beam/io/vcfio.py
+++ b/sdks/python/apache_beam/io/vcfio.py
@@ -285,9 +285,12 @@ def __init__(self,
   try:
 self._vcf_reader = vcf.Reader(fsock=self._create_generator())
   except SyntaxError as e:
-raise ValueError('An exception was raised when reading header from VCF 
'
- 'file %s: %s' % (self._file_name,
-  traceback.format_exc(e)))
+# Throw the exception inside the generator to ensure file is properly
+# closed (it's opened inside TextSource.read_records).
+self._text_lines.throw(
+ValueError('An exception was raised when reading header from VCF '
+   'file %s: %s' % (self._file_name,
+traceback.format_exc(e
 
 def _store_header_lines(self, header_lines):
   self._header_lines = header_lines
@@ -321,11 +324,14 @@ def next(self):
   self._file_name, self._last_record, traceback.format_exc(e))
   return MalformedVcfRecord(self._file_name, self._last_record)
 
-raise ValueError('An exception was raised when reading record from VCF 
'
- 'file %s. Invalid record was %s: %s' % (
- self._file_name,
- self._last_record,
- traceback.format_exc(e)))
+# Throw the exception inside the generator to ensure file is properly
+# closed (it's opened inside TextSource.read_records).
+self._text_lines.throw(
+ValueError('An exception was raised when reading record from VCF '
+   'file %s. Invalid record was %s: %s' % (
+   self._file_name,
+   self._last_record,
+   traceback.format_exc(e
 
 def _convert_to_variant_record(self, record, infos, formats):
   """Converts the PyVCF record to a :class:`Variant` object.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> File is not properly close in VcfSource when exception is thrown
> 
>
> Key: BEAM-3416
> URL: https://issues.apache.org/jira/browse/BEAM-3416
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Asha Rostamianfar
>Assignee: Asha Rostamianfar
>Priority: Minor
>
> Files are not properly closed when an exception is thrown in VcfSource 
> (apache_beam/io/vcfio.py).
> This is because the file is opened within TextSource (using a 'with' 
> statement), but the exception in VcfSource is thrown outside of that scope.
> This only causes an issue in Windows unit tests where files cannot be deleted 
> if they're used by another process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam] branch master updated (5fd97c0 -> ad37707)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5fd97c0  Merge pull request #4369 from robertwb/return-state
 add 813a89d  Fixes unclosed files in VcfSource when exception is thrown.
 new ad37707  Merge pull request #4349 from arostamianfar/closefilefix

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/vcfio.py | 22 ++
 1 file changed, 14 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Commented] (BEAM-2732) State tracking in Python is inefficient and has duplicated code

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319265#comment-16319265
 ] 

ASF GitHub Bot commented on BEAM-2732:
--

pabloem opened a new pull request #4375: [BEAM-2732] Starting refactor of state 
tracking in Python
URL: https://github.com/apache/beam/pull/4375
 
 
   The goal for BEAM-2732 is to refactor the context trackers in the Python SDK 
so that they will all use the same mechanism.
   Currently, Metrics, Logging and StateSampler keep their own contexts. 
BEAM-2732 aims to have all of them rely on the logic in StateSampler to keep 
their context (this is already the case in Java).
   
   This PR does the following:
   * Clean up legacy code in the StateSampler.
   * Adds a global per-thread variable for state sampler, so each execution 
thread will be able to access the current state sampler from this global 
variable (this will then be used for LoggingContext, and 
MetricsEnvironment.current_container).
   * Gives the Python-only state sampler functionality to track context, so 
that non-Cythonized runners can rely on the StateSampler to track context.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> State tracking in Python is inefficient and has duplicated code
> ---
>
> Key: BEAM-2732
> URL: https://issues.apache.org/jira/browse/BEAM-2732
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>
> e.g logging and metrics keep state separately. State tracking should be 
> unified.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam] branch master updated (aff2a1e -> 5fd97c0)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from aff2a1e  Merge pull request #4366 from udim/baseexception
 add c2a0883  Return state from wait_until_finish.
 add 5fd97c0  Merge pull request #4369 from robertwb/return-state

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/runners/portability/fn_api_runner.py  | 2 +-
 sdks/python/apache_beam/runners/portability/universal_local_runner.py | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Commented] (BEAM-3443) Enable running the build within a sandbox without needing to have network access

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319184#comment-16319184
 ] 

ASF GitHub Bot commented on BEAM-3443:
--

lukecwik opened a new pull request #4374: [BEAM-3443] Support a sandbox build 
for running demos/hacking on a plane.
URL: https://github.com/apache/beam/pull/4374
 
 
   Add the ability to create an offline repository using 
https://github.com/mdietrichstein/gradle-offline-dependencies-plugin
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable running the build within a sandbox without needing to have network 
> access
> 
>
> Key: BEAM-3443
> URL: https://issues.apache.org/jira/browse/BEAM-3443
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> Being able to build the project without needing network access allows one to 
> hack on the project within an offline environment or build and run demos 
> without network access.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3443) Enable running the build within a sandbox without needing to have network access

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3443:
---

 Summary: Enable running the build within a sandbox without needing 
to have network access
 Key: BEAM-3443
 URL: https://issues.apache.org/jira/browse/BEAM-3443
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Luke Cwik
Assignee: Luke Cwik
Priority: Minor


Being able to build the project without needing network access allows one to 
hack on the project within an offline environment or build and run demos 
without network access.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-1589) Add OnWindowExpiration method to Stateful DoFn

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1589:
-

Assignee: Batkhuyag Batsaikhan

> Add OnWindowExpiration method to Stateful DoFn
> --
>
> Key: BEAM-1589
> URL: https://issues.apache.org/jira/browse/BEAM-1589
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core, sdk-java-core
>Reporter: Jingsong Lee
>Assignee: Batkhuyag Batsaikhan
>
> See BEAM-1517
> This allows the user to do some work before the state's garbage collection.
> It seems kind of annoying, but on the other hand forgetting to set a final 
> timer to flush state is probably data loss most of the time.
> FlinkRunner does this work very simply, but other runners, such as 
> DirectRunner, need to traverse all the states to do this, and maybe it's a 
> little hard.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4681

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5614

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #610

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5613

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3388) Reduce Go runtime reflective overhead

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318966#comment-16318966
 ] 

ASF GitHub Bot commented on BEAM-3388:
--

herohde opened a new pull request #4373: [BEAM-3388] Add general-purpose 
reflection-free function calls in Go runtime 
URL: https://github.com/apache/beam/pull/4373
 
 
  * This change enables convenient type-specialization more broadly.
  * Currently used for binary combiner merge and filter predicates.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reduce Go runtime reflective overhead
> -
>
> Key: BEAM-3388
> URL: https://issues.apache.org/jira/browse/BEAM-3388
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>
> Go reflection is slow and we should avoid it in the Go SDK at runtime, when 
> possible -- especially on the fast paths. It seems unlikely that the language 
> runtime/libraries will improve any time soon: 
> https://github.com/golang/go/issues/7818.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3368) Go model pipeline doesn't support External transform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318897#comment-16318897
 ] 

ASF GitHub Bot commented on BEAM-3368:
--

aaltay closed pull request #4284: BEAM-3368 fix translation for external
URL: https://github.com/apache/beam/pull/4284
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/core/runtime/graphx/translate.go 
b/sdks/go/pkg/beam/core/runtime/graphx/translate.go
index f8cfc69227a..4d102f27250 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/translate.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/translate.go
@@ -238,6 +238,9 @@ func (m *marshaller) makePayload(edge *graph.MultiEdge) 
*pb.FunctionSpec {
case graph.GBK:
return {Urn: urnGBK}
 
+   case graph.External:
+   return {Urn: edge.Payload.URN, Payload: 
edge.Payload.Data}
+
default:
panic(fmt.Sprintf("Unexpected opcode: %v", edge.Op))
}


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Go model pipeline doesn't support External transform
> 
>
> Key: BEAM-3368
> URL: https://issues.apache.org/jira/browse/BEAM-3368
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The model pipeline translation process doesn't support the external 
> transform, causing the translation process to panic when it encounters it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3442) Clean up usage of deprecated BaseException.message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318886#comment-16318886
 ] 

ASF GitHub Bot commented on BEAM-3442:
--

chamikaramj closed pull request #4366: [BEAM-3442] Cleanup DeprecationWarning 
for BaseException.message.
URL: https://github.com/apache/beam/pull/4366
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/filesystems_test.py 
b/sdks/python/apache_beam/io/filesystems_test.py
index eaaa40f9f7e..d8cc9711550 100644
--- a/sdks/python/apache_beam/io/filesystems_test.py
+++ b/sdks/python/apache_beam/io/filesystems_test.py
@@ -120,10 +120,9 @@ def test_match_file_empty(self):
 
   def test_match_file_exception(self):
 # Match files with None so that it throws an exception
-with self.assertRaises(BeamIOError) as error:
+with self.assertRaisesRegexp(BeamIOError,
+ r'^Unable to get the Filesystem') as error:
   FileSystems.match([None])
-self.assertTrue(
-error.exception.message.startswith('Unable to get the Filesystem'))
 self.assertEqual(error.exception.exception_details.keys(), [None])
 
   def test_match_directory(self):
@@ -155,10 +154,9 @@ def test_copy(self):
   def test_copy_error(self):
 path1 = os.path.join(self.tmpdir, 'f1')
 path2 = os.path.join(self.tmpdir, 'f2')
-with self.assertRaises(BeamIOError) as error:
+with self.assertRaisesRegexp(BeamIOError,
+ r'^Copy operation failed') as error:
   FileSystems.copy([path1], [path2])
-self.assertTrue(
-error.exception.message.startswith('Copy operation failed'))
 self.assertEqual(error.exception.exception_details.keys(), [(path1, 
path2)])
 
   def test_copy_directory(self):
@@ -188,10 +186,9 @@ def test_rename(self):
   def test_rename_error(self):
 path1 = os.path.join(self.tmpdir, 'f1')
 path2 = os.path.join(self.tmpdir, 'f2')
-with self.assertRaises(BeamIOError) as error:
+with self.assertRaisesRegexp(BeamIOError,
+ r'^Rename operation failed') as error:
   FileSystems.rename([path1], [path2])
-self.assertTrue(
-error.exception.message.startswith('Rename operation failed'))
 self.assertEqual(error.exception.exception_details.keys(), [(path1, 
path2)])
 
   def test_rename_directory(self):
@@ -230,8 +227,7 @@ def test_delete(self):
 
   def test_delete_error(self):
 path1 = os.path.join(self.tmpdir, 'f1')
-with self.assertRaises(BeamIOError) as error:
+with self.assertRaisesRegexp(BeamIOError,
+ r'^Delete operation failed') as error:
   FileSystems.delete([path1])
-self.assertTrue(
-error.exception.message.startswith('Delete operation failed'))
 self.assertEqual(error.exception.exception_details.keys(), [path1])
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_test.py 
b/sdks/python/apache_beam/io/gcp/bigquery_test.py
index 84904816434..ff6721e6d91 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_test.py
@@ -20,6 +20,7 @@
 import datetime
 import json
 import logging
+import re
 import time
 import unittest
 
@@ -28,6 +29,7 @@
 
 import apache_beam as beam
 from apache_beam.internal.gcp.json_value import to_json_value
+from apache_beam.io.gcp.bigquery import JSON_COMPLIANCE_ERROR
 from apache_beam.io.gcp.bigquery import RowAsDictJsonCoder
 from apache_beam.io.gcp.bigquery import TableRowJsonCoder
 from apache_beam.io.gcp.bigquery import parse_table_schema_from_json
@@ -54,11 +56,10 @@ def test_row_as_dict(self):
 self.assertEqual(test_value, coder.decode(coder.encode(test_value)))
 
   def json_compliance_exception(self, value):
-with self.assertRaises(ValueError) as exn:
+with self.assertRaisesRegexp(ValueError, re.escape(JSON_COMPLIANCE_ERROR)):
   coder = RowAsDictJsonCoder()
   test_value = {'s': value}
-  self.assertEqual(test_value, coder.decode(coder.encode(test_value)))
-  self.assertTrue(bigquery.JSON_COMPLIANCE_ERROR in exn.exception.message)
+  coder.decode(coder.encode(test_value))
 
   def test_invalid_json_nan(self):
 self.json_compliance_exception(float('nan'))
@@ -105,13 +106,12 @@ def test_row_and_no_schema(self):
 test_row = bigquery.TableRow(
 f=[bigquery.TableCell(v=to_json_value(e))
for e in ['abc', 123, 123.456, True]])
-with self.assertRaises(AttributeError) as ctx:
+with self.assertRaisesRegexp(AttributeError,
+ r'^The TableRowJsonCoder requires'):
   coder.encode(test_row)
-self.assertTrue(
-

[beam] 01/01: Merge pull request #4366 from udim/baseexception

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit aff2a1e685283e25a1a564ba3e37143cc57cdd60
Merge: ddc4a18 9261635
Author: Chamikara Jayalath 
AuthorDate: Tue Jan 9 10:34:09 2018 -0800

Merge pull request #4366 from udim/baseexception

[BEAM-3442] Cleanup DeprecationWarning for BaseException.message.

 sdks/python/apache_beam/io/filesystems_test.py | 20 +++-
 sdks/python/apache_beam/io/gcp/bigquery_test.py| 58 ++
 .../apache_beam/io/gcp/gcsfilesystem_test.py   | 20 +++-
 sdks/python/apache_beam/io/gcp/tests/utils_test.py | 21 
 sdks/python/apache_beam/io/localfilesystem_test.py | 20 +++-
 sdks/python/apache_beam/io/tfrecordio_test.py  |  4 +-
 6 files changed, 60 insertions(+), 83 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] branch master updated (ddc4a18 -> aff2a1e)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ddc4a18  Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO
 add 9261635  Cleanup DeprecationWarning for BaseException.message.
 new aff2a1e  Merge pull request #4366 from udim/baseexception

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/filesystems_test.py | 20 +++-
 sdks/python/apache_beam/io/gcp/bigquery_test.py| 58 ++
 .../apache_beam/io/gcp/gcsfilesystem_test.py   | 20 +++-
 sdks/python/apache_beam/io/gcp/tests/utils_test.py | 21 
 sdks/python/apache_beam/io/localfilesystem_test.py | 20 +++-
 sdks/python/apache_beam/io/tfrecordio_test.py  |  4 +-
 6 files changed, 60 insertions(+), 83 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Commented] (BEAM-3441) Allow ValueProvider for JdbcIO.DataSourceConfiguration

[Sameer Abhyankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318875#comment-16318875
 ] 

Sameer Abhyankar commented on BEAM-3441:


I would like to work on this unless this is already being worked somewhere else.

> Allow ValueProvider for JdbcIO.DataSourceConfiguration
> --
>
> Key: BEAM-3441
> URL: https://issues.apache.org/jira/browse/BEAM-3441
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Sameer Abhyankar
>Assignee: Kenneth Knowles
>
> Currently JdbcIO only supports ValueProviders for queries but not for the 
> DataSourceConfiguration itself (i.e. driverClassName, url, username, password 
> etc.) These should support ValueProviders to allow the use of JdbcIO in 
> templates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3441) Allow ValueProvider for JdbcIO.DataSourceConfiguration

[Sameer Abhyankar (JIRA)]

Sameer Abhyankar created BEAM-3441:
--

 Summary: Allow ValueProvider for JdbcIO.DataSourceConfiguration
 Key: BEAM-3441
 URL: https://issues.apache.org/jira/browse/BEAM-3441
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Sameer Abhyankar
Assignee: Kenneth Knowles


Currently JdbcIO only supports ValueProviders for queries but not for the 
DataSourceConfiguration itself (i.e. driverClassName, url, username, password 
etc.) These should support ValueProviders to allow the use of JdbcIO in 
templates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3442) Clean up usage of deprecated BaseException.message

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-3442:
---

 Summary: Clean up usage of deprecated BaseException.message
 Key: BEAM-3442
 URL: https://issues.apache.org/jira/browse/BEAM-3442
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Udi Meiri
Assignee: Udi Meiri
Priority: Minor


I believe this warning only appears in unit tests.

sdks/python/apache_beam/io/gcp/bigquery_test.py:610: DeprecationWarning: 
BaseException.message has been deprecated as of Python 2.6



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #770

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f
Commit message: "Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO"
 > git rev-list ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4753521313361720047.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6055039888027691544.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7935606249535877520.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement 

Jenkins build is back to normal : beam_PerformanceTests_FileBasedIO_IT #81

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1214

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f
Commit message: "Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO"
 > git rev-list ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6225922497390215692.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2678461563075431837.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5301620399018802150.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already 

[jira] [Updated] (BEAM-3435) Python SDK examples should use beam.io.WriteToBigQuery transform rather than the BigQuerySink to interact with BQ.

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-3435:
--
Labels: starter  (was: )

> Python SDK examples should use beam.io.WriteToBigQuery transform rather than 
> the BigQuerySink to interact with BQ.
> --
>
> Key: BEAM-3435
> URL: https://issues.apache.org/jira/browse/BEAM-3435
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: starter
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam] branch go-sdk updated (f68f05e -> 2a0ba98)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f68f05e  Merge pull request #4350: Integrate master into go-sdk
 add d4b3624  Type-specialize custom decoders and encoders in Go SDK runtime
 add 7bdffd6  Type-specialize iterators and side input in the Go SDK
 add 9be0836  Add warnings if Go runtime registrations are overwritten
 add 0b43308  Add reusable element coders in Go SDK runtime
 new 2a0ba98  Merge pull request #4336 from herohde/runtime5

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/exec/coder.go|   264 +-
 sdks/go/pkg/beam/core/runtime/exec/datasink.go | 7 +-
 sdks/go/pkg/beam/core/runtime/exec/datasource.go   |52 +-
 sdks/go/pkg/beam/core/runtime/exec/decode.go   |90 +
 sdks/go/pkg/beam/core/runtime/exec/decoders.go |  1347 ++
 sdks/go/pkg/beam/core/runtime/exec/decoders.tmpl   |89 +
 sdks/go/pkg/beam/core/runtime/exec/emit.go | 7 +-
 sdks/go/pkg/beam/core/runtime/exec/encode.go   |91 +
 sdks/go/pkg/beam/core/runtime/exec/encoders.go |  1171 +
 sdks/go/pkg/beam/core/runtime/exec/encoders.tmpl   |81 +
 sdks/go/pkg/beam/core/runtime/exec/fn.go   |   138 +-
 .../pkg/beam/core/runtime/exec/fullvalue_test.go   | 4 +-
 sdks/go/pkg/beam/core/runtime/exec/input.go|   190 +
 sdks/go/pkg/beam/core/runtime/exec/inputs.go   | 21768 +++
 sdks/go/pkg/beam/core/runtime/exec/inputs.tmpl |   144 +
 sdks/go/pkg/beam/core/runtime/exec/pardo.go| 2 +-
 sdks/go/pkg/beam/runners/direct/gbk.go | 6 +-
 sdks/go/pkg/beam/testing/passert/passert.go|16 +-
 18 files changed, 25192 insertions(+), 275 deletions(-)
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decode.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decoders.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decoders.tmpl
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encode.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encoders.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encoders.tmpl
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/input.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/inputs.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/inputs.tmpl

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/01: Merge pull request #4336 from herohde/runtime5

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2a0ba9818c7f8468cdd1e2f8d28206882a5686bf
Merge: f68f05e 0b43308
Author: Robert Bradshaw 
AuthorDate: Tue Jan 9 09:47:41 2018 -0800

Merge pull request #4336 from herohde/runtime5

[BEAM-3388] Avoid various reflection and re-processing in the Go SDK runtime

 sdks/go/pkg/beam/core/runtime/exec/coder.go|   264 +-
 sdks/go/pkg/beam/core/runtime/exec/datasink.go | 7 +-
 sdks/go/pkg/beam/core/runtime/exec/datasource.go   |52 +-
 sdks/go/pkg/beam/core/runtime/exec/decode.go   |90 +
 sdks/go/pkg/beam/core/runtime/exec/decoders.go |  1347 ++
 sdks/go/pkg/beam/core/runtime/exec/decoders.tmpl   |89 +
 sdks/go/pkg/beam/core/runtime/exec/emit.go | 7 +-
 sdks/go/pkg/beam/core/runtime/exec/encode.go   |91 +
 sdks/go/pkg/beam/core/runtime/exec/encoders.go |  1171 +
 sdks/go/pkg/beam/core/runtime/exec/encoders.tmpl   |81 +
 sdks/go/pkg/beam/core/runtime/exec/fn.go   |   138 +-
 .../pkg/beam/core/runtime/exec/fullvalue_test.go   | 4 +-
 sdks/go/pkg/beam/core/runtime/exec/input.go|   190 +
 sdks/go/pkg/beam/core/runtime/exec/inputs.go   | 21768 +++
 sdks/go/pkg/beam/core/runtime/exec/inputs.tmpl |   144 +
 sdks/go/pkg/beam/core/runtime/exec/pardo.go| 2 +-
 sdks/go/pkg/beam/runners/direct/gbk.go | 6 +-
 sdks/go/pkg/beam/testing/passert/passert.go|16 +-
 18 files changed, 25192 insertions(+), 275 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Commented] (BEAM-3143) Fix type inference in Python 3 for generators

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318789#comment-16318789
 ] 

Ahmet Altay commented on BEAM-3143:
---

Issue is already resolved. Do you want me to assign to [~luke-zhu] and close it?

> Fix type inference in Python 3 for generators
> -
>
> Key: BEAM-3143
> URL: https://issues.apache.org/jira/browse/BEAM-3143
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: Holden Karau
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3440) Generic IOIT profiles configuration in io's parent pom.xml

[JIRA]

Łukasz Gajowy created BEAM-3440:
---

 Summary: Generic IOIT profiles configuration in io's parent pom.xml
 Key: BEAM-3440
 URL: https://issues.apache.org/jira/browse/BEAM-3440
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Łukasz Gajowy
Assignee: Kenneth Knowles


Currently developing IOITs requires copying maven profile configuration to 
every's IO pom for profiles like: io-it-suite, runner profiles etc. This can be 
avoided by moving all the common profiles to sdks/java/io/pom.xml parent pom 
and leave only some test specific argument settings in concrete child modules. 
This can be done using maven properties ( section in appropriate 
child pom's profile section). This would reduce lots of xml code duplication.







--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-3406.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
> Fix For: Not applicable
>
> Attachments: schema, textioit
>
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3912

[Apache Jenkins Server]

See 


--
[...truncated 1.19 MB...]
  "component_encodings": [
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": 
"%0AJ%22H%0A%1Dref_Coder_GlobalWindowCoder_1%12%27%0A%25%0A%23%0A%21urn%3Abeam%3Acoders%3Aglobal_window%3A0.1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "monthly count/GroupByKey"
  }
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #609

[Apache Jenkins Server]

See 


--
[...truncated 1.06 MB...]
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s11"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s13", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform 

[jira] [Created] (BEAM-3439) Examples and Dataflow runner tests fail due to quota exceeded

[JIRA]

Jean-Baptiste Onofré created BEAM-3439:
--

 Summary: Examples and Dataflow runner tests fail due to quota 
exceeded
 Key: BEAM-3439
 URL: https://issues.apache.org/jira/browse/BEAM-3439
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow, testing
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


Jenkins tests are failing due to Dataflow quota exceeded:

1. {{Startup of the worker pool in zone us-central1-f failed to bring up any of 
the desired 1 workers. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 
21000.0 in region us-central1.(7a8439dba801f22c): Workflow failed.}}

2. {{Project apache-beam-testing has insufficient quota(s) to execute this 
workflow with 1 instances in region us-central1. Quota summary 
(required/available): 1/1429 instances, 4/27 CPUs, 430/280 disk GB, 0/1998 SSD 
disk GB, 1/65 instance groups, 1/15 managed instance groups, 1/41 instance 
templates, 1/281 in-use IP addresses.}}

3. {{Dataflow quota error for jobs-per-project quota. Project 
apache-beam-testing is running 50 jobs. Please check the quota usage via 
Pantheon. If it exceeds the limit, please wait for a workflow to finish or 
contact dataflow-feedb...@google.com to request an increase in quota. If it 
does not, contact dataflow-feedb...@google.com.}}

I think all is related to the disk quota exceeded. I'm investigating why the 
tests consume so much space.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3438) KinesisIO IT fail due to an unpredictable comparison and missing PipelineOptions property

[JIRA]

Jean-Baptiste Onofré created BEAM-3438:
--

 Summary: KinesisIO IT fail due to an unpredictable comparison and 
missing PipelineOptions property
 Key: BEAM-3438
 URL: https://issues.apache.org/jira/browse/BEAM-3438
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions, testing
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


The KinesisIO IT are failing for two reasons:

1. {{KinesisReaderIT}} fails due to a missing property in the 
{{PipelineOptions}}:

{code}
java.lang.IllegalArgumentException: Class interface 
org.apache.beam.sdk.testing.TestPipelineOptions missing a property named 
'project'.
{code}

2. {{KinesisReaderTest.watermarkAdvancesWhenEnoughRecordsReadRecently}} fails 
due to a data comparison:

{code}
org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
was:<[1970-01-01T00:00:01.000]Z>
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4680

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5612

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #769

[Apache Jenkins Server]

See 


Changes:

[jb] Exclude go from rat configuration, use .gitignore for excludes and align

[iemejia] [BEAM-3422] Update reproducible-build-maven-plugin to version 0.4

[jbonofre] [BEAM-3282] MqttIO reader now use receive for timeout in order to 
return

--
[...truncated 90.29 KB...]
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s8"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s16", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "SideInput-s15": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s15"
  }, 
  "SideInput-s16": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s16"
  }
}, 
 

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #80

[Apache Jenkins Server]

See 


Changes:

[jb] Exclude go from rat configuration, use .gitignore for excludes and align

[iemejia] [BEAM-3422] Update reproducible-build-maven-plugin to version 0.4

[jbonofre] [BEAM-3282] MqttIO reader now use receive for timeout in order to 
return

--
[...truncated 40.78 KB...]
[INFO] --- maven-compiler-plugin:3.6.2:testCompile (default-testCompile) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to 
/tmp/perfkitbenchmarker/runs/2f955723/beam/sdks/java/io/file-based-io-tests/target/test-classes
[INFO] 
/tmp/perfkitbenchmarker/runs/2f955723/beam/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java:
 Some input files use or override a deprecated API.
[INFO] 
/tmp/perfkitbenchmarker/runs/2f955723/beam/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (default) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- build-helper-maven-plugin:3.0.0:regex-properties 
(render-artifact-id) @ beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/2f955723/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:test-jar (default-test-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/2f955723/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-shade-plugin:3.0.0:shade (bundle-and-repackage) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Excluding com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from 
the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from the shaded jar.
[INFO] Excluding joda-time:joda-time:jar:2.4 from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.3.0-SNAPSHOT
 from the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.3.0-SNAPSHOT from the 

Build failed in Jenkins: beam_PerformanceTests_Spark #1213

[Apache Jenkins Server]

See 


Changes:

[jb] Exclude go from rat configuration, use .gitignore for excludes and align

[iemejia] [BEAM-3422] Update reproducible-build-maven-plugin to version 0.4

[jbonofre] [BEAM-3282] MqttIO reader now use receive for timeout in order to 
return

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f
Commit message: "Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO"
 > git rev-list a7cd939853da7eedb0c3b890aaf47fcbcab89063 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3726847212405008160.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3133361788939276843.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1096289336358158251.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4679

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5611

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #608

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-3282] MqttIO reader now use receive for timeout in order to 
return

--
[...truncated 1.06 MB...]
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s11"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s13", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
   

Build failed in Jenkins: beam_PostCommit_Python_Verify #3911

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-3282] MqttIO reader now use receive for timeout in order to 
return

--
[...truncated 1.19 MB...]
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/Extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/Extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s15", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s8"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s16", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": 

[jira] [Commented] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318222#comment-16318222
 ] 

ASF GitHub Bot commented on BEAM-3409:
--

rmannibucau opened a new pull request #4372: BEAM-3409 starting to fix the wait 
of direct runner
URL: https://github.com/apache/beam/pull/4372
 
 
   Using the direct runner, p.run().waitUntilFinish() doesn't wait for the 
teardown execution
   
   This PR contains these fixes:
   
   1. for each dofn registered it will create a latch the execution will wait 
on for waitUntilFinish and guarantee
   the teardown methods are included into this "wait"
   2. For SDF it ensure the dofnmanager are released and therefore than 
teardown is called (propagating previous logic as well)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3282) MqttIO reader should use receive with timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318206#comment-16318206
 ] 

ASF GitHub Bot commented on BEAM-3282:
--

jbonofre closed pull request #4206: [BEAM-3282] MqttIO reader now use receive 
for timeout in order to ret…
URL: https://github.com/apache/beam/pull/4206
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java 
b/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java
index f9083bb5e7e..ef9c9d293d9 100644
--- a/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java
+++ b/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java
@@ -20,6 +20,7 @@
 import static com.google.common.base.Preconditions.checkArgument;
 
 import com.google.auto.value.AutoValue;
+import com.google.common.annotations.VisibleForTesting;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.ArrayList;
@@ -27,7 +28,9 @@
 import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.UUID;
+import java.util.concurrent.TimeUnit;
 import javax.annotation.Nullable;
+
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.ByteArrayCoder;
 import org.apache.beam.sdk.coders.Coder;
@@ -291,7 +294,8 @@ public void populateDisplayData(DisplayData.Builder 
builder) {
* Checkpoint for an unbounded MQTT source. Consists of the MQTT messages 
waiting to be
* acknowledged and oldest pending message timestamp.
*/
-  private static class MqttCheckpointMark implements 
UnboundedSource.CheckpointMark, Serializable {
+  @VisibleForTesting
+  static class MqttCheckpointMark implements UnboundedSource.CheckpointMark, 
Serializable {
 
 private String clientId;
 private Instant oldestMessageTimestamp = Instant.now();
@@ -329,8 +333,8 @@ private void readObject(java.io.ObjectInputStream stream)
 
   }
 
-  private static class UnboundedMqttSource
-  extends UnboundedSource {
+  @VisibleForTesting
+  static class UnboundedMqttSource extends UnboundedSource {
 
 private final Read spec;
 
@@ -370,7 +374,8 @@ public void populateDisplayData(DisplayData.Builder 
builder) {
 }
   }
 
-  private static class UnboundedMqttReader extends 
UnboundedSource.UnboundedReader {
+  @VisibleForTesting
+  static class UnboundedMqttReader extends 
UnboundedSource.UnboundedReader {
 
 private final UnboundedMqttSource source;
 
@@ -411,8 +416,11 @@ public boolean start() throws IOException {
 @Override
 public boolean advance() throws IOException {
   try {
-LOG.debug("MQTT reader (client ID {}) waiting message ...", 
client.getClientId());
-Message message = connection.receive();
+LOG.trace("MQTT reader (client ID {}) waiting message ...", 
client.getClientId());
+Message message = connection.receive(1, TimeUnit.SECONDS);
+if (message == null) {
+  return false;
+}
 current = message.getPayload();
 currentTimestamp = Instant.now();
 checkpointMark.add(message, currentTimestamp);
diff --git 
a/sdks/java/io/mqtt/src/test/java/org/apache/beam/sdk/io/mqtt/MqttIOTest.java 
b/sdks/java/io/mqtt/src/test/java/org/apache/beam/sdk/io/mqtt/MqttIOTest.java
index 1b3d2da3542..d7baf3bf989 100644
--- 
a/sdks/java/io/mqtt/src/test/java/org/apache/beam/sdk/io/mqtt/MqttIOTest.java
+++ 
b/sdks/java/io/mqtt/src/test/java/org/apache/beam/sdk/io/mqtt/MqttIOTest.java
@@ -24,6 +24,7 @@
 import java.util.ArrayList;
 import java.util.HashSet;
 import java.util.Set;
+
 import org.apache.activemq.broker.BrokerService;
 import org.apache.activemq.broker.Connection;
 import org.apache.beam.sdk.io.mqtt.MqttIO.Read;
@@ -37,6 +38,7 @@
 import org.fusesource.mqtt.client.Message;
 import org.fusesource.mqtt.client.QoS;
 import org.fusesource.mqtt.client.Topic;
+import org.joda.time.Duration;
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Rule;
@@ -134,7 +136,7 @@ public void run() {
 publisherThread.join();
   }
 
-  @Test(timeout = 60 * 1000)
+  @Test(timeout = 5 * 1000)
   public void testRead() throws Exception {
 PCollection output = pipeline.apply(
 MqttIO.read()
@@ -143,7 +145,7 @@ public void testRead() throws Exception {
 "tcp://localhost:" + port,
 "READ_TOPIC",
 "READ_PIPELINE"))
-  .withMaxNumRecords(10));
+.withMaxReadTime(Duration.standardSeconds(3)));
 PAssert.that(output).containsInAnyOrder(
 "This is test 

[jira] [Resolved] (BEAM-3282) MqttIO reader should use receive with timeout

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3282.

   Resolution: Fixed
Fix Version/s: 2.3.0

> MqttIO reader should use receive with timeout
> -
>
> Key: BEAM-3282
> URL: https://issues.apache.org/jira/browse/BEAM-3282
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.3.0
>
>
> Now, {{MqttIO}} reader uses {{connection.receive()}} which is a blocking 
> method, waiting for messages. This prevents the reader to return {{false}} 
> when there's no message and so to advance the processing.
> Instead, the reader should use {{connection.receive(timeout, timeunit)}} and 
> return {{false}} in {{advance()}} when message is null.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam] branch master updated (65ab1cc -> ddc4a18)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 65ab1cc  Merge pull request #4253 from jbonofre/GO_RAT
 add d22fa01  [BEAM-3282] MqttIO reader now use receive for timeout in 
order to return false when there's no message to read
 new ddc4a18  Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/io/mqtt/MqttIO.java   | 20 ++--
 .../org/apache/beam/sdk/io/mqtt/MqttIOTest.java| 22 --
 2 files changed, 34 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/01: Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ddc4a188363e21d5d9d89ffbbc84cbfb20c6284f
Merge: 65ab1cc d22fa01
Author: Jean-Baptiste Onofré 
AuthorDate: Tue Jan 9 11:20:16 2018 +0100

Merge pull request #4206 from jbonofre/BEAM-3282-MQTTIO

[BEAM-3282] MqttIO reader now use receive for timeout in order to ret…

 .../java/org/apache/beam/sdk/io/mqtt/MqttIO.java   | 20 ++--
 .../org/apache/beam/sdk/io/mqtt/MqttIOTest.java| 22 --
 2 files changed, 34 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #79

[Apache Jenkins Server]

See 


--
[...truncated 21.62 KB...]
[INFO] --- maven-compiler-plugin:3.6.2:testCompile (default-testCompile) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to 
/tmp/perfkitbenchmarker/runs/98cd87d7/beam/sdks/java/io/file-based-io-tests/target/test-classes
[INFO] 
/tmp/perfkitbenchmarker/runs/98cd87d7/beam/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java:
 Some input files use or override a deprecated API.
[INFO] 
/tmp/perfkitbenchmarker/runs/98cd87d7/beam/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (default) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- build-helper-maven-plugin:3.0.0:regex-properties 
(render-artifact-id) @ beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/98cd87d7/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:test-jar (default-test-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/98cd87d7/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-shade-plugin:3.0.0:shade (bundle-and-repackage) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Excluding com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from 
the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from the shaded jar.
[INFO] Excluding joda-time:joda-time:jar:2.4 from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.3.0-SNAPSHOT
 from the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.3.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding 

[jira] [Created] (BEAM-3437) Support schema in PCollections

[JIRA]

Jean-Baptiste Onofré created BEAM-3437:
--

 Summary: Support schema in PCollections
 Key: BEAM-3437
 URL: https://issues.apache.org/jira/browse/BEAM-3437
 Project: Beam
  Issue Type: Wish
  Components: beam-model
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


As discussed with some people in the team, it would be great to add schema 
support in {{PCollections}}. It will allow us:

1. To expect some data type in {{PTransforms}}
2. Improve some runners with additional features (I'm thinking about Spark 
runner with data frames for instance).

A technical draft document has been created: 

https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit?disco=BhykQIs=5a203b46=comment_email_document

I also started a PoC on a branch, I will update this Jira with a "discussion" 
PR.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318194#comment-16318194
 ] 

ASF GitHub Bot commented on BEAM-3060:
--

DariuszAniszewski opened a new pull request #4305: [BEAM-3060] Allow to specify 
timeout for FileBasedIOIT ran via PerfKit
URL: https://github.com/apache/beam/pull/4305
 
 
   with default set to 10 mins (which is PerfKit's timeout).
   
   Background: large-scale tests run via PerfKit were failing. this PR allows 
to specify timeout so tests are passing.
   
   --
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318193#comment-16318193
 ] 

ASF GitHub Bot commented on BEAM-3060:
--

DariuszAniszewski closed pull request #4305: [BEAM-3060] Allow to specify 
timeout for FileBasedIOIT ran via PerfKit
URL: https://github.com/apache/beam/pull/4305
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/io/file-based-io-tests/pom.xml 
b/sdks/java/io/file-based-io-tests/pom.xml
index 44119ec79ff..4de2e70615f 100644
--- a/sdks/java/io/file-based-io-tests/pom.xml
+++ b/sdks/java/io/file-based-io-tests/pom.xml
@@ -113,6 +113,7 @@
 ${pkbLocation}
 
-benchmarks=beam_integration_benchmark
 -beam_it_profile=io-it
+
-beam_it_timeout=${pkbTimeout}
 
-beam_location=${beamRootProjectDir}
 -beam_prebuilt=true
 -beam_sdk=java
diff --git a/sdks/java/io/pom.xml b/sdks/java/io/pom.xml
index 07e1b5cb9ff..0710df05d89 100644
--- a/sdks/java/io/pom.xml
+++ b/sdks/java/io/pom.xml
@@ -38,6 +38,7 @@
 
 
 
+600
   
 
   


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #78

[Apache Jenkins Server]

See 


--
[...truncated 21.62 KB...]
[INFO] --- maven-compiler-plugin:3.6.2:testCompile (default-testCompile) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to 
/tmp/perfkitbenchmarker/runs/c626cef6/beam/sdks/java/io/file-based-io-tests/target/test-classes
[INFO] 
/tmp/perfkitbenchmarker/runs/c626cef6/beam/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java:
 Some input files use or override a deprecated API.
[INFO] 
/tmp/perfkitbenchmarker/runs/c626cef6/beam/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (default) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- build-helper-maven-plugin:3.0.0:regex-properties 
(render-artifact-id) @ beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/c626cef6/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:test-jar (default-test-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/c626cef6/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-shade-plugin:3.0.0:shade (bundle-and-repackage) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Excluding com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from 
the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from the shaded jar.
[INFO] Excluding joda-time:joda-time:jar:2.4 from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.3.0-SNAPSHOT
 from the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.3.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3910

[Apache Jenkins Server]

See 


--
[...truncated 1.14 MB...]
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "pair_with_one"
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "group.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": 
"%0AJ%22H%0A%1Dref_Coder_GlobalWindowCoder_1%12%27%0A%25%0A%23%0A%21urn%3Abeam%3Acoders%3Aglobal_window%3A0.1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "group"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "count_ones"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "count.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "count"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s6", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #607

[Apache Jenkins Server]

See 


--
[...truncated 1.06 MB...]
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s15", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s16", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 

[jira] [Created] (BEAM-3436) RetryHttpRequestInitializerTest takes 4min to complete

[JIRA]

Ismaël Mejía created BEAM-3436:
--

 Summary: RetryHttpRequestInitializerTest takes 4min to complete
 Key: BEAM-3436
 URL: https://issues.apache.org/jira/browse/BEAM-3436
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions, testing
Affects Versions: 2.2.0
Reporter: Ismaël Mejía
Assignee: Reuven Lax


The class RetryHttpRequestInitializerTest that is part of the 
beam-sdks-java-extensions-google-cloud-platform-core takes too long to execute 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4678

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3422) The jar files uploaded to maven do not work with Java 9

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318029#comment-16318029
 ] 

Ismaël Mejía commented on BEAM-3422:


Thanks for reporting [~cheister] the fix is in time for the 2.3.0 release.

> The jar files uploaded to maven do not work with Java 9
> ---
>
> Key: BEAM-3422
> URL: https://issues.apache.org/jira/browse/BEAM-3422
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Affects Versions: 2.2.0
> Environment: OSX Darwin 17.3.0
>Reporter: Chris Heisterkamp
>Assignee: Ismaël Mejía
> Fix For: 2.3.0
>
>
> The apache beam jars uploaded to maven have invalid date modified times in 
> the jar files and cannot be loaded by JDK 9 because of 
> https://bugs.openjdk.java.net/browse/JDK-8184940.  I've submitted a new bug 
> to http://bugreport.java.com asking that the fix be back ported to the next 
> Java 9 release.
> In the meantime, can you repackage the jars with valid modified times? And 
> uploaded fixed jars to Maven?
> To reproduce you just need to put one of the jars on the classpath for javac. 
>  So far I've found the following jars on maven central are affected.
> beam-runners-google-cloud-dataflow-java-2.2.0.jar
> beam-sdks-common-runner-api-2.2.0.jar
> beam-sdks-java-core-2.2.0.jar
> beam-sdks-java-extensions-google-cloud-platform-core-2.2.0.jar
> beam-sdks-java-extensions-protobuf-2.2.0.jar
> beam-sdks-java-io-google-cloud-platform-2.2.0.jar
> e.g.
> ```
> > curl -O 
> > http://central.maven.org/maven2/org/apache/beam/beam-sdks-java-core/2.2.0/beam-sdks-java-core-2.2.0.jar
> > javac -cp beam-sdks-java-core-2.2.0.jar HelloWorld.java
> An exception has occurred in the compiler (9.0.1). Please file a bug against 
> the Java compiler via the Java bug reporting page (http://bugreport.java.com) 
> after checking the Bug Database (http://bugs.java.com) for duplicates. 
> Include your program and the following diagnostic in your report. Thank you.
> java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 
> 12): 0
>   at 
> java.base/java.time.temporal.ValueRange.checkValidValue(ValueRange.java:311)
>   at 
> java.base/java.time.temporal.ChronoField.checkValidValue(ChronoField.java:714)
>   at java.base/java.time.LocalDate.of(LocalDate.java:269)
>   at java.base/java.time.LocalDateTime.of(LocalDateTime.java:336)
>   at jdk.zipfs/jdk.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:109)
>   at 
> jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:1950)
>   at 
> jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1937)
>   at 
> jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getEntry(ZipFileSystem.java:1324)
>   at 
> jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.newInputStream(ZipFileSystem.java:550)
>   at 
> jdk.zipfs/jdk.nio.zipfs.JarFileSystem.isMultiReleaseJar(JarFileSystem.java:91)
>   at jdk.zipfs/jdk.nio.zipfs.JarFileSystem.(JarFileSystem.java:67)
>   at 
> jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:134)
>   at 
> jdk.compiler/com.sun.tools.javac.file.JavacFileManager$ArchiveContainer.(JavacFileManager.java:517)
>   at 
> jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getContainer(JavacFileManager.java:319)
>   at 
> jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:715)
>   at 
> jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:722)
>   at 
> jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:648)
>   at 
> jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:529)
>   at 
> jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:293)
>   at 
> jdk.compiler/com.sun.tools.javac.code.Symtab.lambda$addRootPackageFor$7(Symtab.java:758)
>   at 
> jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:633)
>   at 
> jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:354)
>   at 
> jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
>   at 
> jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:285)
>   at 
> jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:300)
>   at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:570)
>   at jdk.compiler/com.sun.tools.javac.comp.Enter.main(Enter.java:554)
>   at 
> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:1052)
>   at 
> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:923)
>   at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:302)
>   at