Build failed in Jenkins: beam_PerformanceTests_Python #868

[Apache Jenkins Server]

See 


Changes:

[Pablo] Logging deviation from sampling expectation. This will allow to track

[Pablo] Adding a static getter for RuntimeValueProvider.

[Pablo] Addressing comments.

[github] Removing unnecessary code.

[cclauss] global INT64_MAX, INT64_MIN to placate linters

[tgroh] Add a LocalArtifactStagingLocation

[tgroh] Add LocalArtifactStagingLocation#forExisting

[tgroh] Add an ArtifactRetrievalService interface

[tgroh] Implement a Local ArtifactRetrievalService

[chamikara] Adds a ReadAll transform to tfrecordio.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6
Commit message: "Merge pull request #4578: [BEAM-3603] Adds a ReadAll transform 
to tfrecordio"
 > git rev-list 5e6520a841d3b1b2959e21969f551a8b66d298dc # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins628077633323940676.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4710412204411799586.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4064543524272982369.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9182006881955816846.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4062785761493375369.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3425052077116360519.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #813

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4870

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3603) Add a ReadAll transform to tfrecordio

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-3603.
--
   Resolution: Fixed
Fix Version/s: 2.4.0

> Add a ReadAll transform to tfrecordio
> -
>
> Key: BEAM-3603
> URL: https://issues.apache.org/jira/browse/BEAM-3603
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.4.0
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: 2.4.0
>
>
> We currently have ReadAll transforms for textio and avroio. We should add one 
> for tfrecordio as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4578: [BEAM-3603] Adds a ReadAll transform to tfrecordio

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6
Merge: 1693e7d 28ce7a5
Author: Chamikara Jayalath 
AuthorDate: Fri Feb 2 19:44:58 2018 -0800

Merge pull request #4578: [BEAM-3603] Adds a ReadAll transform to tfrecordio

 sdks/python/apache_beam/io/filebasedsource.py |  27 +-
 sdks/python/apache_beam/io/tfrecordio.py  |  47 ++-
 sdks/python/apache_beam/io/tfrecordio_test.py | 482 +++---
 3 files changed, 324 insertions(+), 232 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] branch master updated (1693e7d -> 8130c20)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1693e7d  Merge pull request #4562 global INT64_MAX, INT64_MIN to 
placate linters
 add 28ce7a5  Adds a ReadAll transform to tfrecordio.
 new 8130c20  Merge pull request #4578: [BEAM-3603] Adds a ReadAll 
transform to tfrecordio

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/filebasedsource.py |  27 +-
 sdks/python/apache_beam/io/tfrecordio.py  |  47 ++-
 sdks/python/apache_beam/io/tfrecordio_test.py | 482 +++---
 3 files changed, 324 insertions(+), 232 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #812

[Apache Jenkins Server]

See 


Changes:

[Pablo] Logging deviation from sampling expectation. This will allow to track

[Pablo] Adding a static getter for RuntimeValueProvider.

[Pablo] Addressing comments.

[github] Removing unnecessary code.

[cclauss] global INT64_MAX, INT64_MIN to placate linters

[tgroh] Add a LocalArtifactStagingLocation

[tgroh] Add LocalArtifactStagingLocation#forExisting

[tgroh] Add an ArtifactRetrievalService interface

[tgroh] Implement a Local ArtifactRetrievalService

--
[...truncated 183.28 KB...]
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s28"
}, 
"serialized_fn": 
"%0AJ%22H%0A%1Dref_Coder_GlobalWindowCoder_1%12%27%0A%25%0A%23%0A%21urn%3Abeam%3Acoders%3Aglobal_window%3A0.1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert:even/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:even/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s29"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4869

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3613) SpannerIO: Typo in "witHost"

[David Yan (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351178#comment-16351178
 ] 

David Yan commented on BEAM-3613:
-

[https://github.com/apache/beam/pull/4592] open.

> SpannerIO: Typo in "witHost"
> 
>
> Key: BEAM-3613
> URL: https://issues.apache.org/jira/browse/BEAM-3613
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: David Yan
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> There is a typo in SpannerIO.java.
> "witHost" should be "withHost".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3613) SpannerIO: Typo in "witHost"

[David Yan (JIRA)]

David Yan created BEAM-3613:
---

 Summary: SpannerIO: Typo in "witHost"
 Key: BEAM-3613
 URL: https://issues.apache.org/jira/browse/BEAM-3613
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: David Yan
Assignee: Chamikara Jayalath


There is a typo in SpannerIO.java.

"witHost" should be "withHost".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3545) Fn API metrics in Go SDK harness

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-3545:

Component/s: (was: sdk-java-harness)
 sdk-go

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3584) Java dataflow job fails with 2.3.0 RC1, due to missing worker image

[Alan Myrvold (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Myrvold resolved BEAM-3584.

   Resolution: Fixed
Fix Version/s: 2.3.0

Fixed. Image has been published and dataflow jobs succeed now.

> Java dataflow job fails with 2.3.0 RC1, due to missing worker image
> ---
>
> Key: BEAM-3584
> URL: https://issues.apache.org/jira/browse/BEAM-3584
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Affects Versions: 2.3.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Dataflow java jobs currently fail due to a missing docker image when using 
> 2.3.0 RC1. This is not a bug in the SDK, the worker image needs to be 
> published by google. I will be coordinating the worker image publication.
> # Update to your own project and bucket.
> GCS_BUCKET=my-cloud-storage-bucket
> GCP_PROJECT=my-cloud-project
> mkdir -p ~/try2.3.0/.m2/repository
> cd ~/try2.3.0/.m2/repository
> wget 
> [https://repository.apache.org/content/repositories/orgapachebeam-1026/archetype-catalog.xml]
> cd ~/try2.3.0
> echo 
> "$PWD/.m2/repositorytestreltestrel[https://repository.apache.org/content/repositories/orgapachebeam-1026]"
>  > settings.xml
> mvn archetype:generate -DarchetypeCatalog=local  
> -DarchetypeGroupId=org.apache.beam 
>  -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples   
> -DarchetypeVersion=2.3.0  -DgroupId=org.example   
> -DartifactId=word-count-beam   -Dversion="0.1"   
> -Dpackage=org.apache.beam.examples -s settings.xml
> # answer yes
> cd word-count-beam
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount 
>    -Dexec.args="--runner=DataflowRunner --project=alan-jenkins-test  
> --gcpTempLocation=gs://${GCS_BUCKET}/tmp  
> --inputFile=gs://apache-beam-samples/shakespeare/* 
> --output=gs://${GCP_PROJECT}e/counts"   -s ../settings.xml 
> -Ptestrel,dataflow-runner
>  
> Dataflow logs contain:
> E  Handler for GET 
> /v1.27/images/dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.3.0/json 
> returned error: No such image: 
> dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.3.0 
> E  PullImage "dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.3.0" from image 
> service failed: rpc error: code = 2 desc = Error: Status 405 trying to pull 
> repository v1beta3/beam-java-batch: "v1 Registry API is disabled. If you are 
> not explicitly using the v1 Registry API, it is possible your v2 image could 
> not be found. Verify that your image is available, or retry with `dockerd 
> --disable-legacy-registry`. See 
> https://cloud.google.com/container-registry/docs/support/deprecation-notices; 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5849

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351157#comment-16351157
 ] 

Ben Sidhom commented on BEAM-3587:
--

Just repeated it with literally just a read step. It still seems to run without 
error, but obviously there's no output to inspect:

{{p.apply(TextIO.read().from(options.getInputPath())).run().waitUntilFinish()}}

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4562 global INT64_MAX, INT64_MIN to placate linters

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1693e7d1affef6ebed83a64d46ef6f39b465d299
Merge: af864b8 fd07d72
Author: Robert Bradshaw 
AuthorDate: Fri Feb 2 17:14:38 2018 -0800

Merge pull request #4562 global INT64_MAX, INT64_MIN to placate linters

global INT64_MAX, INT64_MIN to placate linters

 sdks/python/apache_beam/transforms/cy_combiners.py | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (af864b8 -> 1693e7d)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from af864b8  Merge pull request #4422: Implement a Local Artifact 
Retrieval service
 add fd07d72  global INT64_MAX, INT64_MIN to placate linters
 new 1693e7d  Merge pull request #4562 global INT64_MAX, INT64_MIN to 
placate linters

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/transforms/cy_combiners.py | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351150#comment-16351150
 ] 

Ben Sidhom commented on BEAM-3587:
--

I built at head and ran a simple program that reads in a text file and writes 
it out to a different file. It seems to work for me.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (504ce70 -> af864b8)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 504ce70  Merge pull request #4539 from 
pabloem/static-runtimevalueprovider
 add 5c01e85  Add a LocalArtifactStagingLocation
 add c26191d  Add LocalArtifactStagingLocation#forExisting
 add e2432e2  Add an ArtifactRetrievalService interface
 add 7a537b9  Implement a Local ArtifactRetrievalService
 new af864b8  Merge pull request #4422: Implement a Local Artifact 
Retrieval service

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../artifact/ArtifactRetrievalService.java}|  15 +-
 .../fnexecution/artifact}/package-info.java|   5 +-
 runners/local-artifact-service-java/build.gradle   |   2 +
 runners/local-artifact-service-java/pom.xml|  11 +
 .../local/LocalArtifactStagingLocation.java| 117 ++
 .../LocalFileSystemArtifactRetrievalService.java   | 120 +++
 .../LocalFileSystemArtifactStagerService.java  |  34 ++-
 .../local/LocalArtifactStagingLocationTest.java| 125 +++
 ...ocalFileSystemArtifactRetrievalServiceTest.java | 240 +
 .../LocalFileSystemArtifactStagerServiceTest.java  |   4 +-
 10 files changed, 636 insertions(+), 37 deletions(-)
 copy 
runners/{flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java 
=> 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactRetrievalService.java}
 (77%)
 copy runners/{apex/src/main/java/org/apache/beam/runners/apex/translation => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact}/package-info.java
 (81%)
 create mode 100644 
runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalArtifactStagingLocation.java
 create mode 100644 
runners/local-artifact-service-java/src/main/java/org/apache/beam/artifact/local/LocalFileSystemArtifactRetrievalService.java
 create mode 100644 
runners/local-artifact-service-java/src/test/java/org/apache/beam/artifact/local/LocalArtifactStagingLocationTest.java
 create mode 100644 
runners/local-artifact-service-java/src/test/java/org/apache/beam/artifact/local/LocalFileSystemArtifactRetrievalServiceTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4422: Implement a Local Artifact Retrieval service

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit af864b804931868d6da3af77a388a637d0db8798
Merge: 504ce70 7a537b9
Author: Thomas Groh 
AuthorDate: Fri Feb 2 16:38:46 2018 -0800

Merge pull request #4422: Implement a Local Artifact Retrieval service

[BEAM-2885]

 .../artifact/ArtifactRetrievalService.java |  24 +++
 .../runners/fnexecution/artifact/package-info.java |  23 ++
 runners/local-artifact-service-java/build.gradle   |   2 +
 runners/local-artifact-service-java/pom.xml|  11 +
 .../local/LocalArtifactStagingLocation.java| 117 ++
 .../LocalFileSystemArtifactRetrievalService.java   | 120 +++
 .../LocalFileSystemArtifactStagerService.java  |  34 ++-
 .../local/LocalArtifactStagingLocationTest.java| 125 +++
 ...ocalFileSystemArtifactRetrievalServiceTest.java | 240 +
 .../LocalFileSystemArtifactStagerServiceTest.java  |   4 +-
 10 files changed, 676 insertions(+), 24 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4539 from pabloem/static-runtimevalueprovider

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 504ce708799d5915a3376f7bb6ca8f8c54187d28
Merge: 4f6415b 5fe88ff
Author: Ahmet Altay 
AuthorDate: Fri Feb 2 16:28:10 2018 -0800

Merge pull request #4539 from pabloem/static-runtimevalueprovider

[BEAM-3570] Adding a static getter for RuntimeValueProvider.

 sdks/python/apache_beam/options/value_provider.py| 17 +++--
 sdks/python/apache_beam/runners/worker/operations.py |  4 ++--
 sdks/python/apache_beam/runners/worker/sideinputs.py |  2 +-
 3 files changed, 14 insertions(+), 9 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (4f6415b -> 504ce70)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4f6415b  Merge pull request #4531 from pabloem/log-sampler-deviation
 add 55d8723  Adding a static getter for RuntimeValueProvider.
 add 25887bc  Addressing comments.
 add 5fe88ff  Removing unnecessary code.
 new 504ce70  Merge pull request #4539 from 
pabloem/static-runtimevalueprovider

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/options/value_provider.py| 17 +++--
 sdks/python/apache_beam/runners/worker/operations.py |  4 ++--
 sdks/python/apache_beam/runners/worker/sideinputs.py |  2 +-
 3 files changed, 14 insertions(+), 9 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4531 from pabloem/log-sampler-deviation

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4f6415b224d9e99c17ccf499728ea6f18759bd9f
Merge: 5e6520a e832cfb
Author: Ahmet Altay 
AuthorDate: Fri Feb 2 16:27:07 2018 -0800

Merge pull request #4531 from pabloem/log-sampler-deviation

[BEAM-2732] Logging deviation from sampling expectation.

 sdks/python/apache_beam/runners/worker/statesampler_test.py | 2 ++
 1 file changed, 2 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (5e6520a -> 4f6415b)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5e6520a  Merge pull request #4569: Add `SdkHarnessClientControlService`
 add e832cfb  Logging deviation from sampling expectation. This will allow 
to track performance variation in statesampler over time.
 new 4f6415b  Merge pull request #4531 from pabloem/log-sampler-deviation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/worker/statesampler_test.py | 2 ++
 1 file changed, 2 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #867

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add SdkHarnessClientControlService

[tgroh] Update Synchronization in FnApiControlClient

[coheigea] BEAM-3593 - Remove methods that just call super()

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5e6520a841d3b1b2959e21969f551a8b66d298dc (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5e6520a841d3b1b2959e21969f551a8b66d298dc
Commit message: "Merge pull request #4569: Add `SdkHarnessClientControlService`"
 > git rev-list 21cc003914272883daff36a352bb6a57fcc6dfe2 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4995573222077604108.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4040151245508403802.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4179322237074404973.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1797352361547077022.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1788630527240470200.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5109004949940480837.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #94

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-3481) Query with subquery and aggregates cannot be implemented.

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351089#comment-16351089
 ] 

Anton Kedin edited comment on BEAM-3481 at 2/3/18 12:06 AM:


Ah, I see, the above works, but if you remove the `500`, then it fails


was (Author: kedin):
Ah, I see, the above works, but if you remove the `500` it fails

> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Assignee: Anton Kedin
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>  
> {noformat}
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
>  LogicalAggregate(group=[{0}])
>  LogicalProject(p=[$0])
>  LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be 
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}], 
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu, 
> 0.0 io}, id = 155
>  LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount = 
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
>  LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0, 
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
>  LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]): 
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
>  rel#150:Subset#0.NONE.[], best=null, importance=0.6561
>  rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0, 
> cumulative cost={inf}
>  rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
>  rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0, 
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
>  rel#152:Subset#1.NONE.[], best=null, importance=0.7291
>  rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0), 
> rowcount=100.0, cumulative cost={inf}
>  rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163, 
> importance=0.36455
>  
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
>  rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
>  rel#154:Subset#2.NONE.[], best=null, importance=0.81
>  rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}), 
> rowcount=10.0, cumulative cost={inf}
>  rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>  
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
>  rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
>  rel#156:Subset#3.NONE.[], best=null, importance=0.9
>  
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
>  rowcount=1.0, cumulative cost={inf}
>  rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
>  importance=1.0
>  
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
>  rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
>  (RelSubset.java:441)
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #97

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #111

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3481) Query with subquery and aggregates cannot be implemented.

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351089#comment-16351089
 ] 

Anton Kedin commented on BEAM-3481:
---

Ah, I see, the above works, but if you remove the `500` it fails

> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Assignee: Anton Kedin
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>  
> {noformat}
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
>  LogicalAggregate(group=[{0}])
>  LogicalProject(p=[$0])
>  LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be 
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}], 
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu, 
> 0.0 io}, id = 155
>  LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount = 
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
>  LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0, 
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
>  LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]): 
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
>  rel#150:Subset#0.NONE.[], best=null, importance=0.6561
>  rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0, 
> cumulative cost={inf}
>  rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
>  rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0, 
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
>  rel#152:Subset#1.NONE.[], best=null, importance=0.7291
>  rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0), 
> rowcount=100.0, cumulative cost={inf}
>  rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163, 
> importance=0.36455
>  
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
>  rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
>  rel#154:Subset#2.NONE.[], best=null, importance=0.81
>  rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}), 
> rowcount=10.0, cumulative cost={inf}
>  rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>  
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
>  rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
>  rel#156:Subset#3.NONE.[], best=null, importance=0.9
>  
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
>  rowcount=1.0, cumulative cost={inf}
>  rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
>  importance=1.0
>  
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
>  rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
>  (RelSubset.java:441)
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3481) Query with subquery and aggregates cannot be implemented.

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-3481:
-

Assignee: Anton Kedin

> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Assignee: Anton Kedin
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>  
> {noformat}
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
>  LogicalAggregate(group=[{0}])
>  LogicalProject(p=[$0])
>  LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be 
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}], 
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu, 
> 0.0 io}, id = 155
>  LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount = 
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
>  LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0, 
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
>  LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]): 
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
>  rel#150:Subset#0.NONE.[], best=null, importance=0.6561
>  rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0, 
> cumulative cost={inf}
>  rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
>  rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0, 
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
>  rel#152:Subset#1.NONE.[], best=null, importance=0.7291
>  rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0), 
> rowcount=100.0, cumulative cost={inf}
>  rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163, 
> importance=0.36455
>  
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
>  rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
>  rel#154:Subset#2.NONE.[], best=null, importance=0.81
>  rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}), 
> rowcount=10.0, cumulative cost={inf}
>  rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>  
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
>  rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
>  rel#156:Subset#3.NONE.[], best=null, importance=0.9
>  
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
>  rowcount=1.0, cumulative cost={inf}
>  rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
>  importance=1.0
>  
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
>  rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
>  (RelSubset.java:441)
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3481) Query with subquery and aggregates cannot be implemented.

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351080#comment-16351080
 ] 

Anton Kedin commented on BEAM-3481:
---

[~AustinHaas], is it a full stack trace? Is it the same in the second case? Can 
you post your pipeline configuration?

I am trying to reproduce on the latest master, and this succeeds:
{code:sql|title=SQL}
"SELECT COUNT(`?p`) as `n`, 500 as `f` FROM " +
"(SELECT p as `?p`, 500 FROM `contains` GROUP BY p) as t1";
{code}
{code:sql|title=Plan}
Feb 02, 2018 3:53:02 PM 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
validateAndConvert
INFO: SQL:
SELECT COUNT(`t1`.`?p`) AS `n`, 500 AS `f`
FROM (SELECT `contains`.`p` AS `?p`, 500
FROM `contains` AS `contains`
GROUP BY `contains`.`p`) AS `t1`
Feb 02, 2018 3:53:02 PM 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
convertToBeamRel
INFO: SQLPlan>
LogicalProject(n=[$0], f=[500])
  LogicalAggregate(group=[{}], n=[COUNT()])
LogicalProject(?p=[$0])
  LogicalProject(?p=[$0], EXPR$1=[500])
LogicalAggregate(group=[{0}])
  LogicalProject(?p=[$1])
LogicalTableScan(table=[[contains]])
{code}

> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>  
> {noformat}
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
>  LogicalAggregate(group=[{0}])
>  LogicalProject(p=[$0])
>  LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be 
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}], 
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu, 
> 0.0 io}, id = 155
>  LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount = 
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
>  LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0, 
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
>  LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]): 
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
>  rel#150:Subset#0.NONE.[], best=null, importance=0.6561
>  rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0, 
> cumulative cost={inf}
>  rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
>  rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0, 
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
>  rel#152:Subset#1.NONE.[], best=null, importance=0.7291
>  rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0), 
> rowcount=100.0, cumulative cost={inf}
>  rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163, 
> importance=0.36455
>  
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
>  rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
>  rel#154:Subset#2.NONE.[], best=null, importance=0.81
>  rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}), 
> rowcount=10.0, cumulative cost={inf}
>  rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>  
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
>  rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
>  rel#156:Subset#3.NONE.[], best=null, importance=0.9
>  
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
>  rowcount=1.0, cumulative cost={inf}
>  rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
>  importance=1.0
>  
> 

[jira] [Commented] (BEAM-3460) On fresh checkout, failure of "./gradlew build" for Go due to "Cannot recognized package: common"

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351058#comment-16351058
 ] 

Henning Rohde commented on BEAM-3460:
-

I just get the tox error, when I try to repro. Debug logs show that the task 
completes:
 

15:23:11.653 [DEBUG] 
[org.gradle.internal.progress.DefaultBuildOperationExecutor] Build operation 
'Task :sdks:go:resolveBuildDependencies' completed

[...]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdks:python:lint'.
> A problem occurred starting process 'command 'tox''

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
':sdks:python:lint'.
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:100)
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:70)
at 
org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:62)
[...]


> On fresh checkout, failure of "./gradlew build" for Go due to "Cannot 
> recognized package: common"
> -
>
> Key: BEAM-3460
> URL: https://issues.apache.org/jira/browse/BEAM-3460
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Henning Rohde
>Priority: Major
> Fix For: Not applicable
>
>
> After {{./gradlew clean && ./gradlew build}} the python build fails because 
> of an issue in dependency resolution in Go.
> {code}
> FAILURE: Build completed with 2 failures.
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
> > A problem occurred starting process 'command 'tox''
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> ==
> 2: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task 
> ':beam-sdks-parent:beam-sdks-go:resolveBuildDependencies'.
> > Exception in resolution, message is:
>   Cannot recognized package: common
>   Resolution stack is:
>   +- github.com/apache/beam/sdks/go
>+- golang.org/x/oauth2#30785a2c434e431ef7c507b54617d6a951d5f2b4
> +- cloud.google.com/go#8e80142a8cf430971bc659922cabeb64c553bef5
>  +- go.opencensus.io#2191e3e86ea6a0d94dcf946813ce50cb3b88502f
>   +- 
> github.com/prometheus/client_golang#f02bfc3484a6b03d1fc00d72d86add103ef9567b
>+- 
> github.com/prometheus/common#89604d197083d4781071d3c65855d24ecfb0a563
> +- github.com/go-kit/kit#953e747656a7bbb5e1f998608b460458958b70cc
>  +- github.com/apache/thrift#a15060acd81d72fa364a7aab327aa8c41d02958f
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> ==
> * Get more help at https://help.gradle.org
> BUILD FAILED in 6m 13s
> 92 actionable tasks: 92 executed
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3460) On fresh checkout, failure of "./gradlew build" for Go due to "Cannot recognized package: common"

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3460.
-
   Resolution: Cannot Reproduce
Fix Version/s: Not applicable

> On fresh checkout, failure of "./gradlew build" for Go due to "Cannot 
> recognized package: common"
> -
>
> Key: BEAM-3460
> URL: https://issues.apache.org/jira/browse/BEAM-3460
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Henning Rohde
>Priority: Major
> Fix For: Not applicable
>
>
> After {{./gradlew clean && ./gradlew build}} the python build fails because 
> of an issue in dependency resolution in Go.
> {code}
> FAILURE: Build completed with 2 failures.
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
> > A problem occurred starting process 'command 'tox''
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> ==
> 2: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task 
> ':beam-sdks-parent:beam-sdks-go:resolveBuildDependencies'.
> > Exception in resolution, message is:
>   Cannot recognized package: common
>   Resolution stack is:
>   +- github.com/apache/beam/sdks/go
>+- golang.org/x/oauth2#30785a2c434e431ef7c507b54617d6a951d5f2b4
> +- cloud.google.com/go#8e80142a8cf430971bc659922cabeb64c553bef5
>  +- go.opencensus.io#2191e3e86ea6a0d94dcf946813ce50cb3b88502f
>   +- 
> github.com/prometheus/client_golang#f02bfc3484a6b03d1fc00d72d86add103ef9567b
>+- 
> github.com/prometheus/common#89604d197083d4781071d3c65855d24ecfb0a563
> +- github.com/go-kit/kit#953e747656a7bbb5e1f998608b460458958b70cc
>  +- github.com/apache/thrift#a15060acd81d72fa364a7aab327aa8c41d02958f
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> ==
> * Get more help at https://help.gradle.org
> BUILD FAILED in 6m 13s
> 92 actionable tasks: 92 executed
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-3388] Remove reflection from varint codecs

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 52a598d33d5295aba8e4c15c30d059d905c73c49
Merge: 4b69c36 bd97498
Author: Lukasz Cwik 
AuthorDate: Fri Feb 2 14:59:35 2018 -0800

[BEAM-3388] Remove reflection from varint codecs

 sdks/go/pkg/beam/core/runtime/coderx/varint.go | 65 --
 1 file changed, 60 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch go-sdk updated (4b69c36 -> 52a598d)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4b69c36  Merge pull request #4582: [BEAM-3597] Add optional function 
registration to Go SDK runtime
 add bd97498  fixup! Remove reflection from varint codecs
 new 52a598d  [BEAM-3388] Remove reflection from varint codecs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/coderx/varint.go | 65 --
 1 file changed, 60 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Created] (BEAM-3612) Make it easy to generate type-specialized Go SDK reflectx.Funcs

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-3612:
---

 Summary: Make it easy to generate type-specialized Go SDK 
reflectx.Funcs
 Key: BEAM-3612
 URL: https://issues.apache.org/jira/browse/BEAM-3612
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Henning Rohde






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (c66832c -> 5e6520a)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c66832c  BEAM-3593 - Remove methods that just call super()
 add 1eb9443  Add SdkHarnessClientControlService
 add ae603d2  Update Synchronization in FnApiControlClient
 new 5e6520a  Merge pull request #4569: Add `SdkHarnessClientControlService`

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../fnexecution/control/FnApiControlClient.java| 30 ++---
 .../control/FnApiControlClientPoolService.java |  5 +-
 .../fnexecution/control/SdkHarnessClient.java  |  7 +-
 .../control/SdkHarnessClientControlService.java| 75 ++
 .../sdk/fn/stream/SynchronizedStreamObserver.java  | 64 ++
 5 files changed, 169 insertions(+), 12 deletions(-)
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientControlService.java
 create mode 100644 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Resolved] (BEAM-3593) Remove methods that just call super()

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3593.
-
Resolution: Fixed

> Remove methods that just call super()
> -
>
> Key: BEAM-3593
> URL: https://issues.apache.org/jira/browse/BEAM-3593
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Colm O hEigeartaigh
>Assignee: Colm O hEigeartaigh
>Priority: Trivial
> Fix For: 2.4.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #866

[Apache Jenkins Server]

See 


Changes:

[robertwb] Reduce the flakiness of the state sampler progress metrics.

[kedin] [SQL] Inherit windowing strategy from the input in Aggregate operation

[jbonofre] [BEAM-3551] Define compiler -parameters flag in the default options

[lcwik] Move off of deprecated method in Guava.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 21cc003914272883daff36a352bb6a57fcc6dfe2 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 21cc003914272883daff36a352bb6a57fcc6dfe2
Commit message: "Merge pull request #4546: [SQL] Inherit windowing strategy 
from the input in Aggregate operation"
 > git rev-list 9cf86bcebcdbd8d5a84777cf2871597f0ba1b951 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2578158365985461499.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4574589399925147037.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5728173047620967361.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1561706708763531326.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins905477893136226474.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins593717921323410652.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning

[beam] branch master updated (21cc003 -> c66832c)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 21cc003  Merge pull request #4546: [SQL] Inherit windowing strategy 
from the input in Aggregate operation
 add 0dec2e7  BEAM-3593 - Remove methods that just call super()
 new c66832c  BEAM-3593 - Remove methods that just call super()

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java   | 5 -
 .../java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java  | 6 --
 .../java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java  | 5 -
 3 files changed, 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: BEAM-3593 - Remove methods that just call super()

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c66832cf02522e77db0fe23d73dd7edc1ad922e1
Merge: 21cc003 0dec2e7
Author: Lukasz Cwik 
AuthorDate: Fri Feb 2 12:40:41 2018 -0800

BEAM-3593 - Remove methods that just call super()

 .../core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java   | 5 -
 .../java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java  | 6 --
 .../java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java  | 5 -
 3 files changed, 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Created] (BEAM-3611) Split KafkaIO.java into smaller files

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-3611:
--

 Summary: Split KafkaIO.java into smaller files
 Key: BEAM-3611
 URL: https://issues.apache.org/jira/browse/BEAM-3611
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Raghu Angadi
Assignee: Reuven Lax
 Fix For: 2.4.0


KafkaIO.java has grown too big and includes both source and sink 
implementation. Better to move these to own files. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3611) Split KafkaIO.java into smaller files

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi reassigned BEAM-3611:
--

Assignee: Raghu Angadi  (was: Reuven Lax)

> Split KafkaIO.java into smaller files
> -
>
> Key: BEAM-3611
> URL: https://issues.apache.org/jira/browse/BEAM-3611
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: 2.4.0
>
>
> KafkaIO.java has grown too big and includes both source and sink 
> implementation. Better to move these to own files. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #93

[Apache Jenkins Server]

See 


Changes:

[robertwb] Reduce the flakiness of the state sampler progress metrics.

[kedin] [SQL] Inherit windowing strategy from the input in Aggregate operation

[jbonofre] [BEAM-3551] Define compiler -parameters flag in the default options

[lcwik] Move off of deprecated method in Guava.

--
[...truncated 10.05 KB...]
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7799696085149354719.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: hdfs3<0.4.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.4.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 

[jira] [Resolved] (BEAM-3597) Add function registration in Go SDK to avoid symbol table lookups

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3597.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Add function registration in Go SDK to avoid symbol table lookups
> -
>
> Key: BEAM-3597
> URL: https://issues.apache.org/jira/browse/BEAM-3597
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
> Fix For: Not applicable
>
>
> We should allow optional function registration at init-time to avoid reading 
> symbols from the binary, which is slow. It would also allow runtime 
> environment where we do not have that ability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #110

[Apache Jenkins Server]

See 


Changes:

[robertwb] Reduce the flakiness of the state sampler progress metrics.

[kedin] [SQL] Inherit windowing strategy from the input in Aggregate operation

[jbonofre] [BEAM-3551] Define compiler -parameters flag in the default options

[lcwik] Move off of deprecated method in Guava.

--
[...truncated 12.00 KB...]
  Using cached typing-3.6.4-py2-none-any.whl
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
./.env/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Collecting hdfs3<0.4.0,>=0.3.0 (from apache-beam==2.4.0.dev0)
  Using cached hdfs3-0.3.0-py2.py3-none-any.whl
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.4.0.dev0)
  Using cached google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.4.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.4.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.4.0.dev0)
  Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.4.0.dev0)
  Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.4.0.dev0)
  Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.7 in 
/usr/local/lib/python2.7/dist-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Collecting fasteners>=0.14 (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth-1.3.0-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting cachetools>=2.0.0 (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting future<0.17dev,>=0.16.0 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting ply==3.8 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/usr/local/lib/python2.7/dist-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #96

[Apache Jenkins Server]

See 


Changes:

[robertwb] Reduce the flakiness of the state sampler progress metrics.

[kedin] [SQL] Inherit windowing strategy from the input in Aggregate operation

[jbonofre] [BEAM-3551] Define compiler -parameters flag in the default options

[lcwik] Move off of deprecated method in Guava.

--
[...truncated 12.00 KB...]
  Using cached typing-3.6.4-py2-none-any.whl
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
./.env/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Collecting hdfs3<0.4.0,>=0.3.0 (from apache-beam==2.4.0.dev0)
  Using cached hdfs3-0.3.0-py2.py3-none-any.whl
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.4.0.dev0)
  Using cached google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.4.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.4.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.4.0.dev0)
  Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.4.0.dev0)
  Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.4.0.dev0)
  Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.7 in 
/usr/local/lib/python2.7/dist-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Collecting fasteners>=0.14 (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth-1.3.0-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting cachetools>=2.0.0 (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting future<0.17dev,>=0.16.0 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting ply==3.8 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/usr/local/lib/python2.7/dist-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 

[beam] branch go-sdk updated (f7e4e41 -> 4b69c36)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f7e4e41  Merge pull request #4556: Fix beam.Combine to combine globally
 add 6834287  Add optional function registration to Go SDK runtime
 new 4b69c36  Merge pull request #4582: [BEAM-3597] Add optional function 
registration to Go SDK runtime

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/coder.go  | 10 ++-
 sdks/go/pkg/beam/core/graph/coder/int.go   | 66 ---
 sdks/go/pkg/beam/core/graph/coder/varint.go| 56 -
 sdks/go/pkg/beam/core/graph/coder/varint_test.go   | 64 ---
 .../core/{graph/coder => runtime/coderx}/int.go| 79 +-
 .../core/{graph/coder => runtime/coderx}/varint.go | 96 +++---
 .../{graph/coder => runtime/coderx}/varint_test.go | 76 +
 sdks/go/pkg/beam/core/runtime/graphx/coder_test.go |  6 +-
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  | 11 +--
 .../pkg/beam/core/runtime/graphx/translate_test.go |  4 +-
 sdks/go/pkg/beam/core/runtime/graphx/user.go   |  8 +-
 sdks/go/pkg/beam/core/runtime/symbols.go   | 84 ++-
 sdks/go/pkg/beam/encoding.go   |  8 +-
 sdks/go/pkg/beam/forward.go|  8 ++
 sdks/go/pkg/beam/io/textio/textio.go   |  2 +
 sdks/go/pkg/beam/partition.go  |  4 +
 sdks/go/pkg/beam/partition_test.go |  8 +-
 sdks/go/pkg/beam/testing/ptest/symbols.go  | 67 ---
 sdks/go/pkg/beam/transforms/filter/distinct.go |  5 ++
 sdks/go/pkg/beam/util.go   |  7 ++
 sdks/go/pkg/beam/x/debug/print.go  |  1 +
 21 files changed, 133 insertions(+), 537 deletions(-)
 copy sdks/go/pkg/beam/core/{graph/coder => runtime/coderx}/int.go (51%)
 copy sdks/go/pkg/beam/core/{graph/coder => runtime/coderx}/varint.go (54%)
 copy sdks/go/pkg/beam/core/{graph/coder => runtime/coderx}/varint_test.go (55%)
 delete mode 100644 sdks/go/pkg/beam/testing/ptest/symbols.go

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4582: [BEAM-3597] Add optional function registration to Go SDK runtime

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4b69c364d5549ea5bb161457c13a228ee8637cce
Merge: f7e4e41 6834287
Author: Kenn Knowles 
AuthorDate: Fri Feb 2 11:50:31 2018 -0800

Merge pull request #4582: [BEAM-3597] Add optional function registration to 
Go SDK runtime

 sdks/go/pkg/beam/coder.go  | 10 ++-
 sdks/go/pkg/beam/core/graph/coder/int.go   | 66 ---
 sdks/go/pkg/beam/core/graph/coder/varint.go| 56 -
 sdks/go/pkg/beam/core/graph/coder/varint_test.go   | 64 ---
 .../core/{graph/coder => runtime/coderx}/int.go| 79 +-
 .../core/{graph/coder => runtime/coderx}/varint.go | 96 +++---
 .../{graph/coder => runtime/coderx}/varint_test.go | 76 +
 sdks/go/pkg/beam/core/runtime/graphx/coder_test.go |  6 +-
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  | 11 +--
 .../pkg/beam/core/runtime/graphx/translate_test.go |  4 +-
 sdks/go/pkg/beam/core/runtime/graphx/user.go   |  8 +-
 sdks/go/pkg/beam/core/runtime/symbols.go   | 84 ++-
 sdks/go/pkg/beam/encoding.go   |  8 +-
 sdks/go/pkg/beam/forward.go|  8 ++
 sdks/go/pkg/beam/io/textio/textio.go   |  2 +
 sdks/go/pkg/beam/partition.go  |  4 +
 sdks/go/pkg/beam/partition_test.go |  8 +-
 sdks/go/pkg/beam/testing/ptest/symbols.go  | 67 ---
 sdks/go/pkg/beam/transforms/filter/distinct.go |  5 ++
 sdks/go/pkg/beam/util.go   |  7 ++
 sdks/go/pkg/beam/x/debug/print.go  |  1 +
 21 files changed, 133 insertions(+), 537 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #5848

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin resolved BEAM-3610.
---
   Resolution: Fixed
 Assignee: Anton Kedin
Fix Version/s: Not applicable

> [SQL] Fix HOP definition
> 
>
> Key: BEAM-3610
> URL: https://issues.apache.org/jira/browse/BEAM-3610
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> Current HOP implementation has parameters in wrong order, according to 
> Calcite doc. Sliding window size and frequency are swapped
>  
> According to [Calcite 
> docs|https://calcite.apache.org/docs/reference.html#grouped-window-functions] 
> the syntax is:
> {code:SQL}
> HOP(timestamp_field, frequency_interval, window_size)
> {code}
> We have 
> {code:SQL}
> HOP(timestamp_field, window_size, frequency_interval)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-3610.
-

> [SQL] Fix HOP definition
> 
>
> Key: BEAM-3610
> URL: https://issues.apache.org/jira/browse/BEAM-3610
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> Current HOP implementation has parameters in wrong order, according to 
> Calcite doc. Sliding window size and frequency are swapped
>  
> According to [Calcite 
> docs|https://calcite.apache.org/docs/reference.html#grouped-window-functions] 
> the syntax is:
> {code:SQL}
> HOP(timestamp_field, frequency_interval, window_size)
> {code}
> We have 
> {code:SQL}
> HOP(timestamp_field, window_size, frequency_interval)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3362) Create an example pipeline that uses State.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3362.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> Create an example pipeline that uses State.
> ---
>
> Key: BEAM-3362
> URL: https://issues.apache.org/jira/browse/BEAM-3362
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Minor
> Fix For: 2.3.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Expand the Gaming example to include Stateful processing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #227

[Apache Jenkins Server]

See 


--
GitHub pull request #4585 of commit df5d5202d28fa811c9f9acbb1c5b2c7a3f94163e, 
no merge conflicts.
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4585/*:refs/remotes/origin/pr/4585/*
 > git rev-parse refs/remotes/origin/pr/4585/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4585/merge^{commit} # timeout=10
Checking out Revision faf575fde93b54c7359565c4c2fecdf1956eab85 
(refs/remotes/origin/pr/4585/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f faf575fde93b54c7359565c4c2fecdf1956eab85
Commit message: "Merge df5d5202d28fa811c9f9acbb1c5b2c7a3f94163e into 
2bbcb122bc34e668cdbebfae95c0505bfc58f43b"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5943464719987172990.sh
+ pwd

[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins295412269632828.sh
+ ls -la
total 20
drwxr-xr-x  5 jenkins jenkins 4096 Jan 29 13:10 .
drwxr-xr-x 34 jenkins jenkins 4096 Feb  2 12:14 ..
drwxr-xr-x  6 jenkins jenkins 4096 Jan 29 13:10 .env
drwxr-xr-x  7 jenkins jenkins 4096 Jan 29 13:10 PerfKitBenchmarker
drwxr-xr-x 11 jenkins jenkins 4096 Feb  2 19:21 src
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8424462220748789240.sh
+ ls -la .kube/
ls: cannot access .kube/: No such file or directory
Build step 'Execute shell' marked build as failure

[jira] [Comment Edited] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350784#comment-16350784
 ] 

Anton Kedin edited comment on BEAM-3610 at 2/2/18 7:11 PM:
---

[This PR|https://github.com/apache/beam/pull/4546] fixes this:
 - [Old 
implementation|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-b0b2c6af2f694e5f6d9a1c64cbc482a1L105];
 - [New 
Implementation|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-7cc1c86643991914521ada54ecf53ceeR104];
 - 
[Tests|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-4f4ffa265fe666e99c37c346d50da67dR320];


was (Author: kedin):
[This PR|https://github.com/apache/beam/pull/4546] fixes this:
 - [Old 
implementation|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-b0b2c6af2f694e5f6d9a1c64cbc482a1L105];
 - 
[Implementation|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-7cc1c86643991914521ada54ecf53ceeR104];
 - 
[Tests|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-4f4ffa265fe666e99c37c346d50da67dR320];

> [SQL] Fix HOP definition
> 
>
> Key: BEAM-3610
> URL: https://issues.apache.org/jira/browse/BEAM-3610
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Current HOP implementation has parameters in wrong order, according to 
> Calcite doc. Sliding window size and frequency are swapped
>  
> According to [Calcite 
> docs|https://calcite.apache.org/docs/reference.html#grouped-window-functions] 
> the syntax is:
> {code:SQL}
> HOP(timestamp_field, frequency_interval, window_size)
> {code}
> We have 
> {code:SQL}
> HOP(timestamp_field, window_size, frequency_interval)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350784#comment-16350784
 ] 

Anton Kedin edited comment on BEAM-3610 at 2/2/18 7:10 PM:
---

[This PR|https://github.com/apache/beam/pull/4546] fixes this:
 - [Old 
implementation|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-b0b2c6af2f694e5f6d9a1c64cbc482a1L105];
 - 
[Implementation|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-7cc1c86643991914521ada54ecf53ceeR104];
 - 
[Tests|https://github.com/apache/beam/commit/ecd89b856ffd85baa769ecbe27615ffaf79f2195#diff-4f4ffa265fe666e99c37c346d50da67dR320];


was (Author: kedin):
[This PR|https://github.com/apache/beam/pull/4546] fixes this

> [SQL] Fix HOP definition
> 
>
> Key: BEAM-3610
> URL: https://issues.apache.org/jira/browse/BEAM-3610
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Current HOP implementation has parameters in wrong order, according to 
> Calcite doc. Sliding window size and frequency are swapped
>  
> According to [Calcite 
> docs|https://calcite.apache.org/docs/reference.html#grouped-window-functions] 
> the syntax is:
> {code:SQL}
> HOP(timestamp_field, frequency_interval, window_size)
> {code}
> We have 
> {code:SQL}
> HOP(timestamp_field, window_size, frequency_interval)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3610:
--
Description: 
Current HOP implementation has parameters in wrong order, according to Calcite 
doc. Sliding window size and frequency are swapped

 

According to [Calcite 
docs|https://calcite.apache.org/docs/reference.html#grouped-window-functions] 
the syntax is:
{code:SQL}
HOP(timestamp_field, frequency_interval, window_size)
{code}

We have 

{code:SQL}
HOP(timestamp_field, window_size, frequency_interval)
{code}

  was:Current HOP implementation has parameters in wrong order, according to 
Calcite doc. Sliding window size and frequency are swapped


> [SQL] Fix HOP definition
> 
>
> Key: BEAM-3610
> URL: https://issues.apache.org/jira/browse/BEAM-3610
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Current HOP implementation has parameters in wrong order, according to 
> Calcite doc. Sliding window size and frequency are swapped
>  
> According to [Calcite 
> docs|https://calcite.apache.org/docs/reference.html#grouped-window-functions] 
> the syntax is:
> {code:SQL}
> HOP(timestamp_field, frequency_interval, window_size)
> {code}
> We have 
> {code:SQL}
> HOP(timestamp_field, window_size, frequency_interval)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4546: [SQL] Inherit windowing strategy from the input in Aggregate operation

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 21cc003914272883daff36a352bb6a57fcc6dfe2
Merge: 2bbcb12 ecd89b8
Author: Kenn Knowles 
AuthorDate: Fri Feb 2 11:02:02 2018 -0800

Merge pull request #4546: [SQL] Inherit windowing strategy from the input 
in Aggregate operation

 .../sql/impl/rel/BeamAggregationRel.java   | 120 ++--
 .../sdk/extensions/sql/impl/rel/BeamUnionRel.java  |   5 -
 .../sdk/extensions/sql/impl/rel/package-info.java  |   4 +
 .../sql/impl/rule/AggregateWindowFactory.java  | 158 +
 .../sql/impl/rule/AggregateWindowField.java|  48 +++
 .../sql/impl/rule/BeamAggregationRule.java |  94 ++--
 .../sdk/extensions/sql/impl/rule/package-info.java |   4 +
 .../extensions/sql/BeamSqlDslAggregationTest.java  | 144 ++-
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java|  15 +-
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  83 +++
 10 files changed, 538 insertions(+), 137 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (2bbcb12 -> 21cc003)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2bbcb12  Move off of deprecated method in Guava.
 add ecd89b8  [SQL] Inherit windowing strategy from the input in Aggregate 
operation
 new 21cc003  Merge pull request #4546: [SQL] Inherit windowing strategy 
from the input in Aggregate operation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sql/impl/rel/BeamAggregationRel.java   | 120 ++--
 .../sdk/extensions/sql/impl/rel/BeamUnionRel.java  |   5 -
 .../sdk/extensions/sql/impl/rel/package-info.java  |   4 +
 .../sql/impl/rule/AggregateWindowFactory.java  | 158 +
 .../sql/impl/rule/AggregateWindowField.java}   |  41 +++---
 .../sql/impl/rule/BeamAggregationRule.java |  94 ++--
 .../sdk/extensions/sql/impl/rule/package-info.java |   4 +
 .../extensions/sql/BeamSqlDslAggregationTest.java  | 144 ++-
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java|  15 +-
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  83 +++
 10 files changed, 508 insertions(+), 160 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/AggregateWindowFactory.java
 copy 
sdks/java/extensions/{google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/NoopPathValidator.java
 => 
sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/AggregateWindowField.java}
 (53%)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350806#comment-16350806
 ] 

Ben Sidhom commented on BEAM-3587:
--

As Ken mentioned, this is strange since we have ValidatesRunner tests that 
should in theory trigger this. Can we get a minimal example pipeline that hits 
this issue?

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2943) Beam Flink deployment results in ClassNotFoundException

[Guenther Grill (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350790#comment-16350790
 ] 

Guenther Grill commented on BEAM-2943:
--

As far as I can remember, I just followed the instructions under 
[https://beam.apache.org/get-started/quickstart-java/]  where the library was 
built via 
{code:java}
mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ 
-Dexec.args="--runner=FlinkRunner --flinkMaster= 
--filesToStage=target/word-count-beam-bundled-0.1.jar \ 
--inputFile=/path/to/quickstart/pom.xml --output=/tmp/counts" 
-Pflink-runner{code}

> Beam Flink deployment results in ClassNotFoundException
> ---
>
> Key: BEAM-2943
> URL: https://issues.apache.org/jira/browse/BEAM-2943
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0
> Environment: Debian 9.1 / 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 
> (2017-08-06) x86_64 GNU/Linux
>Reporter: Guenther Grill
>Assignee: Aljoscha Krettek
>Priority: Major
>  Labels: flink
>
> Hi,
> I followed the guide https://beam.apache.org/get-started/quickstart-java/ to 
> run beam program within a flink cluster. 
> The output of the dependency-command is:
> {code}
> mvn dependency:tree -Pflink-runner |grep flink
>   
> [INFO] \- org.apache.beam:beam-runners-flink_2.10:jar:2.1.0:runtime
> [INFO]+- org.apache.flink:flink-clients_2.10:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-optimizer_2.10:jar:1.3.0:runtime
> [INFO]|  \- org.apache.flink:force-shading:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-core:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-annotations:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-metrics-core:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-java:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-shaded-hadoop2:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-runtime_2.10:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-streaming-java_2.10:jar:1.3.0:runtime
> {code}
> Then I started the flink cluster with the correct version with docker-compose
> {code}
> export JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
> export FLINK_DOCKER_IMAGE_NAME=flink:1.3.0-hadoop27-scala_2.10
> docker-compose up -d
> {code}
> The compose file looks like this:
> {code}
> version: '3.3'
> services:
>   jobmanager:
> image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
> expose:
>   - "6123"
> ports:
>   - "6123:6123"
>   - "8081:8081"
> volumes:
>   - /tmp:/tmp
> command: jobmanager
> environment:
>   - JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
>   taskmanager:
> image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
> expose:
>   - "6121"
>   - "6122"
> depends_on:
>   - jobmanager
> command: taskmanager
> environment:
>   - JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
> {code}
> The flink cluster works, but when I execute 
> {code}
> mvn exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
> -Pflink-runner \
> -Dexec.args="--runner=FlinkRunner \
>   --inputFile=pom.xml \
>   --output=/path/to/counts \
>   --flinkMaster=[HOST_IP]:6123 \
>   --filesToStage=target/word-count-beam-bundled-0.1.jar"
> {code}
> I get:
> {code}
> 2017-09-12 06:39:57,226 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Submitting job a913f922506053e65e732eeb8336b3bd 
> (wordcount-grg-0912063956-c7ea6199).
> 2017-09-12 06:39:57,227 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Using restart strategy NoRestartStrategy for 
> a913f922506053e65e732eeb8336b3bd.
> 2017-09-12 06:39:57,227 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> 2017-09-12 06:39:57,229 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Running initialization on master for job 
> wordcount-grg-0912063956-c7ea6199 (a913f922506053e65e732eeb8336b3bd).
> 2017-09-12 06:39:57,230 ERROR org.apache.flink.runtime.jobmanager.JobManager  
>   - Failed to submit job a913f922506053e65e732eeb8336b3bd 
> (wordcount-grg-0912063956-c7ea6199)
> org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 
> 'DataSource (at Read(CompressedSource) 
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))': 
> Deserializing the InputFormat 
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat@58e7a91a)
>  failed: Could not read the user code wrapper: 
> org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat
>   at 
> 

[jira] [Commented] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350784#comment-16350784
 ] 

Anton Kedin commented on BEAM-3610:
---

[This PR|https://github.com/apache/beam/pull/4546] fixes this

> [SQL] Fix HOP definition
> 
>
> Key: BEAM-3610
> URL: https://issues.apache.org/jira/browse/BEAM-3610
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Current HOP implementation has parameters in wrong order, according to 
> Calcite doc. Sliding window size and frequency are swapped



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3610) [SQL] Fix HOP definition

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3610:
-

 Summary: [SQL] Fix HOP definition
 Key: BEAM-3610
 URL: https://issues.apache.org/jira/browse/BEAM-3610
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Anton Kedin


Current HOP implementation has parameters in wrong order, according to Calcite 
doc. Sliding window size and frequency are swapped



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3609) [SQL] Support Hive UDFs

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3609:
-

 Summary: [SQL] Support Hive UDFs
 Key: BEAM-3609
 URL: https://issues.apache.org/jira/browse/BEAM-3609
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Anton Kedin


Consider supporting existing UDF libraries, e.g.:

https://github.com/Esri/spatial-framework-for-hadoop/blob/master/hive/src/main/java/com/esri/hadoop/hive/ST_Aggr_Union.java

Will probably need to implement a layer on top of 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/UDF.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3600) Do not ignore FileSystem errors and document expected behavior

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350763#comment-16350763
 ] 

Udi Meiri commented on BEAM-3600:
-

Discussion: 
https://lists.apache.org/thread.html/92f3e652d3f961d67013ce997c4dfff0a1fab115e9eeefa2d9de5c19@%3Cdev.beam.apache.org%3E

> Do not ignore FileSystem errors and document expected behavior
> --
>
> Key: BEAM-3600
> URL: https://issues.apache.org/jira/browse/BEAM-3600
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> copy/rename:
>  * it should be an error if the dst file exists
>  * it should be an error if the src file doesn't exist
> delete:
>  * it should be an error if the file/dir doesn't exist
> FileBasedSink.finalize_write:
>  * should check (src, dst) pairs for existence:
>  ** src only - regular rename
>  ** dst only - skip rename
>  ** both src and dst - if files don't match on metadata (checksum or size), 
> delete dst then rename, otherwise delete src and skip rename
>  ** neither exist - return error: bad state
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Move off of deprecated method in Guava.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2bbcb12  Move off of deprecated method in Guava.
2bbcb12 is described below

commit 2bbcb122bc34e668cdbebfae95c0505bfc58f43b
Author: Luke Cwik 
AuthorDate: Thu Feb 1 17:06:55 2018 -0800

Move off of deprecated method in Guava.
---
 .../apache/beam/runners/fnexecution/control/SdkHarnessClient.java | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
index 27c9401..adaa2c7 100644
--- 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
@@ -119,7 +119,10 @@ public class SdkHarnessClient {
   processBundleDescriptorId);
 
   ListenableFuture specificResponse =
-  Futures.transform(genericResponse, 
InstructionResponse::getProcessBundle);
+  Futures.transform(
+  genericResponse,
+  InstructionResponse::getProcessBundle,
+  MoreExecutors.directExecutor());
   Map outputClients = new HashMap<>();
   for (Map.Entry targetReceiver 
:
   outputReceivers.entrySet()) {
@@ -237,7 +240,8 @@ public class SdkHarnessClient {
 
 ListenableFuture registerResponseFuture =
 Futures.transform(
-genericResponse, InstructionResponse::getRegister,
+genericResponse,
+InstructionResponse::getRegister,
 MoreExecutors.directExecutor());
 for (Map.Entry>
 descriptorInputEntry : processBundleDescriptors.entrySet()) {

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Merge pull request #4576 Reduce the flakiness of the state sampler progress metrics.

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 537b9b78ace30304b419c859e67b6339fc16e92c
Merge: 6831f2c ef12700
Author: Robert Bradshaw 
AuthorDate: Fri Feb 2 08:20:50 2018 -0800

Merge pull request #4576 Reduce the flakiness of the state sampler progress 
metrics.

Reduce the flakiness of the state sampler progress metrics.

 sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (6831f2c -> 537b9b7)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6831f2c  Merge pull request #4584: [BEAM-3551] Define compiler 
-parameters flag in the default options
 add ef12700  Reduce the flakiness of the state sampler progress metrics.
 new 537b9b7  Merge pull request #4576 Reduce the flakiness of the state 
sampler progress metrics.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Assigned] (BEAM-3607) Move checkNotNull arg checks to a new checkArgumentNotNull

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3607:
-

Assignee: (was: Kenneth Knowles)

> Move checkNotNull arg checks to a new checkArgumentNotNull
> --
>
> Key: BEAM-3607
> URL: https://issues.apache.org/jira/browse/BEAM-3607
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>
> The simple fact is that {{checkNotNull}} throws NPE which to users looks like 
> a core dump sort of failure. It throws a "500 error" when we usually intend a 
> "400 error", so to speak, so the bugs get filed on the wrong components, or 
> users don't know they passed the wrong thing, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3608) Pre-shade Guava for things we want to keep using

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3608:
-

 Summary: Pre-shade Guava for things we want to keep using
 Key: BEAM-3608
 URL: https://issues.apache.org/jira/browse/BEAM-3608
 Project: Beam
  Issue Type: Sub-task
  Components: runner-core, sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Instead of shading as part of our build, we can shade before build so that it 
is apparent when reading code, and in IDEs, that a particular class resides in 
a hidden namespace.

{{import com.google.common.reflect.TypeToken}}

becomes something like

{{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}

So we can very trivially ban `org.apache.beam.private` from public APIs unless 
they are annotated {{@Internal}}, and it makes sharing between our own modules 
never get broken by shading again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3607) Move checkNotNull arg checks to a new checkArgumentNotNull

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3607:
--
Labels: beginner newbie starter  (was: )

> Move checkNotNull arg checks to a new checkArgumentNotNull
> --
>
> Key: BEAM-3607
> URL: https://issues.apache.org/jira/browse/BEAM-3607
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>
> The simple fact is that {{checkNotNull}} throws NPE which to users looks like 
> a core dump sort of failure. It throws a "500 error" when we usually intend a 
> "400 error", so to speak, so the bugs get filed on the wrong components, or 
> users don't know they passed the wrong thing, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3607) Move checkNotNull arg checks to a new checkArgumentNotNull

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3607:
-

 Summary: Move checkNotNull arg checks to a new checkArgumentNotNull
 Key: BEAM-3607
 URL: https://issues.apache.org/jira/browse/BEAM-3607
 Project: Beam
  Issue Type: Sub-task
  Components: runner-core, sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


The simple fact is that {{checkNotNull}} throws NPE which to users looks like a 
core dump sort of failure. It throws a "500 error" when we usually intend a 
"400 error", so to speak, so the bugs get filed on the wrong components, or 
users don't know they passed the wrong thing, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5846

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-3551] Define compiler -parameters flag in the default options

--
[...truncated 2.65 MB...]
2018-02-02T13:45:08.058 [INFO] Excluding 
commons-beanutils:commons-beanutils-core:jar:1.8.0 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.spark:spark-launcher_2.11:jar:2.2.1 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.spark:spark-network-common_2.11:jar:2.2.1 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.11:jar:2.2.1 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.11:jar:2.2.1 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.9.3 
from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding javax.activation:activation:jar:1.1.1 
from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding mx4j:mx4j:jar:3.0.2 from the shaded 
jar.
2018-02-02T13:45:08.058 [INFO] Excluding javax.mail:mail:jar:1.4.7 from the 
shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.bouncycastle:bcprov-jdk15on:jar:1.51 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
com.jamesmurty.utils:java-xmlbuilder:jar:1.0 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding net.iharder:base64:jar:2.3.8 from the 
shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.6.0 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.6.0 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
javax.servlet:javax.servlet-api:jar:3.1.0 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.16 from 
the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.16 
from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2018-02-02T13:45:08.058 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.16 
from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.scala-lang:scala-library:jar:2.11.8 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding 
org.json4s:json4s-jackson_2.11:jar:3.2.11 from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding org.json4s:json4s-core_2.11:jar:3.2.11 
from the shaded jar.
2018-02-02T13:45:08.058 [INFO] Excluding org.json4s:json4s-ast_2.11:jar:3.2.11 
from the shaded jar.
2018-02-02T13:45:08.058 [INFO] 

[jira] [Assigned] (BEAM-3186) In-flight data loss when restoring from savepoint

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3186:
--

Assignee: Dawid Wysakowicz  (was: Jean-Baptiste Onofré)

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350324#comment-16350324
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


Can we get an update on this Jira ? I would like to help. So please don't 
hesitate to ping me.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example];
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3186) In-flight data loss when restoring from savepoint

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3186:
--

Assignee: Jean-Baptiste Onofré  (was: Dawid Wysakowicz)

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3551) Add -parameters flag to javac (and test)

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3551.

Resolution: Fixed

> Add -parameters flag to javac (and test)
> 
>
> Key: BEAM-3551
> URL: https://issues.apache.org/jira/browse/BEAM-3551
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.3.0 updated: [BEAM-3551] Define compiler -parameters flag in the default options

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new 7da24e6  [BEAM-3551] Define compiler -parameters flag in the default 
options
7da24e6 is described below

commit 7da24e630ee954adb577b78d4f2b472e51db5742
Author: Jean-Baptiste Onofré 
AuthorDate: Fri Feb 2 10:32:07 2018 +0100

[BEAM-3551] Define compiler -parameters flag in the default options
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index 4e9000f..dbce3c8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1462,7 +1462,6 @@
   -Xlint:-try
   -Xlint:-unchecked
   -Xlint:-varargs
- -parameters
   
   
   
@@ -1482,6 +1481,7 @@
   
 
   ${compiler.default.pkginfo.flag}
+  -parameters
 
 
   ${compiler.default.exclude}

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] branch master updated (9cf86bc -> 6831f2c)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9cf86bc  Merge pull request #4301: Introduces the Wait transform
 add 39ab03b  [BEAM-3551] Define compiler -parameters flag in the default 
options
 new 6831f2c  Merge pull request #4584: [BEAM-3551] Define compiler 
-parameters flag in the default options

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #865

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 9cf86bcebcdbd8d5a84777cf2871597f0ba1b951 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9cf86bcebcdbd8d5a84777cf2871597f0ba1b951
Commit message: "Merge pull request #4301: Introduces the Wait transform"
 > git rev-list 9cf86bcebcdbd8d5a84777cf2871597f0ba1b951 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5356997287317944221.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6477910650308969277.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4785595407424347375.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5022164939614466206.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8896889881970279591.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8072121964405520941.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #93

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #109

[Apache Jenkins Server]

See 


--
[...truncated 8.35 KB...]
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6148647451500489092.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: hdfs3<0.4.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.4.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #92

[Apache Jenkins Server]

See 


--
[...truncated 10.01 KB...]
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins197553527595491943.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: hdfs3<0.4.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.4.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #95

[Apache Jenkins Server]

See 


--
[...truncated 11.96 KB...]
  Using cached typing-3.6.4-py2-none-any.whl
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
./.env/lib/python2.7/site-packages (from apache-beam==2.4.0.dev0)
Collecting hdfs3<0.4.0,>=0.3.0 (from apache-beam==2.4.0.dev0)
  Using cached hdfs3-0.3.0-py2.py3-none-any.whl
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.4.0.dev0)
  Using cached google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.4.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.4.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.4.0.dev0)
  Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.4.0.dev0)
  Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.4.0.dev0)
  Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio<2,>=1.0->apache-beam==2.4.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.4.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.7 in 
/usr/local/lib/python2.7/dist-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.4.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.4.0.dev0)
Collecting fasteners>=0.14 (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.4.0.dev0)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth-1.3.0-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting cachetools>=2.0.0 (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting future<0.17dev,>=0.16.0 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Collecting ply==3.8 (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/usr/local/lib/python2.7/dist-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement 

[jira] [Reopened] (BEAM-3551) Add -parameters flag to javac (and test)

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reopened BEAM-3551:


The {{-parameters}} flag was not set on the right place. I'm fixing that.

> Add -parameters flag to javac (and test)
> 
>
> Key: BEAM-3551
> URL: https://issues.apache.org/jira/browse/BEAM-3551
> Project: Beam
>  Issue Type: Wish
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2995) can't read/write hdfs in Flink CLUSTER(Standalone)

[Dawid Wysakowicz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Wysakowicz reassigned BEAM-2995:
--

Assignee: Dawid Wysakowicz  (was: Aljoscha Krettek)

> can't read/write hdfs in Flink CLUSTER(Standalone)
> --
>
> Key: BEAM-2995
> URL: https://issues.apache.org/jira/browse/BEAM-2995
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0
>Reporter: huangjianhuang
>Assignee: Dawid Wysakowicz
>Priority: Major
>
> i just write a simple demo like:
> {code:java}
> Configuration conf = new Configuration();
> conf.set("fs.default.name", "hdfs://localhost:9000");
> //other codes
> p.apply("ReadLines", 
> TextIO.read().from("hdfs://localhost:9000/tmp/words"))
> 
> .apply(TextIO.write().to("hdfs://localhost:9000/tmp/hdfsout"));
> {code}
> it works in flink local model with cmd:
> {code:java}
> mvn exec:java -Dexec.mainClass=com.joe.FlinkWithHDFS -Pflink-runner 
> -Dexec.args="--runner=FlinkRunner 
> --filesToStage=target/flinkBeam-2.2.0-SNAPSHOT-shaded.jar"
> {code}
> but not works in CLUSTER mode:
> {code:java}
> mvn exec:java -Dexec.mainClass=com.joe.FlinkWithHDFS -Pflink-runner 
> -Dexec.args="--runner=FlinkRunner 
> --filesToStage=target/flinkBeam-2.2.0-SNAPSHOT-shaded.jar 
> --flinkMaster=localhost:6123 "
> {code}
> it seems the flink cluster regard the hdfs as local file system. 
> The input log from flink-jobmanger.log is:
> {code:java}
> 2017-09-27 20:17:37,962 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Successfully ran initialization on master in 136 ms.
> 2017-09-27 20:17:37,968 INFO  org.apache.beam.sdk.io.FileBasedSource  
>   - {color:red}Filepattern hdfs://localhost:9000/tmp/words2 
> matched 0 files with total size 0{color}
> 2017-09-27 20:17:37,968 INFO  org.apache.beam.sdk.io.FileBasedSource  
>   - Splitting filepattern hdfs://localhost:9000/tmp/words2 into 
> bundles of size 0 took 0 ms and produced 0 files a
> nd 0 bundles
> {code}
> The output  error message is :
> {code:java}
> Caused by: java.lang.ClassCastException: 
> {color:red}org.apache.beam.sdk.io.hdfs.HadoopResourceId cannot be cast to 
> org.apache.beam.sdk.io.LocalResourceId{color}
> at 
> org.apache.beam.sdk.io.LocalFileSystem.create(LocalFileSystem.java:77)
> at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:256)
> at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:243)
> at 
> org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:922)
> at 
> org.apache.beam.sdk.io.FileBasedSink$Writer.openUnwindowed(FileBasedSink.java:884)
> at 
> org.apache.beam.sdk.io.WriteFiles.finalizeForDestinationFillEmptyShards(WriteFiles.java:909)
> at org.apache.beam.sdk.io.WriteFiles.access$900(WriteFiles.java:110)
> at 
> org.apache.beam.sdk.io.WriteFiles$2.processElement(WriteFiles.java:858)
> {code}
> can somebody help me, i've try all the way just can't work it out [cry]
> https://issues.apache.org/jira/browse/BEAM-2457



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)