Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5859

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #872

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 42ac62a067307173da061e7dbdbffd5e03672b96 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 42ac62a067307173da061e7dbdbffd5e03672b96
Commit message: "Merge pull request #4310:  [BEAM-3207] Create a standard 
location to enumerate and document URNs."
 > git rev-list 42ac62a067307173da061e7dbdbffd5e03672b96 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5880310312226461750.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins67790719460520932.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4446317790256788046.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7888241517095399516.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2441799989745222646.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5920163343609698122.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.re

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #99

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2721) Augment BeamRecordType to do slicing and concatenation.

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351651#comment-16351651
 ] 

Anton Kedin commented on BEAM-2721:
---

BeamRecordSqlType removal is merged in BEAM-3292 

 

 Further refactoring is out: [https://github.com/apache/beam/pull/4593]

Intorduces factories, collectors for constructing BeamRecords and 
BeamRecorTypes from java streams

> Augment BeamRecordType to do slicing and concatenation.
> ---
>
> Key: BEAM-2721
> URL: https://issues.apache.org/jira/browse/BEAM-2721
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Robert Bradshaw
>Assignee: Anton Kedin
>Priority: Major
>
> Currently in several places we cast to BeamSqlRecordType, extract the field 
> type ints, do the slicing, and then reconstruct a new BeamSqlRecordType. If 
> BeamRecordType had polymorphic methods to slice/concat this would be cleaner 
> and more flexible. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3292) Remove BeamRecordSqlType

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-3292.
-

[https://github.com/apache/beam/pull/4488] 

> Remove BeamRecordSqlType
> 
>
> Key: BEAM-3292
> URL: https://issues.apache.org/jira/browse/BEAM-3292
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> [BeamRecordType|https://github.com/apache/beam/blob/39e66e953b0f8e16435acb038cad364acf2b3a57/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecordType.java]
>  is implemented as 2 lists: the list of field names, and the list of the 
> coders for those fields. Both lists are ordered.
> [BeamRecordSqlType|https://github.com/apache/beam/blob/2eb7de0fe6e96da9805fc827294da1e1329ff716/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java]
>  additionally has a list of 
> [java.sql.Types|https://docs.oracle.com/javase/7/docs/api/java/sql/Types.html]
>  ints to define types of those fields. It is used to map between Java types, 
> Calcite types, and Beam Coders.
> This information is not used for anything except for that mapping, which in 
> turn is only used to create records and map back to Calcite types.
> But because of this indirect mapping we cannot rely on core BeamRecordType 
> and are forced to have BeamRecordSqlType. This introduces additional 
> complexity, when, for example, generating record types based on pojo classes.
> If we could find another mechanism to map Calcite types and java classes to 
> Beam Coders bypassing java.sql.Types then we can just use the core 
> BeamRecordType and remove the BeamRecordSqlType functionality.
> One approach is to have a predefined set of coders which are then used like 
> types, e.g.:
> {code:java}
> public static class SqlCoders {
>public Coder INTEGER = VarIntCoder.of();
>public Coder VARCHAR = StringUtf8COder.of();
>public Coder TIMESTAMP = DateCoder.of();
> }
> {code}
> Problem with that approach is establishing the coders identity. That is, when 
> a coder is serialized and then deserialized, it becomes a different instance, 
> so we need a mechanism to know the identity or maybe just equality of the 
> coders. If this is solved then replacing java.sql.Types with predefined SQL 
> coders like above becomes trivial.
> Few links on this:
>  - 
> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L56
> - 
> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslator.java#L34
>  - 
> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L391



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3292) Remove BeamRecordSqlType

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin resolved BEAM-3292.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Remove BeamRecordSqlType
> 
>
> Key: BEAM-3292
> URL: https://issues.apache.org/jira/browse/BEAM-3292
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> [BeamRecordType|https://github.com/apache/beam/blob/39e66e953b0f8e16435acb038cad364acf2b3a57/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecordType.java]
>  is implemented as 2 lists: the list of field names, and the list of the 
> coders for those fields. Both lists are ordered.
> [BeamRecordSqlType|https://github.com/apache/beam/blob/2eb7de0fe6e96da9805fc827294da1e1329ff716/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamRecordSqlType.java]
>  additionally has a list of 
> [java.sql.Types|https://docs.oracle.com/javase/7/docs/api/java/sql/Types.html]
>  ints to define types of those fields. It is used to map between Java types, 
> Calcite types, and Beam Coders.
> This information is not used for anything except for that mapping, which in 
> turn is only used to create records and map back to Calcite types.
> But because of this indirect mapping we cannot rely on core BeamRecordType 
> and are forced to have BeamRecordSqlType. This introduces additional 
> complexity, when, for example, generating record types based on pojo classes.
> If we could find another mechanism to map Calcite types and java classes to 
> Beam Coders bypassing java.sql.Types then we can just use the core 
> BeamRecordType and remove the BeamRecordSqlType functionality.
> One approach is to have a predefined set of coders which are then used like 
> types, e.g.:
> {code:java}
> public static class SqlCoders {
>public Coder INTEGER = VarIntCoder.of();
>public Coder VARCHAR = StringUtf8COder.of();
>public Coder TIMESTAMP = DateCoder.of();
> }
> {code}
> Problem with that approach is establishing the coders identity. That is, when 
> a coder is serialized and then deserialized, it becomes a different instance, 
> so we need a mechanism to know the identity or maybe just equality of the 
> coders. If this is solved then replacing java.sql.Types with predefined SQL 
> coders like above becomes trivial.
> Few links on this:
>  - 
> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L56
> - 
> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslator.java#L34
>  - 
> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L391



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #820

[Apache Jenkins Server]

See 


--
[...truncated 186.56 KB...]
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s28"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert:even/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5858

[Apache Jenkins Server]

See 


--
[...truncated 2.97 MB...]
2018-02-04T00:52:54.217 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.14 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.4.0-SNAPSHOT from the 
shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
org.apache.beam:beam-model-job-management:jar:2.4.0-SNAPSHOT from the shaded 
jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
org.apache.beam:beam-runners-core-java:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.6 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2018-02-04T00:52:54.217 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2018-02-04T00:52:56.092 [INFO] Replacing original artifact with shaded artifact.
2018-02-04T00:52:56.092 [INFO] Replacing 

 with 

2018-02-04T00:52:56.092 [INFO] Replacing original test artifact with shaded 
test artifact.
2018-02-04T00:52:56.092 [INFO] Replacing 

 with 

2018-02-04T00:52:56.300 [INFO] 
2018-02-04T00:52:56.300 [INFO] --- maven-assembly-plugin:3.1.0:single 
(source-release-assembly) @ beam-runners-apex ---
2018-02-04T00:52:56.303 [INFO] Skipping the assembly in this project because 
it's not the Execution Root
2018-02-04T00:52:56.410 [INFO] 
2018-02-04T00:52:56.410 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-runners-apex ---
2018-02-04T00:52:56.414 [INFO] Building jar: 

2018-02-04T00:52:56.532 [INFO] 
2018-02-04T00:52:56.532 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-runners-apex ---
2018-02-04T00:52:56.535 [INFO] Building jar: 

2018-02-04T00:52:56.651 [INFO] 
2018-02-04T00:52:56.651 [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar 
(attach-javadocs) @ beam-runners-apex ---
2018-02-04T00:52:59.856 [INFO] 
2 warnings
2018-02-04T00:52:59.856 [WARNING] Javadoc Warnings
2018-02-04T00:52:59.856 [WARNING] 
:33:
 warning: no description for @param
2018-02-04T00:52:59.856 [WARNING] * @param 
2018-02-04T00:52:59.856 [WARNING] ^
2018-02-04T00:52:59.856 [WARNING] 
:48:
 warning: no description for @param
2018-02-04T00:52:59.856 [WARNING] * @param 
2018-02-04T00:52:59.856 [WARNING] ^
2018-02-04T00:52:59.864 [INFO] Building jar: 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #98

[Apache Jenkins Server]

See 


Changes:

[robertwb] [BEAM-3207] Create a standard location to enumerate and document 
URNs.

[robertwb] Revert URNs that are currently hard-coded in the Dataflow worker.

[kedin] [SQL] Add SqlTypeCoder, replace java.sql.Types

--
[...truncated 14.34 KB...]
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 

 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1466929572031890294.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/98/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-04 00:00:34,941 d22f7b50 MainThread INFO Verbose logging to: 

2018-02-04 00:00:34,941 d22f7b50 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-04 00:00:34,942 d22f7b50 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/98/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.tfrecordioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-04 00:00:35,271 d22f7b50 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-04 00:00:35,271 d22f7b50 MainThread INFO Initializing the edw 
service decoder
2018-02-04 00:00:35,426 d22f7b50 MainThread beam_integration_benchmark(1/1) 
INFO Provisioning resources for benchmark beam_integration_benchmark
2018-02-04 00:00:35,430 d22f7b50 MainThr

Build failed in Jenkins: beam_PerformanceTests_Python #871

[Apache Jenkins Server]

See 


Changes:

[robertwb] [BEAM-3207] Create a standard location to enumerate and document 
URNs.

[robertwb] Revert URNs that are currently hard-coded in the Dataflow worker.

[kedin] [SQL] Add SqlTypeCoder, replace java.sql.Types

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 42ac62a067307173da061e7dbdbffd5e03672b96 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 42ac62a067307173da061e7dbdbffd5e03672b96
Commit message: "Merge pull request #4310:  [BEAM-3207] Create a standard 
location to enumerate and document URNs."
 > git rev-list 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6437454834343640197.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins41711574346754431.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5315939704187737314.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3831944567776251657.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/61/c8/3e04b8f555a598de6053bd6e07c6aa389e6801467599d166d17f984af6c0/setuptools-38.4.1-py2.py3-none-any.whl#md5=8f27eaab5b754ebf4fcf4d6daf78fe3e
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3340335583635539481.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6476966559704729581.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/

[beam] branch master updated (78271e0 -> 42ac62a)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 78271e0  Merge pull request #4488: [BEAM-3292][SQL] Replace 
java.sql.Types with SqlTypeCoder
 add 98d9060  [BEAM-3207] Create a standard location to enumerate and 
document URNs.
 add 04c399c  Revert URNs that are currently hard-coded in the Dataflow 
worker.
 new 42ac62a  Merge pull request #4310:  [BEAM-3207] Create a standard 
location to enumerate and document URNs.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .gitignore |   1 +
 .../resources/org/apache/beam/model/common_urns.md | 132 +
 pom.xml|   1 +
 .../core/construction/ModelCoderRegistrar.java |  18 +--
 .../core/construction/PTransformTranslation.java   |  26 ++--
 .../beam/runners/core/construction/UrnUtils.java   |  63 ++
 .../construction/WindowingStrategyTranslation.java |   7 ++
 .../runners/core/construction/UrnUtilsTest.java}   |  41 ---
 .../graph/LengthPrefixUnknownCoders.java   |  19 +--
 sdks/python/apache_beam/coders/coders.py   |  33 --
 sdks/python/apache_beam/io/iobase.py   |   8 +-
 sdks/python/apache_beam/pipeline.py|   4 +-
 .../execution.pxd => portability/python_urns.py}   |  23 ++--
 sdks/python/apache_beam/pvalue.py  |  20 ++--
 .../runners/portability/fn_api_runner.py   |  49 
 .../apache_beam/runners/worker/bundle_processor.py |  40 +++
 sdks/python/apache_beam/transforms/core.py |  56 +++--
 sdks/python/apache_beam/transforms/ptransform.py   |   6 +-
 sdks/python/apache_beam/transforms/window.py   |  22 ++--
 sdks/python/apache_beam/utils/urns.py  |  42 ---
 sdks/python/setup.py   |  30 +
 21 files changed, 421 insertions(+), 220 deletions(-)
 create mode 100644 
model/pipeline/src/main/resources/org/apache/beam/model/common_urns.md
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UrnUtils.java
 copy 
runners/{flink/src/test/java/org/apache/beam/runners/flink/FlinkRunnerRegistrarTest.java
 => 
core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UrnUtilsTest.java}
 (51%)
 copy sdks/python/apache_beam/{metrics/execution.pxd => 
portability/python_urns.py} (56%)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4310: [BEAM-3207] Create a standard location to enumerate and document URNs.

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 42ac62a067307173da061e7dbdbffd5e03672b96
Merge: 78271e0 04c399c
Author: Kenn Knowles 
AuthorDate: Sat Feb 3 14:43:45 2018 -0800

Merge pull request #4310:  [BEAM-3207] Create a standard location to 
enumerate and document URNs.

 .gitignore |   1 +
 .../resources/org/apache/beam/model/common_urns.md | 132 +
 pom.xml|   1 +
 .../core/construction/ModelCoderRegistrar.java |  18 +--
 .../core/construction/PTransformTranslation.java   |  26 ++--
 .../beam/runners/core/construction/UrnUtils.java   |  63 ++
 .../construction/WindowingStrategyTranslation.java |   7 ++
 .../runners/core/construction/UrnUtilsTest.java|  59 +
 .../graph/LengthPrefixUnknownCoders.java   |  19 +--
 sdks/python/apache_beam/coders/coders.py   |  33 --
 sdks/python/apache_beam/io/iobase.py   |   8 +-
 sdks/python/apache_beam/pipeline.py|   4 +-
 sdks/python/apache_beam/portability/python_urns.py |  30 +
 sdks/python/apache_beam/pvalue.py  |  20 ++--
 .../runners/portability/fn_api_runner.py   |  49 
 .../apache_beam/runners/worker/bundle_processor.py |  40 +++
 sdks/python/apache_beam/transforms/core.py |  56 +++--
 sdks/python/apache_beam/transforms/ptransform.py   |   6 +-
 sdks/python/apache_beam/transforms/window.py   |  22 ++--
 sdks/python/apache_beam/utils/urns.py  |  42 ---
 sdks/python/setup.py   |  30 +
 21 files changed, 473 insertions(+), 193 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #817

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #4488: [BEAM-3292][SQL] Replace java.sql.Types with SqlTypeCoder

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 78271e064609400a535d81482c9893f3a2bd1d37
Merge: 8130c20 a30030c
Author: Kenn Knowles 
AuthorDate: Sat Feb 3 13:05:15 2018 -0800

Merge pull request #4488: [BEAM-3292][SQL] Replace java.sql.Types with 
SqlTypeCoder

 .../org/apache/beam/sdk/values/BeamRecordType.java |  35 +++-
 .../beam/sdk/extensions/sql/BeamRecordSqlType.java | 178 +++-
 .../apache/beam/sdk/extensions/sql/BeamSql.java|   3 +-
 .../sdk/extensions/sql/BeamSqlRecordHelper.java|  45 ++
 .../beam/sdk/extensions/sql/BeamSqlTable.java  |   3 +-
 .../beam/sdk/extensions/sql/SqlTypeCoder.java  | 156 ++
 .../beam/sdk/extensions/sql/SqlTypeCoders.java |  67 
 .../sdk/extensions/sql/example/BeamSqlExample.java |  20 ++-
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |  10 +-
 .../extensions/sql/impl/parser/ParserUtils.java|   4 +-
 .../sql/impl/rel/BeamAggregationRel.java   |  21 +--
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  12 +-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |  12 +-
 .../extensions/sql/impl/schema/BaseBeamTable.java  |  12 +-
 .../sql/impl/schema/BeamPCollectionTable.java  |  10 +-
 .../extensions/sql/impl/schema/BeamTableUtils.java |  27 ++--
 .../impl/transform/BeamAggregationTransforms.java  |  45 +++---
 .../sql/impl/transform/BeamJoinTransforms.java |  42 +++--
 .../sql/impl/transform/BeamSqlProjectFn.java   |   8 +-
 .../impl/transform/agg/BigDecimalConverter.java|  31 ++--
 .../extensions/sql/impl/utils/CalciteUtils.java|  85 +-
 .../beam/sdk/extensions/sql/meta/Column.java   |   5 +-
 .../extensions/sql/meta/provider/MetaUtils.java|  18 +--
 .../sql/meta/provider/kafka/BeamKafkaCSVTable.java |  24 +--
 .../sql/meta/provider/kafka/BeamKafkaTable.java|  14 +-
 .../meta/provider/kafka/KafkaTableProvider.java|   6 +-
 .../sql/meta/provider/text/BeamTextCSVTable.java   |  16 +-
 .../provider/text/BeamTextCSVTableIOReader.java|  12 +-
 .../provider/text/BeamTextCSVTableIOWriter.java|  17 +-
 .../sql/meta/provider/text/BeamTextTable.java  |   6 +-
 .../sql/meta/provider/text/TextTableProvider.java  |   6 +-
 .../sdk/extensions/sql/BeamRecordSqlTypeTest.java  | 115 -
 .../extensions/sql/BeamSqlDslAggregationTest.java  | 179 -
 .../sql/BeamSqlDslAggregationVarianceTest.java |  13 +-
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java|  79 +
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java |  62 ---
 .../sdk/extensions/sql/BeamSqlDslProjectTest.java  |  34 ++--
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |  15 +-
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  43 ++---
 .../interpreter/BeamSqlFnExecutorTestBase.java |   4 +-
 .../sql/impl/parser/BeamSqlParserTest.java |   7 +-
 .../sql/impl/rel/BeamIntersectRelTest.java |  26 +--
 .../impl/rel/BeamJoinRelBoundedVsBoundedTest.java  |  62 +++
 .../rel/BeamJoinRelUnboundedVsBoundedTest.java |  57 ---
 .../rel/BeamJoinRelUnboundedVsUnboundedTest.java   |  42 ++---
 .../extensions/sql/impl/rel/BeamMinusRelTest.java  |  26 +--
 .../sql/impl/rel/BeamSetOperatorRelBaseTest.java   |  16 +-
 .../extensions/sql/impl/rel/BeamSortRelTest.java   |  78 -
 .../extensions/sql/impl/rel/BeamUnionRelTest.java  |  20 +--
 .../extensions/sql/impl/rel/BeamValuesRelTest.java |  22 +--
 .../sql/impl/schema/BeamSqlRowCoderTest.java   |   8 +-
 .../transform/BeamAggregationTransformTest.java|  14 +-
 .../schema/transform/BeamTransformBaseTest.java|  10 +-
 .../transform/agg/BigDecimalConverterTest.java |  19 +--
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java |  41 ++---
 .../BeamSqlComparisonOperatorsIntegrationTest.java |  56 ---
 .../meta/provider/kafka/BeamKafkaCSVTableTest.java |   4 +-
 .../provider/kafka/KafkaTableProviderTest.java |   7 +-
 .../meta/provider/text/BeamTextCSVTableTest.java   |  14 +-
 .../meta/provider/text/TextTableProviderTest.java  |   7 +-
 .../sql/meta/store/InMemoryMetaStoreTest.java  |  12 +-
 .../extensions/sql/mock/MockedBoundedTable.java|   8 +-
 .../beam/sdk/extensions/sql/mock/MockedTable.java  |   6 +-
 .../extensions/sql/mock/MockedUnboundedTable.java  |   8 +-
 .../beam/sdk/nexmark/model/sql/BeamRecordSize.java |  39 ++---
 .../model/sql/adapter/ModelFieldsAdapter.java  |   5 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java|  17 +-
 .../sdk/nexmark/model/sql/BeamRecordSizeTest.java  |   3 +-
 .../sql/adapter/ModelAdaptersMappingTest.java  |  22 ++-
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java |  15 +-
 71 files changed, 1100 insertions(+), 1069 deletions(-)

-- 
To stop receiving notification emails like this one, please 

[beam] branch master updated (8130c20 -> 78271e0)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8130c20  Merge pull request #4578: [BEAM-3603] Adds a ReadAll 
transform to tfrecordio
 add a30030c  [SQL] Add SqlTypeCoder, replace java.sql.Types
 new 78271e0  Merge pull request #4488: [BEAM-3292][SQL] Replace 
java.sql.Types with SqlTypeCoder

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/sdk/values/BeamRecordType.java |  35 +++-
 .../beam/sdk/extensions/sql/BeamRecordSqlType.java | 178 +++-
 .../apache/beam/sdk/extensions/sql/BeamSql.java|   3 +-
 .../sdk/extensions/sql/BeamSqlRecordHelper.java|  45 ++
 .../beam/sdk/extensions/sql/BeamSqlTable.java  |   3 +-
 .../beam/sdk/extensions/sql/SqlTypeCoder.java  | 156 ++
 .../beam/sdk/extensions/sql/SqlTypeCoders.java |  67 
 .../sdk/extensions/sql/example/BeamSqlExample.java |  20 ++-
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |  10 +-
 .../extensions/sql/impl/parser/ParserUtils.java|   4 +-
 .../sql/impl/rel/BeamAggregationRel.java   |  21 +--
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  12 +-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |  12 +-
 .../extensions/sql/impl/schema/BaseBeamTable.java  |  12 +-
 .../sql/impl/schema/BeamPCollectionTable.java  |  10 +-
 .../extensions/sql/impl/schema/BeamTableUtils.java |  27 ++--
 .../impl/transform/BeamAggregationTransforms.java  |  45 +++---
 .../sql/impl/transform/BeamJoinTransforms.java |  42 +++--
 .../sql/impl/transform/BeamSqlProjectFn.java   |   8 +-
 .../impl/transform/agg/BigDecimalConverter.java|  31 ++--
 .../extensions/sql/impl/utils/CalciteUtils.java|  85 +-
 .../beam/sdk/extensions/sql/meta/Column.java   |   5 +-
 .../extensions/sql/meta/provider/MetaUtils.java|  18 +--
 .../sql/meta/provider/kafka/BeamKafkaCSVTable.java |  24 +--
 .../sql/meta/provider/kafka/BeamKafkaTable.java|  14 +-
 .../meta/provider/kafka/KafkaTableProvider.java|   6 +-
 .../sql/meta/provider/text/BeamTextCSVTable.java   |  16 +-
 .../provider/text/BeamTextCSVTableIOReader.java|  12 +-
 .../provider/text/BeamTextCSVTableIOWriter.java|  17 +-
 .../sql/meta/provider/text/BeamTextTable.java  |   6 +-
 .../sql/meta/provider/text/TextTableProvider.java  |   6 +-
 .../sdk/extensions/sql/BeamRecordSqlTypeTest.java  | 115 -
 .../extensions/sql/BeamSqlDslAggregationTest.java  | 179 -
 .../sql/BeamSqlDslAggregationVarianceTest.java |  13 +-
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java|  79 +
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java |  62 ---
 .../sdk/extensions/sql/BeamSqlDslProjectTest.java  |  34 ++--
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |  15 +-
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  43 ++---
 .../interpreter/BeamSqlFnExecutorTestBase.java |   4 +-
 .../sql/impl/parser/BeamSqlParserTest.java |   7 +-
 .../sql/impl/rel/BeamIntersectRelTest.java |  26 +--
 .../impl/rel/BeamJoinRelBoundedVsBoundedTest.java  |  62 +++
 .../rel/BeamJoinRelUnboundedVsBoundedTest.java |  57 ---
 .../rel/BeamJoinRelUnboundedVsUnboundedTest.java   |  42 ++---
 .../extensions/sql/impl/rel/BeamMinusRelTest.java  |  26 +--
 .../sql/impl/rel/BeamSetOperatorRelBaseTest.java   |  16 +-
 .../extensions/sql/impl/rel/BeamSortRelTest.java   |  78 -
 .../extensions/sql/impl/rel/BeamUnionRelTest.java  |  20 +--
 .../extensions/sql/impl/rel/BeamValuesRelTest.java |  22 +--
 .../sql/impl/schema/BeamSqlRowCoderTest.java   |   8 +-
 .../transform/BeamAggregationTransformTest.java|  14 +-
 .../schema/transform/BeamTransformBaseTest.java|  10 +-
 .../transform/agg/BigDecimalConverterTest.java |  19 +--
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java |  41 ++---
 .../BeamSqlComparisonOperatorsIntegrationTest.java |  56 ---
 .../meta/provider/kafka/BeamKafkaCSVTableTest.java |   4 +-
 .../provider/kafka/KafkaTableProviderTest.java |   7 +-
 .../meta/provider/text/BeamTextCSVTableTest.java   |  14 +-
 .../meta/provider/text/TextTableProviderTest.java  |   7 +-
 .../sql/meta/store/InMemoryMetaStoreTest.java  |  12 +-
 .../extensions/sql/mock/MockedBoundedTable.java|   8 +-
 .../beam/sdk/extensions/sql/mock/MockedTable.java  |   6 +-
 .../extensions/sql/mock/MockedUnboundedTable.java  |   8 +-
 .../beam/sdk/nexmark/model/sql/BeamRecordSize.java |  39 ++---
 .../model/sql/adapter/ModelFieldsAdapter.java  |   5 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java|  17 +-
 .../sdk/nexm

[jira] [Commented] (BEAM-3615) Dynamic/Default Coder For Data

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351519#comment-16351519
 ] 

Eugene Kirpichov commented on BEAM-3615:


Thanks for the report. Reading Avro files whose schema is unknown at 
construction time is already possible: you need to use 
AvroIO.parseGenericRecords().

> Dynamic/Default Coder For Data
> --
>
> Key: BEAM-3615
> URL: https://issues.apache.org/jira/browse/BEAM-3615
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql, sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Xu Mingmin
>Priority: Critical
>  Labels: newbie, performance
> Fix For: 2.1.0
>
>
> We need to Define Coder Statically in Every Job While Using Beam SQL Like in 
> Previous Version 2.1.0 We Were Reading Avro Coder Files Statically But in 
> 2.2.0 Now We can Read Dynamically Like that only Feature Needed to Introduced 
> So We can Define Dynamically at Run Time Reading From GCS or Table(Table 
> Schema) The Coder So Transformation can Be Performed. Which Makes Dataflow 
> Template More Efficiend For Single Type of Job We can Use it Multiple Times 
> Now We Need to Make Multiple Templates For Different Jobs.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3615) Dynamic/Default Coder For Data

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-3615.
--
   Resolution: Not A Problem
 Assignee: Eugene Kirpichov  (was: Xu Mingmin)
Fix Version/s: 2.1.0

I think it was available starting with Beam 2.1.0. Please reopen if I 
misunderstood what you're asking or if it doesn't solve your problem.

> Dynamic/Default Coder For Data
> --
>
> Key: BEAM-3615
> URL: https://issues.apache.org/jira/browse/BEAM-3615
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql, sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Eugene Kirpichov
>Priority: Critical
>  Labels: newbie, performance
> Fix For: 2.1.0
>
>
> We need to Define Coder Statically in Every Job While Using Beam SQL Like in 
> Previous Version 2.1.0 We Were Reading Avro Coder Files Statically But in 
> 2.2.0 Now We can Read Dynamically Like that only Feature Needed to Introduced 
> So We can Define Dynamically at Run Time Reading From GCS or Table(Table 
> Schema) The Coder So Transformation can Be Performed. Which Makes Dataflow 
> Template More Efficiend For Single Type of Job We can Use it Multiple Times 
> Now We Need to Make Multiple Templates For Different Jobs.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #870

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6
Commit message: "Merge pull request #4578: [BEAM-3603] Adds a ReadAll transform 
to tfrecordio"
 > git rev-list 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8771076743768557403.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3946484048348957082.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2901512981262046497.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins943096339721720063.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/61/c8/3e04b8f555a598de6053bd6e07c6aa389e6801467599d166d17f984af6c0/setuptools-38.4.1-py2.py3-none-any.whl#md5=8f27eaab5b754ebf4fcf4d6daf78fe3e
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6974599252394770215.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3727627421156537624.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #816

[Apache Jenkins Server]

See 


--
[...truncated 96.69 KB...]
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.4.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_default_value_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.4.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_empty_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_multi_valued_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok

==
ERROR: test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest)
--
Traceback (most recent call last):
  File 
"
 line 270, in test_multiple_empty_outputs
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 357, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 370, in run
return self.runner.run_pipeline(self)
  File 
"
 line 322, in run_pipeline
self.dataflow_client.create_job(self.job), self)
  File 
"
 line 175, in wrapper
return fun(*args, **kwargs)
  File 
"
 line 461, in create_job
self.create_job_description(job)
  File 
"
 line 491, in create_job_description
job.options, file_copy=self._gcs_file_copy)
  File 
"
 line 328, in stage_job_resources
setup_options.requirements_file, requirements_cache_path)
  File 
"

Build failed in Jenkins: beam_PerformanceTests_Python #869

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6
Commit message: "Merge pull request #4578: [BEAM-3603] Adds a ReadAll transform 
to tfrecordio"
 > git rev-list 8130c20dec5cfbda120ad0b6b8025526fa5ed6d6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2073589549577809106.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2786882898192093351.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8340508999685444342.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7902913812140214287.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6467281769186736362.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3543063612501090115.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/la

[jira] [Updated] (BEAM-3615) Dynamic/Default Coder For Data

[Kishan Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kishan Kumar updated BEAM-3615:
---
Description: 
We need to Define Coder Statically in Every Job While Using Beam SQL Like in 
Previous Version 2.1.0 We Were Reading Avro Coder Files Statically But in 2.2.0 
Now We can Read Dynamically Like that only Feature Needed to Introduced So We 
can Define Dynamically at Run Time Reading From GCS or Table(Table Schema) The 
Coder So Transformation can Be Performed. Which Makes Dataflow Template More 
Efficiend For Single Type of Job We can Use it Multiple Times Now We Need to 
Make Multiple Templates For Different Jobs.

 

 

  was:
We need to Define Coder Statically in Every Job While Using Beam SQL Like in 
Previous Version 2.1.0 We Were Reading Avro Coder Files Statically Coder Needed 
to Be Defined Dynamically at Run Time Reading From GCS or Table So 
Transformation can Be Performed.

 

 


> Dynamic/Default Coder For Data
> --
>
> Key: BEAM-3615
> URL: https://issues.apache.org/jira/browse/BEAM-3615
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql, sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Xu Mingmin
>Priority: Critical
>  Labels: newbie, performance
>
> We need to Define Coder Statically in Every Job While Using Beam SQL Like in 
> Previous Version 2.1.0 We Were Reading Avro Coder Files Statically But in 
> 2.2.0 Now We can Read Dynamically Like that only Feature Needed to Introduced 
> So We can Define Dynamically at Run Time Reading From GCS or Table(Table 
> Schema) The Coder So Transformation can Be Performed. Which Makes Dataflow 
> Template More Efficiend For Single Type of Job We can Use it Multiple Times 
> Now We Need to Make Multiple Templates For Different Jobs.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3615) Dynamic/Default Coder For Data

[Kishan Kumar (JIRA)]

Kishan Kumar created BEAM-3615:
--

 Summary: Dynamic/Default Coder For Data
 Key: BEAM-3615
 URL: https://issues.apache.org/jira/browse/BEAM-3615
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql, sdk-java-extensions
Affects Versions: 2.2.0
Reporter: Kishan Kumar
Assignee: Xu Mingmin


We need to Define Coder Statically in Every Job While Using Beam SQL Like in 
Previous Version 2.1.0 We Were Reading Avro Coder Files Statically Coder Needed 
to Be Defined Dynamically at Run Time Reading From GCS or Table So 
Transformation can Be Performed.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)