[beam] tag nightly-master updated (d4851d7 -> 6aac541)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag nightly-master was modified! *** from d4851d7 (commit) to 6aac541 (commit) from d4851d7 [BEAM-12145][BEAM-12155] Fixes MetricsTest failure (#14508) add f805f1c Merge pull request #14499 from [BEAM-11408, BEAM-11772] Add explicit output typehints to ensure coder determinism for BQ with auto-sharding add 28eec3f [BEAM-11712] Make up-to-date build file and codestyle add a407d79 [BEAM-11712] Add options for input/output paths, make it run via SparkRunner add 8fe0c5c [BEAM-11712] Fix static analysis warnings and typos on TPC-DS module add b3ef203 Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner add 22ccbe6 Fix W201 Trailing whitespace add 8e6a798 Merge pull request #14522: Fix W201 Trailing whitespace add d739669 Allow manual trigger of wheel build add 772cde0 Merge pull request #14514: Allow manual trigger of wheel build add 9d9fea7 [BEAM-12069] Move test_utils.py mock import inside function (#14511) add 822d16a Remove redundant cast add eb057a4 Merge pull request #14523: Remove redundant cast add 67badf6 [BEAM-11945] Add debug capture to SDK harness (#14197) add f2cc926 Eliminate beam_fn_api from KafkaIO expansion add 3b85447 Merge pull request #14419 from [BEAM-12114] Eliminate beam_fn_api from KafkaIO expansion add c9f1fee lint add e094603 Merge pull request #14525 from emilymye/pylint - Fix Python lint add fcf0f50 [BEAM-12011] Eliminate WindowFn.getOutputTime method add 6aac541 Merge pull request #14268: [BEAM-12011] Eliminate WindowFn.getOutputTime method No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml |2 +- .../apache/beam/runners/core/WatermarkHold.java|5 +- .../functions/FlinkNonMergingReduceFunction.java |5 +- .../functions/HashingFlinkCombineRunner.java | 10 +- .../functions/SingleWindowFlinkCombineRunner.java | 11 +- .../functions/SortingFlinkCombineRunner.java | 15 +- runners/google-cloud-dataflow-java/build.gradle|1 + .../beam/runners/dataflow/DataflowRunner.java |4 + .../util/BatchGroupAlsoByWindowAndCombineFn.java |6 +- .../util/BatchGroupAlsoByWindowViaIteratorsFn.java |6 +- .../worker/StreamingGroupAlsoByWindowFnsTest.java | 14 +- .../worker/util/GroupAlsoByWindowProperties.java | 12 +- .../environment/EmbeddedEnvironmentFactory.java|1 + .../fnexecution/control/RemoteExecutionTest.java |1 + .../runners/portability/ExternalWorkerService.java |3 +- .../SparkStructuredStreamingRunner.java|3 + .../translation/batch/AggregatorCombiner.java |6 +- runners/spark/spark_runner.gradle |1 + .../org/apache/beam/runners/spark/SparkRunner.java |3 + .../beam/runners/spark/SparkRunnerDebugger.java|3 + .../GroupNonMergingWindowsFunctions.java |8 +- .../runners/spark/translation/SparkCombineFn.java | 10 +- .../apache/beam/sdk/testing/WindowFnTestUtils.java | 93 -- .../sdk/transforms/windowing/GlobalWindows.java|6 - .../transforms/windowing/PartitioningWindowFn.java |5 - .../sdk/transforms/windowing/SlidingWindows.java | 17 - .../beam/sdk/transforms/windowing/WindowFn.java| 23 - .../org/apache/beam/sdk/util/IdentityWindowFn.java |6 - .../org/apache/beam/sdk/metrics/MetricsTest.java |2 +- .../sdk/transforms/windowing/FixedWindowsTest.java |8 - .../sdk/transforms/windowing/SessionsTest.java |9 - .../transforms/windowing/SlidingWindowsTest.java | 17 - .../sdk/transforms/windowing/WindowingTest.java|6 +- .../java/org/apache/beam/fn/harness/FnHarness.java | 32 +- .../fn/harness/control/ProcessBundleHandler.java | 12 +- .../beam/fn/harness/status/BeamFnStatusClient.java | 230 .../beam/fn/harness/status}/MemoryMonitor.java | 84 +- .../beam/fn/harness/status}/package-info.java |4 +- .../harness/control/ProcessBundleHandlerTest.java |2 +- .../fn/harness/status/BeamFnStatusClientTest.java | 142 +++ .../beam/fn/harness/status}/MemoryMonitorTest.java |6 +- .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java |5 - sdks/java/io/kafka/build.gradle|1 + .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 176 ++- .../beam/sdk/io/kafka/KafkaIOExternalTest.java |4 +- .../beam/sdk/nexmark/queries/WinningBids.java | 23 - sdks/java/testing/tpcds/README.md | 68 + sdks/java/testing/tpcds/build.gradle | 108 +- .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 327 +++-- .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 50
[beam] 01/01: Merge pull request #14268: [BEAM-12011] Eliminate WindowFn.getOutputTime method
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 6aac541d8f712a692349a642d72680144a6bb420 Merge: e094603 fcf0f50 Author: Kenn Knowles AuthorDate: Tue Apr 13 17:10:59 2021 -0700 Merge pull request #14268: [BEAM-12011] Eliminate WindowFn.getOutputTime method .../apache/beam/runners/core/WatermarkHold.java| 5 +- .../functions/FlinkNonMergingReduceFunction.java | 5 +- .../functions/HashingFlinkCombineRunner.java | 10 +-- .../functions/SingleWindowFlinkCombineRunner.java | 11 +-- .../functions/SortingFlinkCombineRunner.java | 15 +--- .../util/BatchGroupAlsoByWindowAndCombineFn.java | 6 +- .../util/BatchGroupAlsoByWindowViaIteratorsFn.java | 6 +- .../worker/StreamingGroupAlsoByWindowFnsTest.java | 14 +--- .../worker/util/GroupAlsoByWindowProperties.java | 12 +-- .../translation/batch/AggregatorCombiner.java | 6 +- .../GroupNonMergingWindowsFunctions.java | 8 +- .../runners/spark/translation/SparkCombineFn.java | 10 +-- .../apache/beam/sdk/testing/WindowFnTestUtils.java | 93 -- .../sdk/transforms/windowing/GlobalWindows.java| 6 -- .../transforms/windowing/PartitioningWindowFn.java | 5 -- .../sdk/transforms/windowing/SlidingWindows.java | 17 .../beam/sdk/transforms/windowing/WindowFn.java| 23 -- .../org/apache/beam/sdk/util/IdentityWindowFn.java | 6 -- .../sdk/transforms/windowing/FixedWindowsTest.java | 8 -- .../sdk/transforms/windowing/SessionsTest.java | 9 --- .../transforms/windowing/SlidingWindowsTest.java | 17 .../sdk/transforms/windowing/WindowingTest.java| 6 +- .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 5 -- .../beam/sdk/nexmark/queries/WinningBids.java | 23 -- 24 files changed, 26 insertions(+), 300 deletions(-)
[beam] branch master updated (e094603 -> 6aac541)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e094603 Merge pull request #14525 from emilymye/pylint - Fix Python lint add fcf0f50 [BEAM-12011] Eliminate WindowFn.getOutputTime method new 6aac541 Merge pull request #14268: [BEAM-12011] Eliminate WindowFn.getOutputTime method The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/runners/core/WatermarkHold.java| 5 +- .../functions/FlinkNonMergingReduceFunction.java | 5 +- .../functions/HashingFlinkCombineRunner.java | 10 +-- .../functions/SingleWindowFlinkCombineRunner.java | 11 +-- .../functions/SortingFlinkCombineRunner.java | 15 +--- .../util/BatchGroupAlsoByWindowAndCombineFn.java | 6 +- .../util/BatchGroupAlsoByWindowViaIteratorsFn.java | 6 +- .../worker/StreamingGroupAlsoByWindowFnsTest.java | 14 +--- .../worker/util/GroupAlsoByWindowProperties.java | 12 +-- .../translation/batch/AggregatorCombiner.java | 6 +- .../GroupNonMergingWindowsFunctions.java | 8 +- .../runners/spark/translation/SparkCombineFn.java | 10 +-- .../apache/beam/sdk/testing/WindowFnTestUtils.java | 93 -- .../sdk/transforms/windowing/GlobalWindows.java| 6 -- .../transforms/windowing/PartitioningWindowFn.java | 5 -- .../sdk/transforms/windowing/SlidingWindows.java | 17 .../beam/sdk/transforms/windowing/WindowFn.java| 23 -- .../org/apache/beam/sdk/util/IdentityWindowFn.java | 6 -- .../sdk/transforms/windowing/FixedWindowsTest.java | 8 -- .../sdk/transforms/windowing/SessionsTest.java | 9 --- .../transforms/windowing/SlidingWindowsTest.java | 17 .../sdk/transforms/windowing/WindowingTest.java| 6 +- .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 5 -- .../beam/sdk/nexmark/queries/WinningBids.java | 23 -- 24 files changed, 26 insertions(+), 300 deletions(-)
[beam] branch master updated: lint
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new c9f1fee lint new e094603 Merge pull request #14525 from emilymye/pylint - Fix Python lint c9f1fee is described below commit c9f1fee8984e368209cfe4a247b66594b5eb3571 Author: Emily Ye AuthorDate: Tue Apr 13 11:10:43 2021 -0700 lint --- sdks/python/apache_beam/io/gcp/bigquery_tools.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sdks/python/apache_beam/io/gcp/bigquery_tools.py b/sdks/python/apache_beam/io/gcp/bigquery_tools.py index 8551e8f..059414c 100644 --- a/sdks/python/apache_beam/io/gcp/bigquery_tools.py +++ b/sdks/python/apache_beam/io/gcp/bigquery_tools.py @@ -72,13 +72,12 @@ except ImportError: pass # pylint: enable=wrong-import-order, wrong-import-position -# pylint: disable=wrong-import-order, wrong-import-position +# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports try: from apache_beam.io.gcp.internal.clients.bigquery import TableReference except ImportError: TableReference = None - -# pylint: enable=wrong-import-order, wrong-import-position +# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports _LOGGER = logging.getLogger(__name__)
[beam] branch master updated: Eliminate beam_fn_api from KafkaIO expansion
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f2cc926 Eliminate beam_fn_api from KafkaIO expansion
new 3b85447 Merge pull request #14419 from [BEAM-12114] Eliminate
beam_fn_api from KafkaIO expansion
f2cc926 is described below
commit f2cc92663ad8ae685183e076cdb652d8fc3ba4e0
Author: Boyuan Zhang
AuthorDate: Fri Apr 2 15:20:53 2021 -0700
Eliminate beam_fn_api from KafkaIO expansion
---
runners/google-cloud-dataflow-java/build.gradle| 1 +
.../beam/runners/dataflow/DataflowRunner.java | 4 +
.../SparkStructuredStreamingRunner.java| 3 +
runners/spark/spark_runner.gradle | 1 +
.../org/apache/beam/runners/spark/SparkRunner.java | 3 +
.../beam/runners/spark/SparkRunnerDebugger.java| 3 +
sdks/java/io/kafka/build.gradle| 1 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 176 +++--
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 4 +-
9 files changed, 147 insertions(+), 49 deletions(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index 290ea94..476e8c5 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -72,6 +72,7 @@ dependencies {
compile project(path: ":model:pipeline", configuration: "shadow")
compile project(path: ":sdks:java:core", configuration: "shadow")
compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:io:kafka")
compile project(":sdks:java:io:google-cloud-platform")
compile project(":runners:core-construction-java")
compile library.java.avro
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index c81631e..0a79cd9 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -115,6 +115,7 @@ import
org.apache.beam.sdk.io.gcp.pubsub.PubsubMessageWithAttributesAndMessageId
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessageWithAttributesCoder;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource;
+import org.apache.beam.sdk.io.kafka.KafkaIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsValidator;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
@@ -491,6 +492,9 @@ public class DataflowRunner extends
PipelineRunner {
new StreamingPubsubIOWriteOverrideFactory(this)));
}
}
+ if (useUnifiedWorker(options)) {
+overridesBuilder.add(KafkaIO.Read.KAFKA_READ_OVERRIDE);
+ }
overridesBuilder.add(
PTransformOverride.of(
PTransformMatchers.writeWithRunnerDeterminedSharding(),
diff --git
a/runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java
b/runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java
index f08c36b..5d8230e 100644
---
a/runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java
+++
b/runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java
@@ -37,12 +37,14 @@ import
org.apache.beam.runners.spark.structuredstreaming.translation.batch.Pipel
import
org.apache.beam.runners.spark.structuredstreaming.translation.streaming.PipelineTranslatorStreaming;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.io.kafka.KafkaIO;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
import org.apache.beam.sdk.metrics.MetricsOptions;
import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.spark.SparkEnv$;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.metrics.MetricsSystem;
@@ -193,6 +195,7 @@ public final class SparkStructuredStreamingRunner
|| ExperimentalOptions.hasExperiment(
pipeline.getOptions(), "beam_fn_api_use_deprecated_read")
|| ExperimentalOptions.hasExperiment(pipeline.getOption
[beam] 01/01: Pin Python unit tests to ubuntu 18.04
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch test-py-1804
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 79c33556c31c50acc62c038a35238dfec6cf3f72
Author: Brian Hulette
AuthorDate: Tue Apr 13 13:28:40 2021 -0700
Pin Python unit tests to ubuntu 18.04
---
.github/workflows/python_tests.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/python_tests.yml
b/.github/workflows/python_tests.yml
index 7278b3a..9cabd48 100644
--- a/.github/workflows/python_tests.yml
+++ b/.github/workflows/python_tests.yml
@@ -95,7 +95,7 @@ jobs:
strategy:
fail-fast: false
matrix:
-os: [ubuntu-latest, macos-latest, windows-latest]
+os: [ubuntu-18.04, macos-latest, windows-latest]
params: [
{"py_ver": "3.6", "tox_env": "py36"},
{"py_ver": "3.7", "tox_env": "py37"},
[beam] branch test-py-1804 created (now 79c3355)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch test-py-1804 in repository https://gitbox.apache.org/repos/asf/beam.git. at 79c3355 Pin Python unit tests to ubuntu 18.04 This branch includes the following new commits: new 79c3355 Pin Python unit tests to ubuntu 18.04 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated: [BEAM-11945] Add debug capture to SDK harness (#14197)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 67badf6 [BEAM-11945] Add debug capture to SDK harness (#14197)
67badf6 is described below
commit 67badf644eddacbbda1d4c2131c22dda1ee29d30
Author: kileys
AuthorDate: Tue Apr 13 12:42:40 2021 -0700
[BEAM-11945] Add debug capture to SDK harness (#14197)
---
.../environment/EmbeddedEnvironmentFactory.java| 1 +
.../fnexecution/control/RemoteExecutionTest.java | 1 +
.../runners/portability/ExternalWorkerService.java | 3 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 32 +-
.../fn/harness/control/ProcessBundleHandler.java | 12 +-
.../beam/fn/harness/status/BeamFnStatusClient.java | 230
.../beam/fn/harness/status/MemoryMonitor.java | 635 +
.../beam/fn/harness/status/package-info.java | 20 +
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../fn/harness/status/BeamFnStatusClientTest.java | 142 +
.../beam/fn/harness/status/MemoryMonitorTest.java | 166 ++
11 files changed, 1238 insertions(+), 6 deletions(-)
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
index 73e14f6..0b617e9 100644
---
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
+++
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
@@ -102,6 +102,7 @@ public class EmbeddedEnvironmentFactory implements
EnvironmentFactory {
options,
loggingServer.getApiServiceDescriptor(),
controlServer.getApiServiceDescriptor(),
+null,
InProcessManagedChannelFactory.create(),
OutboundObserverFactory.clientDirect());
} catch (NoClassDefFoundError e) {
diff --git
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index 43e6c16..d37c41a 100644
---
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@ -214,6 +214,7 @@ public class RemoteExecutionTest implements Serializable {
PipelineOptionsFactory.create(),
loggingServer.getApiServiceDescriptor(),
controlServer.getApiServiceDescriptor(),
+null,
InProcessManagedChannelFactory.create(),
OutboundObserverFactory.clientDirect());
} catch (Exception e) {
diff --git
a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java
b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java
index c4311a3..bbb7760 100644
---
a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java
+++
b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java
@@ -59,7 +59,8 @@ public class ExternalWorkerService extends
BeamFnExternalWorkerPoolImplBase impl
request.getWorkerId(),
options,
request.getLoggingEndpoint(),
-request.getControlEndpoint());
+request.getControlEndpoint(),
+null);
LOG.info("Successfully started worker {}.",
request.getWorkerId());
} catch (Exception exn) {
LOG.error(String.format("Failed to start worker %s.",
request.getWorkerId()), exn);
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
index 3a7b741..34ab171 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
@@ -22,6 +22,7 @@ import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
+import javax.annotation.Nullable;
import org.apache.beam.fn.harness.control.AddHarnessIdInterceptor;
import org.apache.beam.fn.harness.control.BeamFnControlClient;
import org.apache.beam.fn.harness.control.FinalizeBundleHa
svn commit: r47040 - /dev/beam/2.29.0/python/
Author: kenn Date: Tue Apr 13 18:36:11 2021 New Revision: 47040 Log: Staging Python artifacts for Apache Beam 2.29.0 RC1 Added: dev/beam/2.29.0/python/ dev/beam/2.29.0/python/apache-beam-2.29.0.zip (with props) dev/beam/2.29.0/python/apache-beam-2.29.0.zip.asc dev/beam/2.29.0/python/apache-beam-2.29.0.zip.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_x86_64.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_x86_64.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_x86_64.whl.sha512 dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux2010_i686.whl (with props) dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux2010_i686.whl.asc dev/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux2010_i686.whl.sha512
[beam] branch master updated (9d9fea7 -> eb057a4)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9d9fea7 [BEAM-12069] Move test_utils.py mock import inside function (#14511) add 822d16a Remove redundant cast add eb057a4 Merge pull request #14523: Remove redundant cast No new revisions were added by this update. Summary of changes: .../core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (772cde0 -> 9d9fea7)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 772cde0 Merge pull request #14514: Allow manual trigger of wheel build add 9d9fea7 [BEAM-12069] Move test_utils.py mock import inside function (#14511) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/testing/test_utils.py | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-)
[beam] branch master updated (8e6a798 -> 772cde0)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8e6a798 Merge pull request #14522: Fix W201 Trailing whitespace add d739669 Allow manual trigger of wheel build add 772cde0 Merge pull request #14514: Allow manual trigger of wheel build No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (b3ef203 -> 8e6a798)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b3ef203 Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner add 22ccbe6 Fix W201 Trailing whitespace add 8e6a798 Merge pull request #14522: Fix W201 Trailing whitespace No new revisions were added by this update. Summary of changes: sdks/java/testing/tpcds/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] 03/04: [BEAM-11712] Fix static analysis warnings and typos on TPC-DS module
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8fe0c5cee1bab62680ebd92a51aed8f3da80e190
Author: Ismaël Mejía
AuthorDate: Tue Apr 13 14:20:54 2021 +0200
[BEAM-11712] Fix static analysis warnings and typos on TPC-DS module
---
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 3 +-
.../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 4 +-
.../org/apache/beam/sdk/tpcds/QueryReader.java | 3 +-
.../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 2 +-
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 2 +-
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 6 +-
.../beam/sdk/tpcds/TpcdsParametersReader.java | 4 +-
.../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 11 +--
.../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 102 ++---
.../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 3 +-
.../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 6 +-
11 files changed, 68 insertions(+), 78 deletions(-)
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
index 69e676f..304fdd2 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
@@ -211,8 +211,7 @@ public class BeamSqlEnvRunner {
// Transform the result from PCollection into
PCollection, and write it to the
// location where results are stored.
PCollection rowStrings =
-rows.apply(
-MapElements.into(TypeDescriptors.strings()).via((Row row) ->
row.toString()));
+
rows.apply(MapElements.into(TypeDescriptors.strings()).via(Row::toString));
rowStrings.apply(
TextIO.write()
.to(
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
index d66b128..d6c8ed8 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
@@ -33,8 +33,8 @@ import org.apache.commons.csv.CSVFormat;
*/
public class CsvToRow extends PTransform, PCollection>
implements Serializable {
- private Schema schema;
- private CSVFormat csvFormat;
+ private final Schema schema;
+ private final CSVFormat csvFormat;
public CSVFormat getCsvFormat() {
return csvFormat;
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
index 7b00a37..c6f3253 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
@@ -35,7 +35,6 @@ public class QueryReader {
*/
public static String readQuery(String queryFileName) throws Exception {
String path = "queries/" + queryFileName + ".sql";
-String query = Resources.toString(Resources.getResource(path),
Charsets.UTF_8);
-return query;
+return Resources.toString(Resources.getResource(path), Charsets.UTF_8);
}
}
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
index 40a8cc5..a087948 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
@@ -33,7 +33,7 @@ import org.apache.commons.csv.CSVFormat;
*/
public class RowToCsv extends PTransform, PCollection>
implements Serializable {
- private CSVFormat csvFormat;
+ private final CSVFormat csvFormat;
public RowToCsv(CSVFormat csvFormat) {
this.csvFormat = csvFormat;
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
index 4f56c1a..bea0261 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
@@ -189,7 +189,7 @@ public class SqlTransformRunner {
try {
tables
.apply(SqlTransform.query(queryString))
-.apply(MapElements.into(TypeDescriptors.strings()).via((Row row)
-> row.toString()))
+
.apply(MapElements.into(TypeDescriptors.strings()).via(Row::toString))
.apply(
TextIO.write()
.to(
diff --git
a/sdks/
[beam] 04/04: Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit b3ef2035abf9ca2dd94a11a1a6aa4440df28adb9 Merge: f805f1c 8fe0c5c Author: Ismaël Mejía AuthorDate: Tue Apr 13 14:22:12 2021 +0200 Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner sdks/java/testing/tpcds/README.md | 68 + sdks/java/testing/tpcds/build.gradle | 108 +- .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 327 +++-- .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 50 +- .../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 47 +- .../org/apache/beam/sdk/tpcds/QueryReader.java | 51 +- .../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 38 +- .../apache/beam/sdk/tpcds/SqlTransformRunner.java | 314 +++-- .../apache/beam/sdk/tpcds/SummaryGenerator.java| 219 ++-- .../beam/sdk/tpcds/TableSchemaJSONLoader.java | 162 +-- .../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 40 +- .../beam/sdk/tpcds/TpcdsOptionsRegistrar.java | 10 +- .../beam/sdk/tpcds/TpcdsParametersReader.java | 136 +- .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 54 +- .../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 120 +- .../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 1336 ++-- ...pcdsOptionsRegistrar.java => package-info.java} | 16 +- .../org/apache/beam/sdk/tpcds/QueryReaderTest.java | 361 +++--- .../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 261 ++-- .../beam/sdk/tpcds/TpcdsParametersReaderTest.java | 110 +- .../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 183 ++- 21 files changed, 2158 insertions(+), 1853 deletions(-)
[beam] 02/04: [BEAM-11712] Add options for input/output paths, make it run via SparkRunner
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit a407d79680d01c35760f3fe4e76cd4192e34edd1
Author: Alexey Romanenko
AuthorDate: Tue Mar 30 18:04:22 2021 +0200
[BEAM-11712] Add options for input/output paths, make it run via SparkRunner
---
sdks/java/testing/tpcds/README.md | 68 ++
sdks/java/testing/tpcds/build.gradle | 11 +++-
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 14 ++---
.../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 18 +-
.../org/apache/beam/sdk/tpcds/QueryReader.java | 49 +---
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 9 +--
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 43 +-
.../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 14 +
.../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 4 +-
9 files changed, 123 insertions(+), 107 deletions(-)
diff --git a/sdks/java/testing/tpcds/README.md
b/sdks/java/testing/tpcds/README.md
new file mode 100644
index 000..89f8073
--- /dev/null
+++ b/sdks/java/testing/tpcds/README.md
@@ -0,0 +1,68 @@
+
+
+# TPC-DS Benchmark
+
+## Google Dataflow Runner
+
+To execute TPC-DS benchmark for 1Gb dataset on Google Dataflow, run the
following example command from the command line:
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.args="--dataSize=1G \
+ --runner=DataflowRunner \
+ --queries=3,26,55 \
+ --tpcParallel=2 \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --project=apache-beam-testing \
+ --stagingLocation=gs://beamsql_tpcds_1/staging \
+ --tempLocation=gs://beamsql_tpcds_2/temp \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --region=us-west1 \
+ --maxNumWorkers=10"
+```
+
+To run a query using ZetaSQL planner (currently Query96 can be run using
ZetaSQL), set the plannerName as below. If not specified, the default planner
is Calcite.
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.args="--dataSize=1G \
+ --runner=DataflowRunner \
+ --queries=96 \
+ --tpcParallel=2 \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --plannerName=org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner
\
+ --project=apache-beam-testing \
+ --stagingLocation=gs://beamsql_tpcds_1/staging \
+ --tempLocation=gs://beamsql_tpcds_2/temp \
+ --region=us-west1 \
+ --maxNumWorkers=10"
+```
+
+## Spark Runner
+
+To execute TPC-DS benchmark with Query3 for 1Gb dataset on Apache Spark 2.x,
run the following example command from the command line:
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.runner=":runners:spark:2"
-Ptpcds.args=" \
+ --runner=SparkRunner \
+ --queries=3 \
+ --tpcParallel=1 \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --dataSize=1G \
+ --resultsDirectory=/path/to/tpcds_results/"
+```
diff --git a/sdks/java/testing/tpcds/build.gradle
b/sdks/java/testing/tpcds/build.gradle
index 6237776..79fb1e8 100644
--- a/sdks/java/testing/tpcds/build.gradle
+++ b/sdks/java/testing/tpcds/build.gradle
@@ -33,7 +33,7 @@ def tpcdsArgsProperty = "tpcds.args"
def tpcdsRunnerProperty = "tpcds.runner"
def tpcdsRunnerDependency = project.findProperty(tpcdsRunnerProperty)
?: ":runners:direct-java"
-def shouldProvideSpark = ":runners:spark".equals(tpcdsRunnerDependency)
+def shouldProvideSpark = ":runners:spark:2".equals(tpcdsRunnerDependency)
def isDataflowRunner =
":runners:google-cloud-dataflow-java".equals(tpcdsRunnerDependency)
def runnerConfiguration = ":runners:direct-java".equals(tpcdsRunnerDependency)
? "shadow" : null
@@ -88,6 +88,15 @@ if (shouldProvideSpark) {
}
}
+// Execute the TPC-DS queries or suites via Gradle.
+//
+// Parameters:
+// -Ptpcds.runner
+// Specify a runner subproject, such as ":runners:spark:2" or
":runners:flink:1.10"
+// Defaults to ":runners:direct-java"
+//
+// -Ptpcds.args
+// Specify the command line for invoking
org.apache.beam.sdk.tpcds.BeamTpcds
task run(type: JavaExec) {
def tpcdsArgsStr = project.findProperty(tpcdsArgsProperty) ?: ""
def tpcdsArgsList = new ArrayList()
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
index 43b97d2..69e676f 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.sdk.tpcds;
+import static org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull;
+
import com.alibaba.fastjson.JSONObject;
import java.util.ArrayList;
import java.util.Arrays;
@@ -66,16 +68,13 @@ public class BeamSqlEnvRunner {
private static final Logger LOG =
LoggerFactory.getLogger(BeamSqlEnvRunner.class);
private static String buildTableC
[beam] branch master updated (f805f1c -> b3ef203)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from f805f1c Merge pull request #14499 from [BEAM-11408, BEAM-11772] Add
explicit output typehints to ensure coder determinism for BQ with auto-sharding
new 28eec3f [BEAM-11712] Make up-to-date build file and codestyle
new a407d79 [BEAM-11712] Add options for input/output paths, make it run
via SparkRunner
new 8fe0c5c [BEAM-11712] Fix static analysis warnings and typos on TPC-DS
module
new b3ef203 Merge pull request #14373: [BEAM-11712] Run TPC-DS via
BeamSQL and Spark runner
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
sdks/java/testing/tpcds/README.md | 68 +
sdks/java/testing/tpcds/build.gradle | 108 +-
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 327 +++--
.../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 50 +-
.../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 47 +-
.../org/apache/beam/sdk/tpcds/QueryReader.java | 51 +-
.../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 38 +-
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 314 +++--
.../apache/beam/sdk/tpcds/SummaryGenerator.java| 219 ++--
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 162 +--
.../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 40 +-
.../beam/sdk/tpcds/TpcdsOptionsRegistrar.java | 10 +-
.../beam/sdk/tpcds/TpcdsParametersReader.java | 136 +-
.../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 54 +-
.../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 120 +-
.../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 1336 ++--
.../org/apache/beam/sdk/tpcds}/package-info.java |4 +-
.../org/apache/beam/sdk/tpcds/QueryReaderTest.java | 361 +++---
.../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 261 ++--
.../beam/sdk/tpcds/TpcdsParametersReaderTest.java | 110 +-
.../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 183 ++-
21 files changed, 2159 insertions(+), 1840 deletions(-)
create mode 100644 sdks/java/testing/tpcds/README.md
copy sdks/java/testing/{load-tests/src/main/java/org/apache/beam/sdk/loadtests
=> tpcds/src/main/java/org/apache/beam/sdk/tpcds}/package-info.java (92%)
