[GitHub] [beam] mwalenia commented on issue #10985: [BEAM-9396] Match Docker image names between Jenkins jobs and Gradle tasks
mwalenia commented on issue #10985: [BEAM-9396] Match Docker image names between Jenkins jobs and Gradle tasks URL: https://github.com/apache/beam/pull/10985#issuecomment-592379776 Run Python Load Tests ParDo Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mwalenia commented on issue #10985: [BEAM-9396] Match Docker image names between Jenkins jobs and Gradle tasks
mwalenia commented on issue #10985: [BEAM-9396] Match Docker image names between Jenkins jobs and Gradle tasks URL: https://github.com/apache/beam/pull/10985#issuecomment-592364965 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592297867 @rohdesamuel precommit is still failing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ananvay commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2
ananvay commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2 URL: https://github.com/apache/beam/pull/10998#issuecomment-592291950 Thanks a lot Ankur!! Overall LGTM, a few minor comments to track the changes made here. Perhaps we should have a bug track those internally so we don't forget to address these? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2
angoenka commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2 URL: https://github.com/apache/beam/pull/10998#issuecomment-592290433 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2
angoenka commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2 URL: https://github.com/apache/beam/pull/10998#issuecomment-592289884 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2
angoenka commented on issue #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2 URL: https://github.com/apache/beam/pull/10998#issuecomment-592288401 R: @ananvay @markflyhigh This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka opened a new pull request #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2
angoenka opened a new pull request #10998: [BEAM-9287] Add Postcommit tests for dataflow runner v2 URL: https://github.com/apache/beam/pull/10998 with hardcoded image for now **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCo
[GitHub] [beam] aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592282786 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #10997: [BEAM-9402] Remove options overwrite
angoenka commented on issue #10997: [BEAM-9402] Remove options overwrite URL: https://github.com/apache/beam/pull/10997#issuecomment-592277859 R: @HuangLED @rohdesamuel This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka opened a new pull request #10997: [BEAM-9402] Remove options overwrite
angoenka opened a new pull request #10997: [BEAM-9402] Remove options overwrite URL: https://github.com/apache/beam/pull/10997 The option does not allow addition of user provided option like in case of validates runner tests. This means that the test always tries to run against DirectRunner. Further the streaming flag can be provided by the target which is executing this test. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)
[GitHub] [beam] ihji commented on issue #10621: [BEAM-9056] Staging artifacts from environment
ihji commented on issue #10621: [BEAM-9056] Staging artifacts from environment URL: https://github.com/apache/beam/pull/10621#issuecomment-592272245 had to force push to fix conflicts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment
ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment URL: https://github.com/apache/beam/pull/10621#discussion_r385469821 ## File path: sdks/python/apache_beam/runners/portability/stager.py ## @@ -377,34 +436,33 @@ def _stage_jar_packages(self, jar_packages, staging_location, temp_dir): for package in local_packages: basename = os.path.basename(package) - staged_path = FileSystems.join(staging_location, basename) - self.stage_artifact(package, staged_path) - resources.append(basename) + resources.append((package, basename)) return resources - def _stage_extra_packages(self, extra_packages, staging_location, temp_dir): -# type: (...) -> List[str] + @staticmethod + def _stage_extra_packages(extra_packages, temp_dir): Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment
ihji commented on a change in pull request #10621: [BEAM-9056] Staging
artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r385469668
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -88,11 +88,13 @@ public static SdkComponents create(RunnerApi.Components
components) {
public static SdkComponents create(PipelineOptions options) {
SdkComponents sdkComponents = new
SdkComponents(RunnerApi.Components.getDefaultInstance(), "");
PortablePipelineOptions portablePipelineOptions =
options.as(PortablePipelineOptions.class);
-sdkComponents.defaultEnvironmentId =
-sdkComponents.registerEnvironment(
-Environments.createOrGetDefaultEnvironment(
+sdkComponents.registerEnvironment(
+Environments.createOrGetDefaultEnvironment(
portablePipelineOptions.getDefaultEnvironmentType(),
-portablePipelineOptions.getDefaultEnvironmentConfig()));
+portablePipelineOptions.getDefaultEnvironmentConfig())
+.toBuilder()
+.addAllDependencies(Environments.getArtifacts(options))
Review comment:
agreed. moved to createOrGetDefaultEnvironment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment
ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment URL: https://github.com/apache/beam/pull/10621#discussion_r385469838 ## File path: sdks/python/apache_beam/runners/portability/stager.py ## @@ -547,21 +601,22 @@ def _desired_sdk_filename_in_staging_location(sdk_location): else: return DATAFLOW_SDK_TARBALL_FILE - def _stage_beam_sdk(self, sdk_remote_location, staging_location, temp_dir): -# type: (...) -> List[str] + @staticmethod + def _stage_beam_sdk(sdk_remote_location, temp_dir): Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment
ihji commented on a change in pull request #10621: [BEAM-9056] Staging
artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r385469751
##
File path:
runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunner.java
##
@@ -200,11 +143,44 @@ public PipelineResult run(Pipeline pipeline) {
prepareJobResponse.getArtifactStagingEndpoint();
String stagingSessionToken = prepareJobResponse.getStagingSessionToken();
+ ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #10621: [BEAM-9056] Staging artifacts from environment
ihji commented on a change in pull request #10621: [BEAM-9056] Staging
artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r385469800
##
File path: sdks/python/apache_beam/runners/portability/stager.py
##
@@ -331,22 +389,23 @@ def _download_file(from_url, to_path):
def _is_remote_path(path):
return path.find('://') != -1
- def _stage_jar_packages(self, jar_packages, staging_location, temp_dir):
-# type: (...) -> List[str]
+ @staticmethod
+ def _stage_jar_packages(jar_packages, temp_dir):
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592262649 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on issue #10980: Address leftover styling comments from PR/10892
lukecwik commented on issue #10980: Address leftover styling comments from PR/10892 URL: https://github.com/apache/beam/pull/10980#issuecomment-592250235 pinging for update This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592244984 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-592244965 Test failure is not caused by the current PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chunyang commented on a change in pull request #10979: [BEAM-8841] Support writing data to BigQuery via Avro in Python SDK
chunyang commented on a change in pull request #10979: [BEAM-8841] Support writing data to BigQuery via Avro in Python SDK URL: https://github.com/apache/beam/pull/10979#discussion_r385442046 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1361,87 +1369,18 @@ def __init__( self.triggering_frequency = triggering_frequency self.insert_retry_strategy = insert_retry_strategy self._validate = validate +self._temp_file_format = temp_file_format or bigquery_tools.FileFormat.JSON Review comment: AFAICT using Avro has no disadvantages compared to JSON for loading data into BigQuery, but would requiring a schema constitute a breaking API change for semantic versioning purposes? Personally I'm for using Avro as default. I guess when users update Beam, they'll specify a `temp_file_format` explicitly to get the old behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (59e71c8 -> f0930f9)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 59e71c8 Merge pull request #10996 from robinyqiu/tmp new 02abb18 [BEAM-9300] convert struct literal in ZetaSQL new a120916 Test schema does not need to be nullable. new f0930f9 Merge pull request #10967 from ibzib/struct-row The 25820 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../provider/bigquery/BeamBigQuerySqlDialect.java | 8 +++ .../beam/sdk/extensions/sql/zetasql/TypeUtils.java | 2 +- .../zetasql/translation/ExpressionConverter.java | 44 --- .../sql/zetasql/ZetaSQLDialectSpecTest.java| 66 +- 4 files changed, 107 insertions(+), 13 deletions(-)
[GitHub] [beam] ibzib merged pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL
ibzib merged pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL URL: https://github.com/apache/beam/pull/10967 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] kennknowles commented on a change in pull request #10972: [WIP] Add DeduplicationByUniqueId transform
kennknowles commented on a change in pull request #10972: [WIP] Add
DeduplicationByUniqueId transform
URL: https://github.com/apache/beam/pull/10972#discussion_r385438853
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -543,6 +545,15 @@ def process(
self.assertEqual(1, len(counters))
self.assertEqual(counters[0].committed, len(''.join(data)))
+ def test_deduplication_by_id(self):
+with self.create_pipeline() as p:
+ res = (p
+ | beam.Create([('id_1', 'value_1'),
+('id_2', 'value_2'),
+('id_1', 'value_1')])
+ | DeduplicationByUniqueId())
Review comment:
I would call it `ChoosePerKey` or something like that since what it actually
does is choose an arbitrary element for each key. Test cases should include
different values for the same key (will require a flexible result matching)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robinyqiu commented on issue #10967: [BEAM-9300] convert struct literal in ZetaSQL
robinyqiu commented on issue #10967: [BEAM-9300] convert struct literal in ZetaSQL URL: https://github.com/apache/beam/pull/10967#issuecomment-592235477 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] kennknowles commented on a change in pull request #10940: [BEAM-9288] Not bundle conscrypt in gRPC vendor
kennknowles commented on a change in pull request #10940: [BEAM-9288] Not
bundle conscrypt in gRPC vendor
URL: https://github.com/apache/beam/pull/10940#discussion_r385434469
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
##
@@ -286,8 +288,23 @@ artifactId=${project.name}
pom.withXml {
def root = asNode()
def dependenciesNode = root.appendNode('dependencies')
+def unshaded = { dependency ->
+ if (dependency instanceof ProjectDependency) {
+def groupId =
dependency.getDependencyProject().mavenGroupId
+def artifactId =
dependency.getDependencyProject().archivesBaseName
+def version = dependency.version
+return config.unshadedDependencies != null &&
config.unshadedDependencies.contains("$groupId:$artifactId:$version")
+ } else {
+def groupId = dependency.group
+def artifactId = dependency.name
+def version = dependency.version
+return config.unshadedDependencies != null &&
config.unshadedDependencies.contains("$groupId:$artifactId:$version")
Review comment:
This does not include the classifier. And since leaving out the classifier
is equal to `jar` (I think always?) it is not just a string comparison.
Does Gradle utilities have anything to parse the `unshadedDependencies` into
`ProjectDependency` objects and then use equality on that object?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ibzib commented on a change in pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL
ibzib commented on a change in pull request #10967: [BEAM-9300] convert struct
literal in ZetaSQL
URL: https://github.com/apache/beam/pull/10967#discussion_r385430202
##
File path:
sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
##
@@ -3249,6 +3248,68 @@ public void testNamedParameterQuery() {
pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
}
+ @Test
+ public void testParameterStruct() {
+String sql = "SELECT @p as ColA";
+ImmutableMap params =
+ImmutableMap.of(
+"p",
+Value.createStructValue(
+TypeFactory.createStructType(
+ImmutableList.of(
+new StructType.StructField(
+"s",
TypeFactory.createSimpleType(TypeKind.TYPE_STRING)),
+new StructType.StructField(
+"i",
TypeFactory.createSimpleType(TypeKind.TYPE_INT64,
+ImmutableList.of(Value.createStringValue("foo"),
Value.createInt64Value(1L;
+
+ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql,
params);
+PCollection stream = BeamSqlRelUtils.toPCollection(pipeline,
beamRelNode);
+
+final Schema innerSchema =
+Schema.of(Field.of("s", FieldType.STRING), Field.of("i",
FieldType.INT64));
+final Schema schema =
+Schema.builder().addNullableField("field1",
FieldType.row(innerSchema)).build();
Review comment:
That must have been left over from a previous iteration. I removed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
aaltay commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#discussion_r385430114 ## File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py ## @@ -71,16 +72,14 @@ def process( class AddTimestampFn(beam.DoFn): """A DoFn that attaches timestamps to its elements. - It takes a string of integers and it attaches to each of them - a timestamp of its same value. + It just takes an element and adds a current timestamp. - For example, [120, 225, 312] will result in: - [(120, Timestamp(120)), (225, Timestamp(225)), (312, Timestamp(312))]. + For example, Sometext will result in: + (Sometext, Timestamp(1234567890). Review comment: I think it is fair to use int() conversion wherever possible and if it does not work do something else. Maybe default to time() as at least it will be more meaningful than length of a string for the example. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #10972: [WIP] Add DeduplicationByUniqueId transform
robertwb commented on a change in pull request #10972: [WIP] Add DeduplicationByUniqueId transform URL: https://github.com/apache/beam/pull/10972#discussion_r385429733 ## File path: sdks/python/apache_beam/runners/sdf_utils.py ## @@ -244,3 +247,47 @@ def get_estimator_state(self): return None return _NoOpWatermarkEstimator() + + +class DeduplicationByUniqueId(ptransform.PTransform): + def __init__(self, windowfn=GlobalWindows(), trigger=None): Review comment: I think if the user wants to specify things in terms of windows and triggers, it would be more natural to manually do windowing before this operation. Instead, perhaps higher-level semantic information could be provided (namely, over what interval should the deduplication occur), and windowing/triggering should be used to accomplish this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #10972: [WIP] Add DeduplicationByUniqueId transform
robertwb commented on a change in pull request #10972: [WIP] Add
DeduplicationByUniqueId transform
URL: https://github.com/apache/beam/pull/10972#discussion_r385429185
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -543,6 +545,15 @@ def process(
self.assertEqual(1, len(counters))
self.assertEqual(counters[0].committed, len(''.join(data)))
+ def test_deduplication_by_id(self):
+with self.create_pipeline() as p:
+ res = (p
+ | beam.Create([('id_1', 'value_1'),
+('id_2', 'value_2'),
+('id_1', 'value_1')])
+ | DeduplicationByUniqueId())
Review comment:
Perhaps call this "DeduplicateByKey"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ibzib commented on issue #10995: [BEAM-9401] bind Flink MiniCluster to localhost
ibzib commented on issue #10995: [BEAM-9401] bind Flink MiniCluster to localhost URL: https://github.com/apache/beam/pull/10995#issuecomment-592228747 @mxm `FlinkSubmissionTest.testDetachedSubmissionBatch` failed: https://builds.apache.org/job/beam_PreCommit_Java_Commit/10180/testReport/junit/org.apache.beam.runners.flink/FlinkSubmissionTest/testDetachedSubmissionBatch_3/ Could you please take a look? I'm thinking it might be a flake. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #10979: [BEAM-8841] Support writing data to BigQuery via Avro in Python SDK
pabloem commented on a change in pull request #10979: [BEAM-8841] Support
writing data to BigQuery via Avro in Python SDK
URL: https://github.com/apache/beam/pull/10979#discussion_r384799264
##
File path: sdks/python/apache_beam/io/gcp/bigquery_test.py
##
@@ -1025,6 +1027,91 @@ def test_file_loads(self):
WriteToBigQuery.Method.FILE_LOADS, triggering_frequency=20)
+class BigQueryFileLoadsIntegrationTests(unittest.TestCase):
+ BIG_QUERY_DATASET_ID = 'python_bq_file_loads_'
+
+ def setUp(self):
+self.test_pipeline = TestPipeline(is_integration_test=True)
+self.runner_name = type(self.test_pipeline.runner).__name__
+self.project = self.test_pipeline.get_option('project')
+
+self.dataset_id = '%s%s%s' % (
+self.BIG_QUERY_DATASET_ID,
+str(int(time.time())),
+random.randint(0, 1))
+self.bigquery_client = bigquery_tools.BigQueryWrapper()
+self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
+self.output_table = '%s.output_table' % (self.dataset_id)
+self.table_ref = bigquery_tools.parse_table_reference(self.output_table)
+_LOGGER.info(
+'Created dataset %s in project %s', self.dataset_id, self.project)
Review comment:
Can you add code to delete the dataset after the test runs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #10979: [BEAM-8841] Support writing data to BigQuery via Avro in Python SDK
pabloem commented on a change in pull request #10979: [BEAM-8841] Support writing data to BigQuery via Avro in Python SDK URL: https://github.com/apache/beam/pull/10979#discussion_r384769744 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -1361,87 +1369,18 @@ def __init__( self.triggering_frequency = triggering_frequency self.insert_retry_strategy = insert_retry_strategy self._validate = validate +self._temp_file_format = temp_file_format or bigquery_tools.FileFormat.JSON Review comment: I'm happy to make AVRO the default format if possible. I guess the issue is that users need to provide the schema, right? Otherwise we cannot write the avro files. We could make AVRO the default, and add a check that the schema was provided (i.e. is neither None nor autodetect) - and error out if that's the case? What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-59222 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on issue #10992: [WIP][BEAM-2939] Add support for reporting the watermark in SplittableDoFns.
lukecwik commented on issue #10992: [WIP][BEAM-2939] Add support for reporting the watermark in SplittableDoFns. URL: https://github.com/apache/beam/pull/10992#issuecomment-592224753 CC: @boyuanzz @robertwb @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on issue #10989: [BEAM-9397] Pass all supported StartBundleContext/FinishBundleContext except output receiver parameters to start bundle/finish bundle methods.
lukecwik commented on issue #10989: [BEAM-9397] Pass all supported StartBundleContext/FinishBundleContext except output receiver parameters to start bundle/finish bundle methods. URL: https://github.com/apache/beam/pull/10989#issuecomment-59901 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-59533 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592220334 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592219819 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated: Bump Dataflow Java worker container version
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 034cef3 Bump Dataflow Java worker container version
new 59e71c8 Merge pull request #10996 from robinyqiu/tmp
034cef3 is described below
commit 034cef37f3bf80e1e98a4b9598d6df230a3792a5
Author: Yueyang Qiu
AuthorDate: Thu Feb 27 14:11:46 2020 -0800
Bump Dataflow Java worker container version
---
runners/google-cloud-dataflow-java/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index 87aeb53..8992305 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -39,7 +39,7 @@ processResources {
filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
'dataflow.legacy_environment_major_version' : '7',
'dataflow.fnapi_environment_major_version' : '7',
-'dataflow.container_version' : 'beam-master-20200213'
+'dataflow.container_version' : 'beam-master-20200227'
]
}
[GitHub] [beam] boyuanzz merged pull request #10996: Bump Dataflow Java worker container version
boyuanzz merged pull request #10996: Bump Dataflow Java worker container version URL: https://github.com/apache/beam/pull/10996 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] robinyqiu commented on a change in pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL
robinyqiu commented on a change in pull request #10967: [BEAM-9300] convert
struct literal in ZetaSQL
URL: https://github.com/apache/beam/pull/10967#discussion_r384855068
##
File path:
sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
##
@@ -3249,6 +3248,68 @@ public void testNamedParameterQuery() {
pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
}
+ @Test
+ public void testParameterStruct() {
+String sql = "SELECT @p as ColA";
+ImmutableMap params =
+ImmutableMap.of(
+"p",
+Value.createStructValue(
+TypeFactory.createStructType(
+ImmutableList.of(
+new StructType.StructField(
+"s",
TypeFactory.createSimpleType(TypeKind.TYPE_STRING)),
+new StructType.StructField(
+"i",
TypeFactory.createSimpleType(TypeKind.TYPE_INT64,
+ImmutableList.of(Value.createStringValue("foo"),
Value.createInt64Value(1L;
+
+ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql,
params);
+PCollection stream = BeamSqlRelUtils.toPCollection(pipeline,
beamRelNode);
+
+final Schema innerSchema =
+Schema.of(Field.of("s", FieldType.STRING), Field.of("i",
FieldType.INT64));
+final Schema schema =
+Schema.builder().addNullableField("field1",
FieldType.row(innerSchema)).build();
Review comment:
Why does the outerSchema need to be nullable here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robinyqiu opened a new pull request #10996: Bump Dataflow Java worker container version
robinyqiu opened a new pull request #10996: Bump Dataflow Java worker container version URL: https://github.com/apache/beam/pull/10996 r: @boyuanzz Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)[**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[; break; + case ROW: Review comment: I thought about replacing [`SqlRowOperator`](https://calcite.apache.org/apidocs/org/apache/calcite/sql/fun/SqlRowOperator.html) with our own custom operator. But I like this better, because it only overrides unparsing without affecting the rest of the operator code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on a change in pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL
ibzib commented on a change in pull request #10967: [BEAM-9300] convert struct
literal in ZetaSQL
URL: https://github.com/apache/beam/pull/10967#discussion_r385395591
##
File path:
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
##
@@ -962,20 +980,28 @@ private RexNode
convertResolvedParameter(ResolvedParameter parameter) {
}
private RexNode convertResolvedStructFieldAccess(ResolvedGetStructField
resolvedGetStructField) {
-return rexBuilder()
-.makeFieldAccess(
-convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr()),
-(int) resolvedGetStructField.getFieldIdx());
+RexNode referencedExpr =
convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr());
+return convertResolvedStructFieldAccessInternal(
Review comment:
Yes, see test I added (`testParameterStructNested`)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on a change in pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL
amaliujia commented on a change in pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL URL: https://github.com/apache/beam/pull/10967#discussion_r385394145 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQuerySqlDialect.java ## @@ -222,6 +222,14 @@ public void unparseCall( } writer.endFunCall(frame); break; + case ROW: Review comment: nit: ideally such `unparse` should happen within `operator` classes. But since we implement it in Beam it is just ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on a change in pull request #10967: [BEAM-9300] convert struct literal in ZetaSQL
amaliujia commented on a change in pull request #10967: [BEAM-9300] convert
struct literal in ZetaSQL
URL: https://github.com/apache/beam/pull/10967#discussion_r385393357
##
File path:
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
##
@@ -962,20 +980,28 @@ private RexNode
convertResolvedParameter(ResolvedParameter parameter) {
}
private RexNode convertResolvedStructFieldAccess(ResolvedGetStructField
resolvedGetStructField) {
-return rexBuilder()
-.makeFieldAccess(
-convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr()),
-(int) resolvedGetStructField.getFieldIdx());
+RexNode referencedExpr =
convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr());
+return convertResolvedStructFieldAccessInternal(
Review comment:
I have lost some of my memory on this code path:
just a question: does this work with deep nested ROW? E.g.
`ROW(ROW(ROW(...)..)...)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Tesio commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Tesio commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#discussion_r385392760 ## File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py ## @@ -71,16 +72,14 @@ def process( class AddTimestampFn(beam.DoFn): """A DoFn that attaches timestamps to its elements. - It takes a string of integers and it attaches to each of them - a timestamp of its same value. + It just takes an element and adds a current timestamp. - For example, [120, 225, 312] will result in: - [(120, Timestamp(120)), (225, Timestamp(225)), (312, Timestamp(312))]. + For example, Sometext will result in: + (Sometext, Timestamp(1234567890). Review comment: Well, as this is an example of debugging functionality - more complicated logic could be added during real-time development. Just to keep it simple for windowing and applicable for strings as well - what about basing on string's length to keep it predictable. And keeping int-base for int values? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10967: [BEAM-9300] convert struct literal in ZetaSQL
amaliujia commented on issue #10967: [BEAM-9300] convert struct literal in ZetaSQL URL: https://github.com/apache/beam/pull/10967#issuecomment-592193913 Nice! Will take a look soon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
iemejia removed a comment on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592191994 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated: [BEAM-9373] Spark/Flink tests fix string concat
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 41e9520 [BEAM-9373] Spark/Flink tests fix string concat
new b9ca34d Merge pull request #10954 from ibzib/BEAM-9373
41e9520 is described below
commit 41e9520a95be1315d452824078835985d6b81256
Author: Kyle Weaver
AuthorDate: Mon Feb 24 13:39:01 2020 -0800
[BEAM-9373] Spark/Flink tests fix string concat
---
runners/flink/job-server/flink_job_server.gradle | 4 ++--
runners/spark/job-server/build.gradle| 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
index 534a10a..ff2c81e 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -220,7 +220,7 @@ def addTestJavaJarCreator(String pyVersion) {
"--python_root_dir ${project.rootDir}/sdks/python",
"--python_version ${pyVersion}",
"--python_container_image
${project.docker_image_default_repo_root}/"
-+ ${project.docker_image_default_repo_prefix}
++ "${project.docker_image_default_repo_prefix}"
+ "python${pyVersion}_sdk:${project.sdk_version}",
]
args "-c", "../../../portability/test_pipeline_jar.sh ${options.join('
')}"
@@ -264,7 +264,7 @@ def addTestFlinkUberJarPy(String pyVersion) {
"--python_root_dir ${project.rootDir}/sdks/python",
"--python_version ${pyVersion}",
"--python_container_image
${project.docker_image_default_repo_root}/"
-+ ${project.docker_image_default_repo_prefix}
++ "${project.docker_image_default_repo_prefix}"
+ "python${pyVersion}_sdk:${project.sdk_version}",
]
args "-c", "../../job-server/test_flink_uber_jar.sh ${options.join('
')}"
diff --git a/runners/spark/job-server/build.gradle
b/runners/spark/job-server/build.gradle
index b85fb77..478e9f9 100644
--- a/runners/spark/job-server/build.gradle
+++ b/runners/spark/job-server/build.gradle
@@ -166,7 +166,7 @@ def addTestJavaJarCreator(String pyVersion) {
"--python_root_dir ${project.rootDir}/sdks/python",
"--python_version ${pyVersion}",
"--python_container_image
${project.docker_image_default_repo_root}/"
-+ ${project.docker_image_default_repo_prefix}
++ "${project.docker_image_default_repo_prefix}"
+ "python${pyVersion}_sdk:${project.sdk_version}",
]
args "-c", "../../portability/test_pipeline_jar.sh ${options.join('
')}"
[GitHub] [beam] iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types
iemejia commented on issue #10974: [BEAM-9384] Add SchemaRegistry.getSchemaCoder to get SchemaCoders for registered types URL: https://github.com/apache/beam/pull/10974#issuecomment-592191994 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib merged pull request #10954: [BEAM-9373] Spark/Flink tests fix string concat
ibzib merged pull request #10954: [BEAM-9373] Spark/Flink tests fix string concat URL: https://github.com/apache/beam/pull/10954 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #10954: [BEAM-9373] Spark/Flink tests fix string concat
chamikaramj commented on issue #10954: [BEAM-9373] Spark/Flink tests fix string concat URL: https://github.com/apache/beam/pull/10954#issuecomment-592191722 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592187646 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592187719 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia commented on issue #10993: [BEAM-9400] Linkage Checker 1.1.4
iemejia commented on issue #10993: [BEAM-9400] Linkage Checker 1.1.4 URL: https://github.com/apache/beam/pull/10993#issuecomment-592184693 Thanks Tomo ! Eager to see the performance improvements in action! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] iemejia merged pull request #10993: [BEAM-9400] Linkage Checker 1.1.4
iemejia merged pull request #10993: [BEAM-9400] Linkage Checker 1.1.4 URL: https://github.com/apache/beam/pull/10993 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (921a9a8 -> f4f12b1)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 921a9a8 [BEAM-8201] Pass all other endpoints through provisioning service. (#10843) add 8ccd27f Linkage Checker 1.1.4 add f4f12b1 Merge pull request #10993: [BEAM-9400] Linkage Checker 1.1.4 No new revisions were added by this update. Summary of changes: build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [beam] rohdesamuel opened a new pull request #10994: Teststreamservice merge
rohdesamuel opened a new pull request #10994: Teststreamservice merge URL: https://github.com/apache/beam/pull/10994 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds
[GitHub] [beam] lukecwik opened a new pull request #10992: [WIP][BEAM-2939] Add support for reporting the watermark in SplittableDoFns.
lukecwik opened a new pull request #10992: [WIP][BEAM-2939] Add support for reporting the watermark in SplittableDoFns. URL: https://github.com/apache/beam/pull/10992 This exposes 3 methods using new DoFn style semantics: `GetInitialWatermarkEstimatorState` `NewWatermarkEstimator` `GetWatermarkEstimatorStateCoder` and one new parameter: `WatermarkEstimatorState` Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Buil
[GitHub] [beam] youngoli commented on issue #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs.
youngoli commented on issue #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs. URL: https://github.com/apache/beam/pull/10991#issuecomment-592171160 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] youngoli commented on issue #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs.
youngoli commented on issue #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs. URL: https://github.com/apache/beam/pull/10991#issuecomment-592171327 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] youngoli opened a new pull request #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs.
youngoli opened a new pull request #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs. URL: https://github.com/apache/beam/pull/10991 The current version of this validation is a bit permissive because it doesn't require the number of main inputs to be specified. This change allows specifying the number of main inputs, while also preserving the existing code path of not specifying it. Along with that change, I made some refactors to existing validation code to try to improve readability and make it more organized. This is filed under BEAM-3301 (SDF) because it is intended to enable validation for SDFs which is difficult without a known number of main inputs. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[: """A DoFn that attaches timestamps to its elements. - It takes a string of integers and it attaches to each of them - a timestamp of its same value. + It just takes an element and adds a current timestamp. - For example, [120, 225, 312] will result in: - [(120, Timestamp(120)), (225, Timestamp(225)), (312, Timestamp(312))]. + For example, Sometext will result in: + (Sometext, Timestamp(1234567890). Review comment: That will be better by making tests reproducible. That will result in elements with timestamps across a wide range. Testers would not be able to predict what element will be in which window. we change the input to be a tuple , with a comma separation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592149918 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592149742 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] yifanzou commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
yifanzou commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592138186 > > The linter failed due to "ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device" > > @alanmyrvold @udim @yifanzou -- Is this a symptom of the problem of growing temp directory? The Jenkins workspace and the docker images are usually the two monsters eat the disk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
pabloem commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592136551 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] rohdesamuel commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
rohdesamuel commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592136428 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] Tesio commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
Tesio commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#discussion_r385313258 ## File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py ## @@ -71,16 +72,14 @@ def process( class AddTimestampFn(beam.DoFn): """A DoFn that attaches timestamps to its elements. - It takes a string of integers and it attaches to each of them - a timestamp of its same value. + It just takes an element and adds a current timestamp. - For example, [120, 225, 312] will result in: - [(120, Timestamp(120)), (225, Timestamp(225)), (312, Timestamp(312))]. + For example, Sometext will result in: + (Sometext, Timestamp(1234567890). Review comment: Fair point. The reason of that change is to make it work not only with int values, as an example is called wordcount. Will it be ok if we will use the positive hashed value of "Sometext" as a base for timestamp? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chunyang commented on issue #10979: [BEAM-8841] Support writing data to BigQuery via Avro in Python SDK
chunyang commented on issue #10979: [BEAM-8841] Support writing data to
BigQuery via Avro in Python SDK
URL: https://github.com/apache/beam/pull/10979#issuecomment-592120245
The PreCommit failures look like they're being fixed by #10982 .
The failing Python 3.7 PostCommit integration tests might be flaky? I was
not able to reproduce it--got the following instead:
```
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [(None,
None, None, datetime.date(3000, 12, 31), None, None, None, None), (None, None,
None, None, datetime.time(23, 59, 59), None, None, None), (0.33, None, None,
None, None, None, None, None), (None, Decimal('10'), None, None, None, None,
None, None), (None, None, None, None, None, datetime.datetime(2018, 12, 31, 12,
44, 31), None, None), (None, None, None, None, None, None,
datetime.datetime(2018, 12, 31, 12, 44, 31, 744957, tzinfo=), None),
(None, None, None, None, None, None, None, 'POINT(30 10)'), (None, None,
b'\xab\xac', None, None, None, None, None), (0.33, Decimal('10'), b'\xab\xac',
datetime.date(3000, 12, 31), datetime.time(23, 59, 59), datetime.datetime(2018,
12, 31, 12, 44, 31), datetime.datetime(2018, 12, 31, 12, 44, 31, 744957,
tzinfo=), 'POINT(30 10)')]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kennknowles commented on a change in pull request #10990: disable coder inference for rows
kennknowles commented on a change in pull request #10990: disable coder
inference for rows
URL: https://github.com/apache/beam/pull/10990#discussion_r385304944
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
##
@@ -38,61 +35,62 @@
}
/** Filter function used for Set operators. */
- public static class SetOperatorFilteringDoFn extends DoFn, Row> {
-private TupleTag leftTag;
-private TupleTag rightTag;
-private BeamSetOperatorRelBase.OpType opType;
+ public static class SetOperatorFilteringDoFn extends DoFn {
+private final String leftTag;
+private final String rightTag;
+private final String numRowsField;
+private final BeamSetOperatorRelBase.OpType opType;
// ALL?
-private boolean all;
+private final boolean all;
public SetOperatorFilteringDoFn(
-TupleTag leftTag,
-TupleTag rightTag,
+String leftTag,
+String rightTag,
+String numRowsField,
BeamSetOperatorRelBase.OpType opType,
boolean all) {
this.leftTag = leftTag;
this.rightTag = rightTag;
+ this.numRowsField = numRowsField;
this.opType = opType;
this.all = all;
}
@ProcessElement
-public void processElement(ProcessContext ctx) {
- CoGbkResult coGbkResult = ctx.element().getValue();
- Iterable leftRows = coGbkResult.getAll(leftTag);
- Iterable rightRows = coGbkResult.getAll(rightTag);
+public void processElement(@Element Row element, OutputReceiver o) {
+ Row key = element.getRow("key");
+ long numLeftRows = 0;
+ long numRightRows = 0;
+ if (!Iterables.isEmpty(element.getIterable(leftTag))) {
+numLeftRows =
+
Iterables.getOnlyElement(element.getIterable(leftTag)).getInt64(numRowsField);
+ }
+ if (!Iterables.isEmpty(element.getIterable(rightTag))) {
+numRightRows =
+
Iterables.getOnlyElement(element.getIterable(rightTag)).getInt64(numRowsField);
+ }
+
switch (opType) {
case UNION:
if (all) {
-// output both left & right
-Iterator iter = leftRows.iterator();
-while (iter.hasNext()) {
- ctx.output(iter.next());
-}
-iter = rightRows.iterator();
-while (iter.hasNext()) {
- ctx.output(iter.next());
+for (int i = 0; i < numLeftRows + numRightRows; i++) {
+ o.output(key);
Review comment:
This optimization seems to have nothing to do with disabling coder
inference. Please split into separate commit so git history shows it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kennknowles commented on a change in pull request #10990: disable coder inference for rows
kennknowles commented on a change in pull request #10990: disable coder
inference for rows
URL: https://github.com/apache/beam/pull/10990#discussion_r385303509
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
##
@@ -65,6 +66,16 @@ public BeamSetOperatorRelBase(BeamRelNode beamRelNode,
OpType opType, boolean al
inputs);
PCollection leftRows = inputs.get(0);
PCollection rightRows = inputs.get(1);
+Schema leftSchema = leftRows.getSchema();
+Schema rightSchema = rightRows.getSchema();
+if (!leftSchema.typesEqual(rightSchema)) {
+ throw new IllegalArgumentException(
+ "Can't intersect two tables with different schemas."
Review comment:
This is the base class, so shouldn't be calling it "intersect". Incidentally
I don't think having a base class adds much value here, so inlining or
inverting would be a-ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
aaltay commented on a change in pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#discussion_r385304557 ## File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py ## @@ -71,16 +72,14 @@ def process( class AddTimestampFn(beam.DoFn): """A DoFn that attaches timestamps to its elements. - It takes a string of integers and it attaches to each of them - a timestamp of its same value. + It just takes an element and adds a current timestamp. - For example, [120, 225, 312] will result in: - [(120, Timestamp(120)), (225, Timestamp(225)), (312, Timestamp(312))]. + For example, Sometext will result in: + (Sometext, Timestamp(1234567890). Review comment: Note that this change will make timestamp unpredictable would likely result in flaky tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] aaltay commented on issue #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test
aaltay commented on issue #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#issuecomment-592118125 > > As a high level comment do we need an integration test for `streaming_wordcount_debugging_it_test.py`? Integration test for `streaming_wordcount_it_test.py` tests for the basic streaming wordcount use case. debugging version has more debugging information and a more educational purpose. > > As I wrote in the bug, my reasoning is that it's example code so it should be tested to see that it runs properly. Ack. Did not see the bug. Assigned this to you for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chadrik commented on issue #10822: [BEAM-7746] Minor typing updates / fixes
chadrik commented on issue #10822: [BEAM-7746] Minor typing updates / fixes URL: https://github.com/apache/beam/pull/10822#issuecomment-592117791 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] robertwb merged pull request #10843: [BEAM-8201] Pass all other endpoints through provisioning service.
robertwb merged pull request #10843: [BEAM-8201] Pass all other endpoints through provisioning service. URL: https://github.com/apache/beam/pull/10843 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (8b69513 -> 921a9a8)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8b69513 Merge pull request #10987 from lukecwik/beam5524 add 921a9a8 [BEAM-8201] Pass all other endpoints through provisioning service. (#10843) No new revisions were added by this update. Summary of changes: .../control/DefaultJobBundleFactory.java | 7 +- .../environment/DockerEnvironmentFactory.java | 8 +- sdks/go/container/boot.go | 35 ++- .../model/fnexecution_v1/beam_provision_api.pb.go | 291 +++-- sdks/java/container/boot.go| 32 ++- .../runners/portability/fn_api_runner.py | 43 ++- sdks/python/container/boot.go | 32 ++- 7 files changed, 154 insertions(+), 294 deletions(-)
[GitHub] [beam] robertwb commented on issue #10843: [BEAM-8201] Pass all other endpoints through provisioning service.
robertwb commented on issue #10843: [BEAM-8201] Pass all other endpoints through provisioning service. URL: https://github.com/apache/beam/pull/10843#issuecomment-592116771 Thanks. I see JavaPortabilityApiJava11 is failing everywhere (and seems to be irrelevent). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia edited a comment on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia edited a comment on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592113465 Run PythonLint PreCommit trying to hit another Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
amaliujia commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592113465 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on issue #10989: [BEAM-9397] Pass all supported StartBundleContext/FinishBundleContext except output receiver parameters to start bundle/finish bundle methods.
lukecwik commented on issue #10989: [BEAM-9397] Pass all supported StartBundleContext/FinishBundleContext except output receiver parameters to start bundle/finish bundle methods. URL: https://github.com/apache/beam/pull/10989#issuecomment-592111413 Run RAT PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on issue #10989: [BEAM-9397] Pass all supported StartBundleContext/FinishBundleContext except output receiver parameters to start bundle/finish bundle methods.
lukecwik commented on issue #10989: [BEAM-9397] Pass all supported StartBundleContext/FinishBundleContext except output receiver parameters to start bundle/finish bundle methods. URL: https://github.com/apache/beam/pull/10989#issuecomment-592111375 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax opened a new pull request #10990: disable coder inference for rows
reuvenlax opened a new pull request #10990: disable coder inference for rows URL: https://github.com/apache/beam/pull/10990 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch asf-site updated: Publishing website 2020/02/27 18:32:09 at commit 8b69513
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 6a74a84 Publishing website 2020/02/27 18:32:09 at commit 8b69513 6a74a84 is described below commit 6a74a84d1a9f2da3d18f3f8b80b5b9dde1b768da Author: jenkins AuthorDate: Thu Feb 27 18:32:10 2020 + Publishing website 2020/02/27 18:32:09 at commit 8b69513 --- website/generated-content/contribute/ptransform-style-guide/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/contribute/ptransform-style-guide/index.html b/website/generated-content/contribute/ptransform-style-guide/index.html index 91e010f..35808d7 100644 --- a/website/generated-content/contribute/ptransform-style-guide/index.html +++ b/website/generated-content/contribute/ptransform-style-guide/index.html @@ -591,7 +591,7 @@ Strive to make such incompatible behavior changes cause a compile error (e.g. it private final TupleTagmooTag = new TupleTag () {}; private final TupleTag blahTag = new TupleTag () {}; ... - PCollectionTuple apply(... input) { + PCollectionTuple expand(... input) { ... PCollection moo = ...; PCollection blah = ...;
[GitHub] [beam] lukecwik merged pull request #10987: [BEAM-5524] Fix minor issue in style guide.
lukecwik merged pull request #10987: [BEAM-5524] Fix minor issue in style guide. URL: https://github.com/apache/beam/pull/10987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated: [BEAM-5524] Fix minor issue in style guide.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7cd1353 [BEAM-5524] Fix minor issue in style guide.
new 8b69513 Merge pull request #10987 from lukecwik/beam5524
7cd1353 is described below
commit 7cd1353e2125bd5cd40d9b24ace2c8b35f710a3e
Author: Luke Cwik
AuthorDate: Thu Feb 27 09:28:01 2020 -0800
[BEAM-5524] Fix minor issue in style guide.
---
website/src/contribute/ptransform-style-guide.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/src/contribute/ptransform-style-guide.md
b/website/src/contribute/ptransform-style-guide.md
index 4f4d253..de531a4 100644
--- a/website/src/contribute/ptransform-style-guide.md
+++ b/website/src/contribute/ptransform-style-guide.md
@@ -239,7 +239,7 @@ public class MyTransform extends PTransform<...,
PCollectionTuple> {
private final TupleTag mooTag = new TupleTag() {};
private final TupleTag blahTag = new TupleTag() {};
...
- PCollectionTuple apply(... input) {
+ PCollectionTuple expand(... input) {
...
PCollection moo = ...;
PCollection blah = ...;
[GitHub] [beam] aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors
aaltay commented on issue #10982: [BEAM-9392] Fix Multi TestStream assertion errors URL: https://github.com/apache/beam/pull/10982#issuecomment-592107642 > The linter failed due to "ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device" @alanmyrvold @udim @yifanzou -- Is this a symptom of the problem of growing temp directory? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
