[jira] [Updated] (BEAM-9198) BeamSQL aggregation analytics functionality
[
https://issues.apache.org/jira/browse/BEAM-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Wang updated BEAM-9198:
---
Description:
Mentor email: ruw...@google.com. Feel free to send emails for your questions.
Project Information
-
BeamSQL has a long list of of aggregation/aggregation analytics functionalities
to support.
To begin with, you will need to support this syntax:
{code:sql}
analytic_function_name ( [ argument_list ] )
OVER (
[ PARTITION BY partition_expression_list ]
[ ORDER BY expression [{ ASC | DESC }] [, ...] ]
[ window_frame_clause ]
)
{code}
This will requires touch core components of BeamSQL:
1. SQL parser to support the syntax above.
2. SQL core to implement physical relational operator.
3. Distributed algorithms to implement a list of functions in a distributed
manner.
4. Build benchmarks to measure performance of your implementation.
To understand what SQL analytics functionality is, you could check this great
explanation doc:
https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
To know about Beam's programming model, check:
https://beam.apache.org/documentation/programming-guide/#overview
was:
BeamSQL has a long list of of aggregation/aggregation analytics functionalities
to support.
To begin with, you will need to support this syntax:
{code:sql}
analytic_function_name ( [ argument_list ] )
OVER (
[ PARTITION BY partition_expression_list ]
[ ORDER BY expression [{ ASC | DESC }] [, ...] ]
[ window_frame_clause ]
)
{code}
This will requires touch core components of BeamSQL:
1. SQL parser to support the syntax above.
2. SQL core to implement physical relational operator.
3. Distributed algorithms to implement a list of functions in a distributed
manner.
4. Build benchmarks to measure performance of your implementation.
To understand what SQL analytics functionality is, you could check this great
explanation doc:
https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
To know about Beam's programming model, check:
https://beam.apache.org/documentation/programming-guide/#overview
> BeamSQL aggregation analytics functionality
>
>
> Key: BEAM-9198
> URL: https://issues.apache.org/jira/browse/BEAM-9198
> Project: Beam
> Issue Type: Task
> Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> Mentor email: ruw...@google.com. Feel free to send emails for your questions.
> Project Information
> -
> BeamSQL has a long list of of aggregation/aggregation analytics
> functionalities to support.
> To begin with, you will need to support this syntax:
> {code:sql}
> analytic_function_name ( [ argument_list ] )
> OVER (
> [ PARTITION BY partition_expression_list ]
> [ ORDER BY expression [{ ASC | DESC }] [, ...] ]
> [ window_frame_clause ]
> )
> {code}
> This will requires touch core components of BeamSQL:
> 1. SQL parser to support the syntax above.
> 2. SQL core to implement physical relational operator.
> 3. Distributed algorithms to implement a list of functions in a distributed
> manner.
> 4. Build benchmarks to measure performance of your implementation.
> To understand what SQL analytics functionality is, you could check this great
> explanation doc:
> https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
> To know about Beam's programming model, check:
> https://beam.apache.org/documentation/programming-guide/#overview
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9331) The Row object needs better builders
[
https://issues.apache.org/jira/browse/BEAM-9331?focusedWorklogId=390462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390462
]
ASF GitHub Bot logged work on BEAM-9331:
Author: ASF GitHub Bot
Created on: 21/Feb/20 06:56
Start Date: 21/Feb/20 06:56
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on pull request #10883:
[BEAM-9331] Add better Row builders
URL: https://github.com/apache/beam/pull/10883#discussion_r382422957
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
##
@@ -477,6 +477,172 @@ public void testCreateMapWithRowValue() {
assertEquals(data, row.getMap("map"));
}
+ @Test
Review comment:
Elaborating on the above comment:
`getValues` is currently not safe (it makes the Row mutable), if the meaning
changes that it returns the logical type the implementation will need to make a
copy of the content as it needs to translate the basetype to the logicaltype.
As it's a copy the Row is immutable again. See this for test:
```
@Test
public void testImmutability() {
Schema schema =
Schema.builder().addInt32Field("a").addInt32Field("b").build();
Row row1 = Row.withSchema(schema).addValue(1).addValue(2).build();
List values = row1.getValues();
values.set(1, 3);
assertEquals(2,(int)row1.getValue(1));
}
```
Then the question is left for `attach` or `detach`: if you make values an
immutable array it can be returned with `detach`, as it's immutable that's
safe. The `attach` needs then make a copy of the array to an immutable array
(or check if it's immutable) and `build` needs to make the immutable array as
well.
-
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390462)
Time Spent: 3h (was: 2h 50m)
> The Row object needs better builders
>
>
> Key: BEAM-9331
> URL: https://issues.apache.org/jira/browse/BEAM-9331
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Users should be able to build a Row object by specifying field names. Desired
> syntax:
>
> Row.withSchema(schema)
> .withFieldName("field1", "value)
> .withFieldName("field2.field3", value)
> .build()
>
> Users should also have a builder that allows taking an existing row and
> changing specific fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9240) Check for Nullability in typesEqual() method of FieldType class
[
https://issues.apache.org/jira/browse/BEAM-9240?focusedWorklogId=390438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390438
]
ASF GitHub Bot logged work on BEAM-9240:
Author: ASF GitHub Bot
Created on: 21/Feb/20 05:56
Start Date: 21/Feb/20 05:56
Worklog Time Spent: 10m
Work Description: rahul8383 commented on issue #10744: [BEAM-9240]: Check
for Nullability in typesEqual() method of FieldTyp…
URL: https://github.com/apache/beam/pull/10744#issuecomment-589510661
is there anything that I can do to run the tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390438)
Time Spent: 2.5h (was: 2h 20m)
> Check for Nullability in typesEqual() method of FieldType class
> ---
>
> Key: BEAM-9240
> URL: https://issues.apache.org/jira/browse/BEAM-9240
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {{If two schemas are created like this:}}
> {{Schema schema1 = Schema.builder().addStringField("col1").build();}}
> {{Schema schema2 = Schema.builder().addNullableField("col1",
> FieldType.STRING).build();}}
>
> {{schema1.typeEquals(schema2) returns "true" even though the schemas differ
> by Nullability}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9349: Priority: Minor (was: Major) > Upgrade to joda time 2.10.5 to get updated TZDB > --- > > Key: BEAM-9349 > URL: https://issues.apache.org/jira/browse/BEAM-9349 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?focusedWorklogId=390435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390435 ] ASF GitHub Bot logged work on BEAM-9349: Author: ASF GitHub Bot Created on: 21/Feb/20 05:49 Start Date: 21/Feb/20 05:49 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10923: [BEAM-9349] Update joda-time version to 2.10.5 URL: https://github.com/apache/beam/pull/10923#issuecomment-589509092 Run CommunityMetrics PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390435) Time Spent: 40m (was: 0.5h) > Upgrade to joda time 2.10.5 to get updated TZDB > --- > > Key: BEAM-9349 > URL: https://issues.apache.org/jira/browse/BEAM-9349 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?focusedWorklogId=390432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390432 ] ASF GitHub Bot logged work on BEAM-9349: Author: ASF GitHub Bot Created on: 21/Feb/20 05:34 Start Date: 21/Feb/20 05:34 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10923: [BEAM-9349] Update joda-time version to 2.10.5 URL: https://github.com/apache/beam/pull/10923#issuecomment-589506062 R: @jensengrey This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390432) Time Spent: 0.5h (was: 20m) > Upgrade to joda time 2.10.5 to get updated TZDB > --- > > Key: BEAM-9349 > URL: https://issues.apache.org/jira/browse/BEAM-9349 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?focusedWorklogId=390430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390430 ] ASF GitHub Bot logged work on BEAM-9349: Author: ASF GitHub Bot Created on: 21/Feb/20 05:33 Start Date: 21/Feb/20 05:33 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10923: [BEAM-9349] Update joda-time version to 2.10.5 URL: https://github.com/apache/beam/pull/10923#issuecomment-589505762 Linkage checker results: ``` Linkage Check difference on beam-sdks-java-core between master(76501abb) and HEAD(e64f9200): Lines starting with '<' mean the branch remedies the errors (good) Lines starting with '>' mean the branch introduces new errors (bad) 31c31 < 906 actionable tasks: 160 executed, 746 up-to-date --- > 906 actionable tasks: 184 executed, 722 up-to-date Linkage Check difference on beam-sdks-java-io-google-cloud-platform between master(76501abb) and HEAD(e64f9200): (no difference) Linkage Check difference on beam-runners-google-cloud-dataflow-java between master(76501abb) and HEAD(e64f9200): (no difference) Linkage Check difference on beam-sdks-java-io-hadoop-format between master(76501abb) and HEAD(e64f9200): Lines starting with '<' mean the branch remedies the errors (good) Lines starting with '>' mean the branch introduces new errors (bad) 66a67,70 > Class com.google.common.reflect.TypeToken is not found; > referenced by 2 class files > org.joda.convert.TypeStringConverter (joda-convert-1.9.2.jar) > org.joda.convert.TypeTokenStringConverter (joda-convert-1.9.2.jar) 5221a5226,5228 > joda-convert-1.9.2.jar is at: > org.apache.beam:beam-sdks-java-io-hadoop-format:2.20.0-SNAPSHOT (compile) / org.apache.beam:beam-sdks-java-core:2.20.0-SNAPSHOT (compile) / joda-time:joda-time:2.10.5 (compile) / org.joda:joda-convert:1.9.2 (compile, optional) > and 1 dependency path. ``` The joda-convert guava dependency is optional and only accessed via reflection as per https://repo1.maven.org/maven2/org/joda/joda-convert/1.9.2/joda-convert-1.9.2.pom This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390430) Time Spent: 20m (was: 10m) > Upgrade to joda time 2.10.5 to get updated TZDB > --- > > Key: BEAM-9349 > URL: https://issues.apache.org/jira/browse/BEAM-9349 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?focusedWorklogId=390428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390428 ] ASF GitHub Bot logged work on BEAM-9349: Author: ASF GitHub Bot Created on: 21/Feb/20 05:32 Start Date: 21/Feb/20 05:32 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10923: [BEAM-9349] Update joda-time version to 2.10.5 URL: https://github.com/apache/beam/pull/10923 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Updated] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9349: Fix Version/s: (was: 2.16.0) 2.20.0 > Upgrade to joda time 2.10.5 to get updated TZDB > --- > > Key: BEAM-9349 > URL: https://issues.apache.org/jira/browse/BEAM-9349 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.20.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
[ https://issues.apache.org/jira/browse/BEAM-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9349: Status: Open (was: Triage Needed) > Upgrade to joda time 2.10.5 to get updated TZDB > --- > > Key: BEAM-9349 > URL: https://issues.apache.org/jira/browse/BEAM-9349 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Fix For: 2.16.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9349) Upgrade to joda time 2.10.5 to get updated TZDB
Luke Cwik created BEAM-9349: --- Summary: Upgrade to joda time 2.10.5 to get updated TZDB Key: BEAM-9349 URL: https://issues.apache.org/jira/browse/BEAM-9349 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Luke Cwik Assignee: Luke Cwik Fix For: 2.16.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-6628) Update GCP dependencies to a recent version
[ https://issues.apache.org/jira/browse/BEAM-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik resolved BEAM-6628. - Fix Version/s: 2.20.0 Assignee: Tomo Suzuki Resolution: Fixed Many of the core apiary/grpc/google libraries have been updated. > Update GCP dependencies to a recent version > --- > > Key: BEAM-6628 > URL: https://issues.apache.org/jira/browse/BEAM-6628 > Project: Beam > Issue Type: Task > Components: io-java-gcp >Reporter: Kenneth Jung >Assignee: Tomo Suzuki >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Update GCP client libraries to a recent version. These libraries must be > updated together in order to keep shared dependencies in sync. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-6432) Set dependent libraries' versions for the starter archetype automatically
[
https://issues.apache.org/jira/browse/BEAM-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik resolved BEAM-6432.
-
Fix Version/s: 2.12.0
Assignee: Kenneth Knowles (was: Kengo Seki)
Resolution: Fixed
> Set dependent libraries' versions for the starter archetype automatically
> -
>
> Key: BEAM-6432
> URL: https://issues.apache.org/jira/browse/BEAM-6432
> Project: Beam
> Issue Type: Improvement
> Components: examples-java
>Reporter: Kengo Seki
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.12.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I generated an empty project from beam-sdks-java-maven-archetypes-starter and
> found that I had to replace the placeholders for dependency versions
> ({{@...version@}}) with concrete values myself.
> It'd be convenient for users if they were automatically replaced, just like
> beam-sdks-java-maven-archetypes-examples do.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-6432) Set dependent libraries' versions for the starter archetype automatically
[
https://issues.apache.org/jira/browse/BEAM-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik reassigned BEAM-6432:
---
Assignee: Kengo Seki (was: Kenneth Knowles)
> Set dependent libraries' versions for the starter archetype automatically
> -
>
> Key: BEAM-6432
> URL: https://issues.apache.org/jira/browse/BEAM-6432
> Project: Beam
> Issue Type: Improvement
> Components: examples-java
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
> Fix For: 2.12.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I generated an empty project from beam-sdks-java-maven-archetypes-starter and
> found that I had to replace the placeholders for dependency versions
> ({{@...version@}}) with concrete values myself.
> It'd be convenient for users if they were automatically replaced, just like
> beam-sdks-java-maven-archetypes-examples do.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390390
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 02:27
Start Date: 21/Feb/20 02:27
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #10921:
[BEAM-9022] publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382366200
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -236,6 +236,9 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
done
+ echo '-Generating and Pushing Spark job server
image-'
+ ./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
Then let’s add it to documentation some where. If I am a spark user, I would
really wonder which version is being supported.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390390)
Time Spent: 2h (was: 1h 50m)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9319) ResourceExhausted: topics-per-project
[ https://issues.apache.org/jira/browse/BEAM-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041466#comment-17041466 ] Brian Hulette commented on BEAM-9319: - Yeah, we're definitely still leaking topics. It looks like there were 131 leaked today from TestPubSub. I'll have to investigate further > ResourceExhausted: topics-per-project > - > > Key: BEAM-9319 > URL: https://issues.apache.org/jira/browse/BEAM-9319 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Ahmet Altay >Assignee: Brian Hulette >Priority: Major > > Tests are failing due to quota issues. Do we need to clean up topics after > tests or set a shorter TTL? > Log: https://builds.apache.org/job/beam_PreCommit_Python_Commit/11178/ > Error: > 08:24:40 > == > 08:24:40 ERROR: test_streaming_wordcount_it > (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) > 08:24:40 > -- > 08:24:40 Traceback (most recent call last): > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", > line 58, in setUp > 08:24:40 self.pub_client.topic_path(self.project, INPUT_TOPIC + > self.uuid)) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/_gapic.py", > line 40, in > 08:24:40 fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw) # > noqa > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", > line 332, in create_topic > 08:24:40 request, retry=retry, timeout=timeout, metadata=metadata > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", > line 143, in __call__ > 08:24:40 return wrapped_func(*args, **kwargs) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/retry.py", > line 286, in retry_wrapped_func > 08:24:40 on_error=on_error, > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/retry.py", > line 184, in retry_target > 08:24:40 return target() > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/timeout.py", > line 214, in func_with_timeout > 08:24:40 return func(*args, **kwargs) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", > line 59, in error_remapped_callable > 08:24:40 six.raise_from(exceptions.from_grpc_error(exc), exc) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/six.py", > line 738, in raise_from > 08:24:40 raise value > 08:24:40 ResourceExhausted: 429 Your project has exceeded a limit: > (type="topics-per-project", current=1, maximum=1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9319) ResourceExhausted: topics-per-project
[ https://issues.apache.org/jira/browse/BEAM-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041441#comment-17041441 ] Ahmet Altay commented on BEAM-9319: --- Thank you! Do we still need to address the leaking problem? > ResourceExhausted: topics-per-project > - > > Key: BEAM-9319 > URL: https://issues.apache.org/jira/browse/BEAM-9319 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Ahmet Altay >Assignee: Brian Hulette >Priority: Major > > Tests are failing due to quota issues. Do we need to clean up topics after > tests or set a shorter TTL? > Log: https://builds.apache.org/job/beam_PreCommit_Python_Commit/11178/ > Error: > 08:24:40 > == > 08:24:40 ERROR: test_streaming_wordcount_it > (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) > 08:24:40 > -- > 08:24:40 Traceback (most recent call last): > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", > line 58, in setUp > 08:24:40 self.pub_client.topic_path(self.project, INPUT_TOPIC + > self.uuid)) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/_gapic.py", > line 40, in > 08:24:40 fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw) # > noqa > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", > line 332, in create_topic > 08:24:40 request, retry=retry, timeout=timeout, metadata=metadata > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", > line 143, in __call__ > 08:24:40 return wrapped_func(*args, **kwargs) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/retry.py", > line 286, in retry_wrapped_func > 08:24:40 on_error=on_error, > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/retry.py", > line 184, in retry_target > 08:24:40 return target() > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/timeout.py", > line 214, in func_with_timeout > 08:24:40 return func(*args, **kwargs) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", > line 59, in error_remapped_callable > 08:24:40 six.raise_from(exceptions.from_grpc_error(exc), exc) > 08:24:40 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/six.py", > line 738, in raise_from > 08:24:40 raise value > 08:24:40 ResourceExhausted: 429 Your project has exceeded a limit: > (type="topics-per-project", current=1, maximum=1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=390380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390380 ] ASF GitHub Bot logged work on BEAM-9056: Author: ASF GitHub Bot Created on: 21/Feb/20 01:48 Start Date: 21/Feb/20 01:48 Worklog Time Spent: 10m Work Description: ihji commented on issue #10621: [BEAM-9056] Staging artifacts from environment URL: https://github.com/apache/beam/pull/10621#issuecomment-589459736 @chamikaramj @robertwb PTAL. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390380) Time Spent: 2h 20m (was: 2h 10m) > Staging artifacts from environment > -- > > Key: BEAM-9056 > URL: https://issues.apache.org/jira/browse/BEAM-9056 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > staging artifacts from artifact information embedded in environment proto. > detail: > https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=390379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390379
]
ASF GitHub Bot logged work on BEAM-9056:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:48
Start Date: 21/Feb/20 01:48
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10621: [BEAM-9056]
Staging artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r382357053
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -171,6 +193,83 @@ public static Environment createProcessEnvironment(
}
}
+ public static Collection getArtifacts(PipelineOptions
options) {
Review comment:
It gets a list of all artifacts from the experiment as well as from
ClassLoader. The method was moved from `PortableRunner` to pre-generate the
dependencies (originally the dependencies were calculated when submitting
pipeline).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390379)
Time Spent: 2h 10m (was: 2h)
> Staging artifacts from environment
> --
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> staging artifacts from artifact information embedded in environment proto.
> detail:
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=390378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390378
]
ASF GitHub Bot logged work on BEAM-9056:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:47
Start Date: 21/Feb/20 01:47
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10621: [BEAM-9056]
Staging artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r382356918
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -171,6 +193,83 @@ public static Environment createProcessEnvironment(
}
}
+ public static Collection getArtifacts(PipelineOptions
options) {
+Set pathsToStage = Sets.newHashSet();
+List experiments =
options.as(ExperimentalOptions.class).getExperiments();
+if (experiments != null) {
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390378)
Time Spent: 2h (was: 1h 50m)
> Staging artifacts from environment
> --
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> staging artifacts from artifact information embedded in environment proto.
> detail:
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=390377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390377
]
ASF GitHub Bot logged work on BEAM-9056:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:47
Start Date: 21/Feb/20 01:47
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10621: [BEAM-9056]
Staging artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r382356799
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -171,6 +193,83 @@ public static Environment createProcessEnvironment(
}
}
+ public static Collection getArtifacts(PipelineOptions
options) {
+Set pathsToStage = Sets.newHashSet();
+List experiments =
options.as(ExperimentalOptions.class).getExperiments();
+if (experiments != null) {
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390377)
Time Spent: 1h 50m (was: 1h 40m)
> Staging artifacts from environment
> --
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> staging artifacts from artifact information embedded in environment proto.
> detail:
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=390374=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390374
]
ASF GitHub Bot logged work on BEAM-9056:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:41
Start Date: 21/Feb/20 01:41
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10621: [BEAM-9056]
Staging artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r382355182
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -171,6 +193,83 @@ public static Environment createProcessEnvironment(
}
}
+ public static Collection getArtifacts(PipelineOptions
options) {
Review comment:
It gets a list of all artifacts from the experiment as well as from
ClassLoader. The method was moved from `PortableRunner` to pre-generate the
dependencies (originally the dependencies were calculated when submitting
pipeline).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390374)
Time Spent: 1h 40m (was: 1.5h)
> Staging artifacts from environment
> --
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> staging artifacts from artifact information embedded in environment proto.
> detail:
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=390372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390372
]
ASF GitHub Bot logged work on BEAM-9056:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:36
Start Date: 21/Feb/20 01:36
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10621: [BEAM-9056]
Staging artifacts from environment
URL: https://github.com/apache/beam/pull/10621#discussion_r382354060
##
File path:
runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunner.java
##
@@ -203,11 +146,33 @@ public PipelineResult run(Pipeline pipeline) {
prepareJobResponse.getArtifactStagingEndpoint();
String stagingSessionToken = prepareJobResponse.getStagingSessionToken();
+ ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+ for (Map.Entry entry :
+ pipelineProto.getComponents().getEnvironmentsMap().entrySet()) {
+for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+ if
(BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getUrn()))
{
+RunnerApi.ArtifactFilePayload filePayload;
+try {
+ filePayload =
RunnerApi.ArtifactFilePayload.parseFrom(info.getPayload());
+} catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException("Error parsing artifact file
payload.", e);
+}
+filesToStageBuilder.add(
+StagedFile.of(new File(filePayload.getLocalPath()),
filePayload.getStagedName()));
+ } else {
+throw new RuntimeException(
+String.format("expect file artifact type but %s",
info.getUrn()));
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390372)
Time Spent: 1.5h (was: 1h 20m)
> Staging artifacts from environment
> --
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> staging artifacts from artifact information embedded in environment proto.
> detail:
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[ https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390369 ] ASF GitHub Bot logged work on BEAM-9022: Author: ASF GitHub Bot Created on: 21/Feb/20 01:27 Start Date: 21/Feb/20 01:27 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10921: [BEAM-9022] publish Spark job server Docker image URL: https://github.com/apache/beam/pull/10921#issuecomment-589454956 > > Yes! > > Great. In that case, shall we merge this first, and then I'll let you patch it into #10612? Yes, no problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390369) Time Spent: 1h 50m (was: 1h 40m) > Publish spark job server container images in release process > > > Key: BEAM-9022 > URL: https://issues.apache.org/jira/browse/BEAM-9022 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9348) Enable CrossLanguageValidateRunner test for direct runner
[ https://issues.apache.org/jira/browse/BEAM-9348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9348: -- Status: Open (was: Triage Needed) > Enable CrossLanguageValidateRunner test for direct runner > - > > Key: BEAM-9348 > URL: https://issues.apache.org/jira/browse/BEAM-9348 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Enable CrossLanguageValidateRunner test for direct runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9348) Enable CrossLanguageValidateRunner test for direct runner
Heejong Lee created BEAM-9348: - Summary: Enable CrossLanguageValidateRunner test for direct runner Key: BEAM-9348 URL: https://issues.apache.org/jira/browse/BEAM-9348 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee Enable CrossLanguageValidateRunner test for direct runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[ https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390358 ] ASF GitHub Bot logged work on BEAM-9022: Author: ASF GitHub Bot Created on: 21/Feb/20 01:10 Start Date: 21/Feb/20 01:10 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10921: [BEAM-9022] publish Spark job server Docker image URL: https://github.com/apache/beam/pull/10921#issuecomment-589450531 > Yes! Great. In that case, shall we merge this first, and then I'll let you patch it into #10612? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390358) Time Spent: 1h 40m (was: 1.5h) > Publish spark job server container images in release process > > > Key: BEAM-9022 > URL: https://issues.apache.org/jira/browse/BEAM-9022 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390356
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:07
Start Date: 21/Feb/20 01:07
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10921: [BEAM-9022]
publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382346713
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -236,6 +236,9 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
done
+ echo '-Generating and Pushing Spark job server
image-'
+ ./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
Minor clarification to the above: Beam Spark runner supports Spark's latest
_feature_ release, i.e. `2.4.*`.
I think the plan is to create a separate module when moving to Spark 3
([BEAM-9282](https://issues.apache.org/jira/browse/BEAM-9282)). @iemejia knows
more
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390356)
Time Spent: 1.5h (was: 1h 20m)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=390355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390355 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 21/Feb/20 01:06 Start Date: 21/Feb/20 01:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390355) Time Spent: 7.5h (was: 7h 20m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[ https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390351=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390351 ] ASF GitHub Bot logged work on BEAM-9022: Author: ASF GitHub Bot Created on: 21/Feb/20 01:02 Start Date: 21/Feb/20 01:02 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10921: [BEAM-9022] publish Spark job server Docker image URL: https://github.com/apache/beam/pull/10921#issuecomment-589448192 > @Hannah-Jiang are we expecting to migrate Apache's repo to starting with 2.20? Yes! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390351) Time Spent: 1h 20m (was: 1h 10m) > Publish spark job server container images in release process > > > Key: BEAM-9022 > URL: https://issues.apache.org/jira/browse/BEAM-9022 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=390352=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390352 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 21/Feb/20 01:02 Start Date: 21/Feb/20 01:02 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#issuecomment-589448241 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390352) Time Spent: 40m (was: 0.5h) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[ https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390350=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390350 ] ASF GitHub Bot logged work on BEAM-9022: Author: ASF GitHub Bot Created on: 21/Feb/20 01:00 Start Date: 21/Feb/20 01:00 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10921: [BEAM-9022] publish Spark job server Docker image URL: https://github.com/apache/beam/pull/10921#issuecomment-589447390 @Hannah-Jiang are we expecting to migrate Apache's repo to starting with 2.20? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390350) Time Spent: 1h 10m (was: 1h) > Publish spark job server container images in release process > > > Key: BEAM-9022 > URL: https://issues.apache.org/jira/browse/BEAM-9022 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390349
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 01:00
Start Date: 21/Feb/20 01:00
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #10921:
[BEAM-9022] publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382344887
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -236,6 +236,9 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
done
+ echo '-Generating and Pushing Spark job server
image-'
+ ./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
How about in future? In addition, I think users should know which spark
version is supported even if only one version is supported.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390349)
Time Spent: 1h (was: 50m)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390348
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 00:58
Start Date: 21/Feb/20 00:58
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10921: [BEAM-9022]
publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382343820
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -236,6 +236,9 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
done
+ echo '-Generating and Pushing Spark job server
image-'
+ ./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
No, the Spark runner only supports one Spark version per Beam release
(currently
[2.4.5](https://github.com/apache/beam/blob/800754399baa939232b9863b5104111c638e88aa/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L399)),
unlike Flink, which supports the latest three Flink versions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390348)
Time Spent: 50m (was: 40m)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390347
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 00:58
Start Date: 21/Feb/20 00:58
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10921: [BEAM-9022]
publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382343770
##
File path: website/src/contribute/release-guide.md
##
@@ -717,6 +717,12 @@ for ver in "${FLINK_VER[@]}"; do
done
```
+* Build Spark job server image and push to DockerHub.
+
+```
+./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
Whoops, good catch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390347)
Time Spent: 40m (was: 0.5h)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=390346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390346 ] ASF GitHub Bot logged work on BEAM-5605: Author: ASF GitHub Bot Created on: 21/Feb/20 00:58 Start Date: 21/Feb/20 00:58 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10920: [BEAM-5605] Eagerly close the BoundedReader once we have read everything or have failed. URL: https://github.com/apache/beam/pull/10920#issuecomment-589446799 R: @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390346) Time Spent: 17h 10m (was: 17h) > Support Portable SplittableDoFn for batch > - > > Key: BEAM-5605 > URL: https://issues.apache.org/jira/browse/BEAM-5605 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Assignee: Luke Cwik >Priority: Major > Labels: portability > Time Spent: 17h 10m > Remaining Estimate: 0h > > Roll-up item tracking work towards supporting portable SplittableDoFn for > batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390342
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 00:55
Start Date: 21/Feb/20 00:55
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #10921:
[BEAM-9022] publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382341099
##
File path: release/src/main/scripts/build_release_candidate.sh
##
@@ -236,6 +236,9 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
done
+ echo '-Generating and Pushing Spark job server
image-'
+ ./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
Don't we need to specify spark version here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390342)
Time Spent: 20m (was: 10m)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[
https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390344
]
ASF GitHub Bot logged work on BEAM-9022:
Author: ASF GitHub Bot
Created on: 21/Feb/20 00:55
Start Date: 21/Feb/20 00:55
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #10921:
[BEAM-9022] publish Spark job server Docker image
URL: https://github.com/apache/beam/pull/10921#discussion_r382341388
##
File path: website/src/contribute/release-guide.md
##
@@ -717,6 +717,12 @@ for ver in "${FLINK_VER[@]}"; do
done
```
+* Build Spark job server image and push to DockerHub.
+
+```
+./gradlew "./gradlew :runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
Review comment:
`./gradlew` is repeated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390344)
Time Spent: 0.5h (was: 20m)
> Publish spark job server container images in release process
>
>
> Key: BEAM-9022
> URL: https://issues.apache.org/jira/browse/BEAM-9022
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9022) Publish spark job server container images in release process
[ https://issues.apache.org/jira/browse/BEAM-9022?focusedWorklogId=390339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390339 ] ASF GitHub Bot logged work on BEAM-9022: Author: ASF GitHub Bot Created on: 21/Feb/20 00:43 Start Date: 21/Feb/20 00:43 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10921: [BEAM-9022] publish Spark job server Docker image URL: https://github.com/apache/beam/pull/10921 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=390338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390338 ] ASF GitHub Bot logged work on BEAM-5605: Author: ASF GitHub Bot Created on: 21/Feb/20 00:38 Start Date: 21/Feb/20 00:38 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10920: [BEAM-5605] Eagerly close the BoundedReader once we have read everything or have failed. URL: https://github.com/apache/beam/pull/10920#issuecomment-589441993 R: @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390338) Time Spent: 17h (was: 16h 50m) > Support Portable SplittableDoFn for batch > - > > Key: BEAM-5605 > URL: https://issues.apache.org/jira/browse/BEAM-5605 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Assignee: Luke Cwik >Priority: Major > Labels: portability > Time Spent: 17h > Remaining Estimate: 0h > > Roll-up item tracking work towards supporting portable SplittableDoFn for > batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=390337=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390337 ] ASF GitHub Bot logged work on BEAM-5605: Author: ASF GitHub Bot Created on: 21/Feb/20 00:37 Start Date: 21/Feb/20 00:37 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10920: [BEAM-5605] Eagerly close the BoundedReader once we have read everything or have failed. URL: https://github.com/apache/beam/pull/10920 If the bundle throws an exception for some other reason, the currentReader will go out of scope and will not be closed cleanly. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=390335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390335 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 21/Feb/20 00:30 Start Date: 21/Feb/20 00:30 Worklog Time Spent: 10m Work Description: ananvay commented on issue #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919#issuecomment-589440130 Thanks Ankur! LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390335) Time Spent: 0.5h (was: 20m) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=390333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390333 ] ASF GitHub Bot logged work on BEAM-5605: Author: ASF GitHub Bot Created on: 21/Feb/20 00:26 Start Date: 21/Feb/20 00:26 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10893: [BEAM-5605] Honor the bounded source timestamps timestamp in the SDF wrapper. URL: https://github.com/apache/beam/pull/10893 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390333) Time Spent: 16h 40m (was: 16.5h) > Support Portable SplittableDoFn for batch > - > > Key: BEAM-5605 > URL: https://issues.apache.org/jira/browse/BEAM-5605 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Assignee: Luke Cwik >Priority: Major > Labels: portability > Time Spent: 16h 40m > Remaining Estimate: 0h > > Roll-up item tracking work towards supporting portable SplittableDoFn for > batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?focusedWorklogId=390332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390332 ] ASF GitHub Bot logged work on BEAM-8019: Author: ASF GitHub Bot Created on: 21/Feb/20 00:16 Start Date: 21/Feb/20 00:16 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #10886: [BEAM-8019] Updates DataflowRunner to support multiple SDK environments. URL: https://github.com/apache/beam/pull/10886 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390332) Time Spent: 5h 20m (was: 5h 10m) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Goenka updated BEAM-9347: --- Description: The runner will choose the Runner Harness image for UW so we don't need to overwrite the image in default behavior. Also, this will help us distinguish between user requested overwrites for the default overwrites(which is not used). > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=390331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390331 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 21/Feb/20 00:06 Start Date: 21/Feb/20 00:06 Worklog Time Spent: 10m Work Description: angoenka commented on issue #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919#issuecomment-589433667 R: @ananvay @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390331) Time Spent: 20m (was: 10m) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The runner will choose the Runner Harness image for UW so we don't need to > overwrite the image in default behavior. > Also, this will help us distinguish between user requested overwrites for the > default overwrites(which is not used). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?focusedWorklogId=390330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390330 ] ASF GitHub Bot logged work on BEAM-9347: Author: ASF GitHub Bot Created on: 21/Feb/20 00:04 Start Date: 21/Feb/20 00:04 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10919: [BEAM-9347] Don't overwrite default runner harness for unified worker URL: https://github.com/apache/beam/pull/10919 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?focusedWorklogId=390328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390328 ] ASF GitHub Bot logged work on BEAM-9298: Author: ASF GitHub Bot Created on: 21/Feb/20 00:04 Start Date: 21/Feb/20 00:04 Worklog Time Spent: 10m Work Description: tweise commented on pull request #10884: [BEAM-9298] Drop support for Flink 1.7 URL: https://github.com/apache/beam/pull/10884#discussion_r382328806 ## File path: website/src/documentation/runners/flink.md ## @@ -103,7 +103,7 @@ To find out which version of Flink is compatible with Beam please see the table Artifact Id Review comment: We need a new row in this table that specifies the version support from 2.20 (or whatever version we release this change with) going forward. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390328) Time Spent: 20m (was: 10m) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9252) Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/BEAM-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041374#comment-17041374
]
Kenneth Knowles commented on BEAM-9252:
---
[~suztomo] or [~iemejia] would you have time to look at the module-info.class
and Main.class issues? I think BEAM-9288 is separate.
> Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
> ---
>
> Key: BEAM-9252
> URL: https://issues.apache.org/jira/browse/BEAM-9252
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.20.0
>Reporter: Ismaël Mejía
>Priority: Critical
> Fix For: 2.20.0
>
>
> I was checking today a pipeline against the latest 2.20.0-SNAPSHOT and I
> found that it works perfectly with version 2.19.0, but it is failing with a
> shade related exception that refers to grpc 1.26.0:
> {{[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project
> EventsToIOs: Error creating shaded jar: Problem shading JAR
> /home/ismael/.m2/repository/org/apache/beam/beam-vendor-grpc-1_26_0/0.1/beam-vendor-grpc-1_26_0-0.1.jar
> entry org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class:
> org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class
> org/apache/beam/vendor/grpc/v1p26p0/org/jboss/modules/Main.class: 65536 ->
> [Help 1]}}
> {{There is also a warning that is not present in the build against 2.19.0}}
> {{[WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.}}
>
> I wonder if we are not doing something wrong during our vendoring, can
> someone take a look please.
> This is relatively easy to reproduce with the beam-samples repo, just clone
> it and run:
> {noformat}
> git clone https://github.com/jbonofre/beam-samples
> mvn clean verify -Pbeam-release-repo -Dbeam.version=2.20.0-SNAPSHOT
> {noformat}
> Available logs of the latest run:
> [https://github.com/jbonofre/beam-samples/runs/427537544?check_suite_focus=true]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9240) Check for Nullability in typesEqual() method of FieldType class
[
https://issues.apache.org/jira/browse/BEAM-9240?focusedWorklogId=390323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390323
]
ASF GitHub Bot logged work on BEAM-9240:
Author: ASF GitHub Bot
Created on: 20/Feb/20 23:49
Start Date: 20/Feb/20 23:49
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #10744: [BEAM-9240]: Check
for Nullability in typesEqual() method of FieldTyp…
URL: https://github.com/apache/beam/pull/10744#issuecomment-589427096
can't seem to trigger tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390323)
Time Spent: 2h 20m (was: 2h 10m)
> Check for Nullability in typesEqual() method of FieldType class
> ---
>
> Key: BEAM-9240
> URL: https://issues.apache.org/jira/browse/BEAM-9240
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {{If two schemas are created like this:}}
> {{Schema schema1 = Schema.builder().addStringField("col1").build();}}
> {{Schema schema2 = Schema.builder().addNullableField("col1",
> FieldType.STRING).build();}}
>
> {{schema1.typeEquals(schema2) returns "true" even though the schemas differ
> by Nullability}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9240) Check for Nullability in typesEqual() method of FieldType class
[
https://issues.apache.org/jira/browse/BEAM-9240?focusedWorklogId=390321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390321
]
ASF GitHub Bot logged work on BEAM-9240:
Author: ASF GitHub Bot
Created on: 20/Feb/20 23:49
Start Date: 20/Feb/20 23:49
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #10744: [BEAM-9240]: Check
for Nullability in typesEqual() method of FieldTyp…
URL: https://github.com/apache/beam/pull/10744#issuecomment-589426520
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390321)
Time Spent: 2h 10m (was: 2h)
> Check for Nullability in typesEqual() method of FieldType class
> ---
>
> Key: BEAM-9240
> URL: https://issues.apache.org/jira/browse/BEAM-9240
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {{If two schemas are created like this:}}
> {{Schema schema1 = Schema.builder().addStringField("col1").build();}}
> {{Schema schema2 = Schema.builder().addNullableField("col1",
> FieldType.STRING).build();}}
>
> {{schema1.typeEquals(schema2) returns "true" even though the schemas differ
> by Nullability}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9288) Conscrypt shaded dependency
[ https://issues.apache.org/jira/browse/BEAM-9288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041366#comment-17041366 ] Kenneth Knowles commented on BEAM-9288: --- In the near term, I think we can exclude it from vendor/grpc. Then vendor/grpc will have a transitive dependency that users can be aware of, instead of relocated and bundled. It still can have diamond dependency problems, but at least they are exposed and "normal". > Conscrypt shaded dependency > --- > > Key: BEAM-9288 > URL: https://issues.apache.org/jira/browse/BEAM-9288 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Esun Kim >Assignee: sunjincheng >Priority: Critical > Fix For: 2.20.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Conscrypt is not designed to be shaded properly mainly because of so files. I > happened to see BEAM-9030 (*1) creating a new vendored gRPC shading Conscrypt > (*2) in it. I think this could make a problem when new Conscrypt is brought > by new gcsio depending on gRPC-alts (*4) in a dependency chain. (*5) In this > case, it may have a conflict when finding proper so files for Conscrypt. > *1: https://issues.apache.org/jira/browse/BEAM-9030 > *2: > [https://github.com/apache/beam/blob/e24d1e51cbabe27cb3cc381fd95b334db639c45d/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy#L78] > *3: https://issues.apache.org/jira/browse/BEAM-6136 > *4: [https://mvnrepository.com/artifact/io.grpc/grpc-alts/1.27.0] > *5: https://issues.apache.org/jira/browse/BEAM-8889 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8458) BigQueryIO.Read needs permissions to create datasets to be able to run queries
[
https://issues.apache.org/jira/browse/BEAM-8458?focusedWorklogId=390319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390319
]
ASF GitHub Bot logged work on BEAM-8458:
Author: ASF GitHub Bot
Created on: 20/Feb/20 23:44
Start Date: 20/Feb/20 23:44
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #9852: [BEAM-8458] Add option
to set temp dataset in BigQueryIO.Read
URL: https://github.com/apache/beam/pull/9852#issuecomment-589420201
R: @chamikaramj / @pabloem -- could you please take a look?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390319)
Time Spent: 3h (was: 2h 50m)
> BigQueryIO.Read needs permissions to create datasets to be able to run queries
> --
>
> Key: BEAM-8458
> URL: https://issues.apache.org/jira/browse/BEAM-8458
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Israel Herraiz
>Assignee: Israel Herraiz
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> When using {{fromQuery}}, BigQueryIO creates a temp dataset to store the
> results of the query.
> Therefore, Beam requires permissions to create datasets just to be able to
> run a query. In practice, this means that Beam requires the role
> bigQuery.User just to run queries, whereas if you use {{from}} (to read from
> a table), the role bigQuery.jobUser suffices.
> BigQueryIO.Read should have an option to set an existing dataset to write
> the temp results of
> a query, so it would be enough with having the role bigQuery.jobUser.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9240) Check for Nullability in typesEqual() method of FieldType class
[
https://issues.apache.org/jira/browse/BEAM-9240?focusedWorklogId=390317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390317
]
ASF GitHub Bot logged work on BEAM-9240:
Author: ASF GitHub Bot
Created on: 20/Feb/20 23:41
Start Date: 20/Feb/20 23:41
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #10744: [BEAM-9240]: Check
for Nullability in typesEqual() method of FieldTyp…
URL: https://github.com/apache/beam/pull/10744#issuecomment-589415336
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390317)
Time Spent: 1h 50m (was: 1h 40m)
> Check for Nullability in typesEqual() method of FieldType class
> ---
>
> Key: BEAM-9240
> URL: https://issues.apache.org/jira/browse/BEAM-9240
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {{If two schemas are created like this:}}
> {{Schema schema1 = Schema.builder().addStringField("col1").build();}}
> {{Schema schema2 = Schema.builder().addNullableField("col1",
> FieldType.STRING).build();}}
>
> {{schema1.typeEquals(schema2) returns "true" even though the schemas differ
> by Nullability}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9240) Check for Nullability in typesEqual() method of FieldType class
[
https://issues.apache.org/jira/browse/BEAM-9240?focusedWorklogId=390318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390318
]
ASF GitHub Bot logged work on BEAM-9240:
Author: ASF GitHub Bot
Created on: 20/Feb/20 23:41
Start Date: 20/Feb/20 23:41
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #10744: [BEAM-9240]: Check
for Nullability in typesEqual() method of FieldTyp…
URL: https://github.com/apache/beam/pull/10744#issuecomment-589415739
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390318)
Time Spent: 2h (was: 1h 50m)
> Check for Nullability in typesEqual() method of FieldType class
> ---
>
> Key: BEAM-9240
> URL: https://issues.apache.org/jira/browse/BEAM-9240
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.18.0
>Reporter: Rahul Patwari
>Assignee: Rahul Patwari
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {{If two schemas are created like this:}}
> {{Schema schema1 = Schema.builder().addStringField("col1").build();}}
> {{Schema schema2 = Schema.builder().addNullableField("col1",
> FieldType.STRING).build();}}
>
> {{schema1.typeEquals(schema2) returns "true" even though the schemas differ
> by Nullability}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[ https://issues.apache.org/jira/browse/BEAM-9343?focusedWorklogId=390316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390316 ] ASF GitHub Bot logged work on BEAM-9343: Author: ASF GitHub Bot Created on: 20/Feb/20 23:40 Start Date: 20/Feb/20 23:40 Worklog Time Spent: 10m Work Description: amaliujia commented on pull request #10918: [BEAM-9343]Upgrade ZetaSQL to 2020.02.1 URL: https://github.com/apache/beam/pull/10918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390316) Time Spent: 1h 40m (was: 1.5h) > Upgrade ZetaSQL to 2020.02.1 > > > Key: BEAM-9343 > URL: https://issues.apache.org/jira/browse/BEAM-9343 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Goenka updated BEAM-9347: --- Component/s: (was: testing) > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9347) Remove default image for Unified Worker
[ https://issues.apache.org/jira/browse/BEAM-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Goenka updated BEAM-9347: --- Fix Version/s: 2.20.0 > Remove default image for Unified Worker > --- > > Key: BEAM-9347 > URL: https://issues.apache.org/jira/browse/BEAM-9347 > Project: Beam > Issue Type: Test > Components: runner-dataflow, testing >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9347) Remove default image for Unified Worker
Ankur Goenka created BEAM-9347: -- Summary: Remove default image for Unified Worker Key: BEAM-9347 URL: https://issues.apache.org/jira/browse/BEAM-9347 Project: Beam Issue Type: Test Components: runner-dataflow, testing Reporter: Ankur Goenka -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9290) runner_harness_container_image experiment is not honored in python released sdks.
[
https://issues.apache.org/jira/browse/BEAM-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankur Goenka resolved BEAM-9290.
Resolution: Fixed
> runner_harness_container_image experiment is not honored in python released
> sdks.
> -
>
> Key: BEAM-9290
> URL: https://issues.apache.org/jira/browse/BEAM-9290
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
>
> {code:java}
> --experiments=runner_harness_container_image=foo_image{code}
> does not have any affect on the job.
>
>
> cc: [~tvalentyn]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9346) TFRecordIO inefficient read from sideinput causing pipeline to be slow
Ban Piao created BEAM-9346:
--
Summary: TFRecordIO inefficient read from sideinput causing
pipeline to be slow
Key: BEAM-9346
URL: https://issues.apache.org/jira/browse/BEAM-9346
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Ban Piao
In TFRecordIO, Reify.viewInGlobalWindow(input.apply(View.asList()),
ListCoder.of(resultCoder)) is an inefficient way of reading large set of side
input.
Pipeline can be sped up significantly by combinging the PCollection to
a single element PCollection>.
Sample code:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L412
from
```
return input
.getPipeline()
.apply(Reify.viewInGlobalWindow(input.apply(View.asList()),
ListCoder.of(resultCoder)));
```
to
```
return input.apply("ToList", Combine.globally(new ToListCombineFn<>()));
```
where ToListCombineFn is defined as
```
public static class ToListCombineFn extends CombineFn, List> {
@Override
public List createAccumulator() {
return new ArrayList<>();
}
@Override
public List addInput(List mutableAccumulator, ResultT
input) {
mutableAccumulator.add(input);
return mutableAccumulator;
}
@Override
public List mergeAccumulators(Iterable>
accumulators) {
Iterator> iter = accumulators.iterator();
if (!iter.hasNext()) {
return new ArrayList<>();
}
List merged = iter.next();
while (iter.hasNext()) {
merged.addAll(iter.next());
}
return merged;
}
@Override
public List extractOutput(List accumulator) {
return accumulator;
}
}
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390311 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 23:14 Start Date: 20/Feb/20 23:14 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#discussion_r382311690 ## File path: sdks/go/pkg/beam/core/runtime/graphx/translate.go ## @@ -26,6 +27,7 @@ import ( "github.com/apache/beam/sdks/go/pkg/beam/core/util/protox" "github.com/apache/beam/sdks/go/pkg/beam/internal/errors" pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1" + "github.com/apache/beam/sdks/go/pkg/beam/options/jobopts" Review comment: OK, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390311) Time Spent: 2.5h (was: 2h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9228) _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
[
https://issues.apache.org/jira/browse/BEAM-9228?focusedWorklogId=390308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390308
]
ASF GitHub Bot logged work on BEAM-9228:
Author: ASF GitHub Bot
Created on: 20/Feb/20 23:11
Start Date: 20/Feb/20 23:11
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #10847: [BEAM-9228]
Support further partition for FnApi ListBuffer
URL: https://github.com/apache/beam/pull/10847#issuecomment-589401291
@robertwb, please take a look.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390308)
Time Spent: 2h 40m (was: 2.5h)
> _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
>
>
> Key: BEAM-9228
> URL: https://issues.apache.org/jira/browse/BEAM-9228
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.16.0, 2.18.0, 2.19.0
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> A user reported following issue.
> -
> I have a set of tfrecord files, obtained by converting parquet files with
> Spark. Each file is roughly 1GB and I have 11 of those.
> I would expect simple statistics gathering (ie counting number of items of
> all files) to scale linearly with respect to the number of cores on my system.
> I am able to reproduce the issue with the minimal snippet below
> {code:java}
> import apache_beam as beam
> from apache_beam.options.pipeline_options import PipelineOptions
> from apache_beam.runners.portability import fn_api_runner
> from apache_beam.portability.api import beam_runner_api_pb2
> from apache_beam.portability import python_urns
> import sys
> pipeline_options = PipelineOptions(['--direct_num_workers', '4'])
> file_pattern = 'part-r-00*
> runner=fn_api_runner.FnApiRunner(
> default_environment=beam_runner_api_pb2.Environment(
> urn=python_urns.SUBPROCESS_SDK,
> payload=b'%s -m apache_beam.runners.worker.sdk_worker_main'
> % sys.executable.encode('ascii')))
> p = beam.Pipeline(runner=runner, options=pipeline_options)
> lines = (p | 'read' >> beam.io.tfrecordio.ReadFromTFRecord(file_pattern)
> | beam.combiners.Count.Globally()
> | beam.io.WriteToText('/tmp/output'))
> p.run()
> {code}
> Only one combination of apache_beam revision / worker type seems to work (I
> refer to https://beam.apache.org/documentation/runners/direct/ for the worker
> types)
> * beam 2.16; neither multithread nor multiprocess achieve high cpu usage on
> multiple cores
> * beam 2.17: able to achieve high cpu usage on all 4 cores
> * beam 2.18: not tested the mulithreaded mode but the multiprocess mode fails
> when trying to serialize the Environment instance most likely because of a
> change from 2.17 to 2.18.
> I also tried briefly SparkRunner with version 2.16 but was no able to achieve
> any throughput.
> What is the recommnended way to achieve what I am trying to ? How can I
> troubleshoot ?
> --
> This is caused by [this
> PR|https://github.com/apache/beam/commit/02f8ad4eee3ec0ea8cbdc0f99c1dad29f00a9f60].
> A [workaround|https://github.com/apache/beam/pull/10729] is tried, which is
> rolling back iobase.py not to use _SDFBoundedSourceWrapper. This confirmed
> that data is distributed to multiple workers, however, there are some
> regressions with SDF wrapper tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9345) "Multiple environments cannot be created in detached mode"
[ https://issues.apache.org/jira/browse/BEAM-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9345: -- Status: Open (was: Triage Needed) > "Multiple environments cannot be created in detached mode" > -- > > Key: BEAM-9345 > URL: https://issues.apache.org/jira/browse/BEAM-9345 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Maximilian Michels >Priority: Major > > Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when > running in standalone session cluster. Since FLINK-15504 is resolved in Flink > 1.10, maybe we can remove the workaround in Beam? > From user@ > (https://lists.apache.org/thread.html/r1fb9456055ae51f998be67531f0ac55d3da9cf2647238b991eee7f97%40%3Cuser.beam.apache.org%3E): > I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam > 2.16.0 to 2.19.0. > Everything went quite smoothly, the local runner and the local Flink runner > work flawlessly. > However when I: > 1. Generate a Beam jar for the FlinkRunner via maven (mvn package > -PFlinkRunner) > 2. Glue that into a Flink 1.9 docker image > 3. Start the image as a Standalone Session Cluster > When I try to launch the first pipeline I get the following exception > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: Failed to construct instance from factory method > FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at > org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: java.lang.RuntimeException: Failed to construct instance from > factory method FlinkRunner#fromOptions(interface > org.apache.beam.sdk.options.PipelineOptions) > at > org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) > at > org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) > at > org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) > at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 9 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) > ... 19 more > Caused by: org.apache.flink.api.common.InvalidProgramException: Multiple > environments cannot be created in detached mode > at > org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67) > at java.util.Optional.map(Optional.java:215) > at > org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068) > at > org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43) > at > org.apache.beam.runners.flink.FlinkRunner.(FlinkRunner.java:96) > at >
[jira] [Updated] (BEAM-9345) "Multiple environments cannot be created in detached mode"
[ https://issues.apache.org/jira/browse/BEAM-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9345: -- Description: Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when running in standalone session cluster. Since FLINK-15504 is resolved in Flink 1.10, maybe we can remove the workaround in Beam? >From user@ >(https://lists.apache.org/thread.html/r1fb9456055ae51f998be67531f0ac55d3da9cf2647238b991eee7f97%40%3Cuser.beam.apache.org%3E): I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 2.16.0 to 2.19.0. Everything went quite smoothly, the local runner and the local Flink runner work flawlessly. However when I: 1. Generate a Beam jar for the FlinkRunner via maven (mvn package -PFlinkRunner) 2. Glue that into a Flink 1.9 docker image 3. Start the image as a Standalone Session Cluster When I try to launch the first pipeline I get the following exception org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to construct instance from factory method FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) Caused by: java.lang.RuntimeException: Failed to construct instance from factory method FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) ... 9 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) ... 19 more Caused by: org.apache.flink.api.common.InvalidProgramException: Multiple environments cannot be created in detached mode at org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67) at java.util.Optional.map(Optional.java:215) at org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068) at org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43) at org.apache.beam.runners.flink.FlinkRunner.(FlinkRunner.java:96) at org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90) ... 24 more was: Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when running in standalone session cluster. Since FLINK-15504 is resolved in Flink 1.10, maybe we can remove the workaround in Beam? >From user@: I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 2.16.0 to 2.19.0. Everything went quite smoothly, the local runner and the local Flink runner work flawlessly. However when I: 1. Generate a Beam jar for the FlinkRunner via maven (mvn package -PFlinkRunner) 2.
[jira] [Created] (BEAM-9345) "Multiple environments cannot be created in detached mode"
Kyle Weaver created BEAM-9345: - Summary: "Multiple environments cannot be created in detached mode" Key: BEAM-9345 URL: https://issues.apache.org/jira/browse/BEAM-9345 Project: Beam Issue Type: Bug Components: runner-flink Reporter: Kyle Weaver Assignee: Maximilian Michels Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when running in standalone session cluster. Since FLINK-15504 is resolved in Flink 1.10, maybe we can remove the workaround in Beam? >From user@: I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 2.16.0 to 2.19.0. Everything went quite smoothly, the local runner and the local Flink runner work flawlessly. However when I: 1. Generate a Beam jar for the FlinkRunner via maven (mvn package -PFlinkRunner) 2. Glue that into a Flink 1.9 docker image 3. Start the image as a Standalone Session Cluster When I try to launch the first pipeline I get the following exception org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to construct instance from factory method FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) Caused by: java.lang.RuntimeException: Failed to construct instance from factory method FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) ... 9 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) ... 19 more Caused by: org.apache.flink.api.common.InvalidProgramException: Multiple environments cannot be created in detached mode at org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67) at java.util.Optional.map(Optional.java:215) at org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068) at org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43) at org.apache.beam.runners.flink.FlinkRunner.(FlinkRunner.java:96) at org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90) ... 24 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=390301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390301 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 20/Feb/20 22:37 Start Date: 20/Feb/20 22:37 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#issuecomment-589384551 This is needed for the unbounded source wrapper to be able to get an initial version working without watermark reporting. R: @Ardagan CC: @robertwb @boyuanzz @chamikaramj @ananvay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390301) Time Spent: 0.5h (was: 20m) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=390300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390300 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 20/Feb/20 22:37 Start Date: 20/Feb/20 22:37 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#issuecomment-589384551 This is needed for the unbounded source wrapper to be able to get an initial version working without watermark reporting. KafkaIO requires support for "checkpointing". R: @Ardagan CC: @robertwb @boyuanzz @chamikaramj @ananvay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390300) Time Spent: 20m (was: 10m) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?focusedWorklogId=390299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390299 ] ASF GitHub Bot logged work on BEAM-9344: Author: ASF GitHub Bot Created on: 20/Feb/20 22:31 Start Date: 20/Feb/20 22:31 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10910: [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. URL: https://github.com/apache/beam/pull/10910#issuecomment-589384551 This is needed for the unbounded source wrapper to be able to get an initial version working without watermark reporting. R: @Ardagan CC: @robertwb @boyuanzz @chamikaramj @ananvay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390299) Remaining Estimate: 0h Time Spent: 10m > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390291
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 20/Feb/20 22:16
Start Date: 20/Feb/20 22:16
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10911: [BEAM-9339]
Declare capabilities for Go SDK.
URL: https://github.com/apache/beam/pull/10911#discussion_r382286309
##
File path: sdks/go/pkg/beam/core/runtime/graphx/coder.go
##
@@ -50,6 +50,22 @@ const (
urnCoGBKList = "beam:go:coder:cogbklist:v1" // CoGBK representation.
Not a coder.
)
+func KnownStandardCoders() []string {
Review comment:
Since this lives in the same package as the caller, we can make the function
un-exported (lowercase k for knownStandardCoders() instead).
It's reasonably to have this function live here, due to locality with the
coder URNs
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390291)
Time Spent: 2h 20m (was: 2h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390292 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 22:16 Start Date: 20/Feb/20 22:16 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#discussion_r382287004 ## File path: sdks/go/pkg/beam/core/runtime/graphx/translate.go ## @@ -26,6 +27,7 @@ import ( "github.com/apache/beam/sdks/go/pkg/beam/core/util/protox" "github.com/apache/beam/sdks/go/pkg/beam/internal/errors" pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1" + "github.com/apache/beam/sdks/go/pkg/beam/options/jobopts" Review comment: I don't like that we're importing this package here. jobopts creates flags, which could be unexpected or unnecessary for certain runners. Could we instead just pass the environment urn in as a string? (the only thing we actually use from the package). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390292) Time Spent: 2h 20m (was: 2h 10m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=390284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390284 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 20/Feb/20 22:06 Start Date: 20/Feb/20 22:06 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10899: [BEAM-8335] Background Caching job URL: https://github.com/apache/beam/pull/10899#issuecomment-589364572 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390284) Time Spent: 70.5h (was: 70h 20m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 70.5h > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=390285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390285 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 20/Feb/20 22:06 Start Date: 20/Feb/20 22:06 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10899: [BEAM-8335] Background Caching job URL: https://github.com/apache/beam/pull/10899#issuecomment-589365338 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390285) Time Spent: 70h 40m (was: 70.5h) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 70h 40m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=390283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390283 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 20/Feb/20 22:04 Start Date: 20/Feb/20 22:04 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10911: [BEAM-9339] Declare capabilities for Go SDK. URL: https://github.com/apache/beam/pull/10911#issuecomment-589361208 OK, we do need this for dataflow. I refactored the code to be shared in translate.go. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390283) Time Spent: 2h 10m (was: 2h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=390276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390276 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 20/Feb/20 21:53 Start Date: 20/Feb/20 21:53 Worklog Time Spent: 10m Work Description: udim commented on pull request #10894: [BEAM-8280] Enable and improve IOTypeHints debug_str traceback URL: https://github.com/apache/beam/pull/10894 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390276) Time Spent: 6h 10m (was: 6h) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9230. --- Fix Version/s: 2.20.0 Resolution: Fixed > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-7961. --- Fix Version/s: 2.19.0 Resolution: Fixed > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.19.0 > > Time Spent: 23h 20m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9338) add postcommit XVR spark badge
[ https://issues.apache.org/jira/browse/BEAM-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9338. --- Fix Version/s: Not applicable Resolution: Fixed > add postcommit XVR spark badge > -- > > Key: BEAM-9338 > URL: https://issues.apache.org/jira/browse/BEAM-9338 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: Not applicable > > Time Spent: 0.5h > Remaining Estimate: 0h > > add postcommit xvr spark badges -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9341) postcommit xvr flink, spark failure
[ https://issues.apache.org/jira/browse/BEAM-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9341. --- Fix Version/s: Not applicable Resolution: Fixed > postcommit xvr flink, spark failure > --- > > Key: BEAM-9341 > URL: https://issues.apache.org/jira/browse/BEAM-9341 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > started from [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1738/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9344) Enable bundle finalization in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-9344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik updated BEAM-9344: Status: Open (was: Triage Needed) > Enable bundle finalization in Java SDK > -- > > Key: BEAM-9344 > URL: https://issues.apache.org/jira/browse/BEAM-9344 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core, sdk-java-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Major > > [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9344) Enable bundle finalization in Java SDK
Luke Cwik created BEAM-9344: --- Summary: Enable bundle finalization in Java SDK Key: BEAM-9344 URL: https://issues.apache.org/jira/browse/BEAM-9344 Project: Beam Issue Type: New Feature Components: sdk-java-core, sdk-java-harness Reporter: Luke Cwik Assignee: Luke Cwik [_https://s.apache.org/beam-finalizing-bundles_] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=390268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390268 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 20/Feb/20 21:32 Start Date: 20/Feb/20 21:32 Worklog Time Spent: 10m Work Description: ihji commented on issue #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733#issuecomment-589328702 @robertwb comments addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390268) Time Spent: 7h 20m (was: 7h 10m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9341) postcommit xvr flink, spark failure
[ https://issues.apache.org/jira/browse/BEAM-9341?focusedWorklogId=390267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390267 ] ASF GitHub Bot logged work on BEAM-9341: Author: ASF GitHub Bot Created on: 20/Feb/20 21:29 Start Date: 20/Feb/20 21:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #10912: [BEAM-9341] postcommit xvr flink fix URL: https://github.com/apache/beam/pull/10912 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390267) Time Spent: 1h 10m (was: 1h) > postcommit xvr flink, spark failure > --- > > Key: BEAM-9341 > URL: https://issues.apache.org/jira/browse/BEAM-9341 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > started from [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1738/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9335) update hard-coded coder id when translating Java external transforms
[ https://issues.apache.org/jira/browse/BEAM-9335?focusedWorklogId=390266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390266 ] ASF GitHub Bot logged work on BEAM-9335: Author: ASF GitHub Bot Created on: 20/Feb/20 21:25 Start Date: 20/Feb/20 21:25 Worklog Time Spent: 10m Work Description: ihji commented on issue #10900: [BEAM-9335] update hard-coded coder id when translating Java external transforms URL: https://github.com/apache/beam/pull/10900#issuecomment-589326145 @robertwb Namespace prefix only works for the components which are originally created in the expansion service. The problem is that the components passed from the expansion request could be used in expanded components. For example, when input PCollection to the expansion service uses the same coder as any intermediate PCollection in the expanded subtransforms, the coder id specified in the input PCollection will be hard-coded in the expanded intermediate PCollection. That hard-coded coder id could be changed anytime later in the new proto translation and resulted in a wrong coder. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390266) Time Spent: 0.5h (was: 20m) > update hard-coded coder id when translating Java external transforms > > > Key: BEAM-9335 > URL: https://issues.apache.org/jira/browse/BEAM-9335 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > hard-coded coder id needs to be updated when translating Java external > transforms. Otherwise pipeline will fail if coder id is reused. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9335) update hard-coded coder id when translating Java external transforms
[ https://issues.apache.org/jira/browse/BEAM-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041300#comment-17041300 ] Robert Bradshaw commented on BEAM-9335: --- Is this simply to avoid collisions between coder ids between SDKs? > update hard-coded coder id when translating Java external transforms > > > Key: BEAM-9335 > URL: https://issues.apache.org/jira/browse/BEAM-9335 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > hard-coded coder id needs to be updated when translating Java external > transforms. Otherwise pipeline will fail if coder id is reused. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7870) Externally configured KafkaIO / PubsubIO consumer causes coder problems
[ https://issues.apache.org/jira/browse/BEAM-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041299#comment-17041299 ] Brian Hulette commented on BEAM-7870: - Whoops I meant NamedTuple, not TypedDict. > Externally configured KafkaIO / PubsubIO consumer causes coder problems > --- > > Key: BEAM-7870 > URL: https://issues.apache.org/jira/browse/BEAM-7870 > Project: Beam > Issue Type: Bug > Components: runner-flink, sdk-java-core >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > > There are limitations for the consumer to work correctly. The biggest issue > is the structure of KafkaIO itself, which uses a combination of the source > interface and DoFns to generate the desired output. The problem is that the > source interface is natively translated by the Flink Runner to support > unbounded sources in portability, while the DoFn runs in a Java environment. > To transfer data between the two a coder needs to be involved. It happens to > be that the initial read does not immediately drop the KafakRecord structure > which does not work together well with our current assumption of only > supporting "standard coders" present in all SDKs. Only the subsequent DoFn > converts the KafkaRecord structure into a raw KV[byte, byte], but the DoFn > won't have the coder available in its environment. > There are several possible solutions: > 1. Make the DoFn which drops the KafkaRecordCoder a native Java transform in > the Flink Runner > 2. Modify KafkaIO to immediately drop the KafkaRecord structure > 3. Add the KafkaRecordCoder to all SDKs > 4. Add a generic coder, e.g. AvroCoder to all SDKs > For a workaround which uses (3), please see this patch which is not a proper > fix but adds KafkaRecordCoder to the SDK such that it can be used > encode/decode records: > [https://github.com/mxm/beam/commit/b31cf99c75b3972018180d8ccc7e73d311f4cfed] > > See also > [https://github.com/apache/beam/pull/8251|https://github.com/apache/beam/pull/8251:] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=390263=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390263 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 20/Feb/20 20:57 Start Date: 20/Feb/20 20:57 Worklog Time Spent: 10m Work Description: KevinGG commented on issue #10899: [BEAM-8335] Background Caching job URL: https://github.com/apache/beam/pull/10899#issuecomment-589304823 > Can't seem to trigger the tests :/ Thanks! Yeah, the test triggering is so annoying :( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390263) Time Spent: 70h 20m (was: 70h 10m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 70h 20m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[ https://issues.apache.org/jira/browse/BEAM-9343?focusedWorklogId=390262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390262 ] ASF GitHub Bot logged work on BEAM-9343: Author: ASF GitHub Bot Created on: 20/Feb/20 20:55 Start Date: 20/Feb/20 20:55 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10918: [BEAM-9343]Upgrade ZetaSQL to 2020.02.1 URL: https://github.com/apache/beam/pull/10918#issuecomment-589285981 R: @apilloud cc: @robinyqiu Have verified by running local tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390262) Time Spent: 1.5h (was: 1h 20m) > Upgrade ZetaSQL to 2020.02.1 > > > Key: BEAM-9343 > URL: https://issues.apache.org/jira/browse/BEAM-9343 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9228) _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
[
https://issues.apache.org/jira/browse/BEAM-9228?focusedWorklogId=390261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390261
]
ASF GitHub Bot logged work on BEAM-9228:
Author: ASF GitHub Bot
Created on: 20/Feb/20 20:54
Start Date: 20/Feb/20 20:54
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #10847: [BEAM-9228]
Support further partition for FnApi ListBuffer
URL: https://github.com/apache/beam/pull/10847#issuecomment-589303274
retest it please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390261)
Time Spent: 2.5h (was: 2h 20m)
> _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
>
>
> Key: BEAM-9228
> URL: https://issues.apache.org/jira/browse/BEAM-9228
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.16.0, 2.18.0, 2.19.0
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> A user reported following issue.
> -
> I have a set of tfrecord files, obtained by converting parquet files with
> Spark. Each file is roughly 1GB and I have 11 of those.
> I would expect simple statistics gathering (ie counting number of items of
> all files) to scale linearly with respect to the number of cores on my system.
> I am able to reproduce the issue with the minimal snippet below
> {code:java}
> import apache_beam as beam
> from apache_beam.options.pipeline_options import PipelineOptions
> from apache_beam.runners.portability import fn_api_runner
> from apache_beam.portability.api import beam_runner_api_pb2
> from apache_beam.portability import python_urns
> import sys
> pipeline_options = PipelineOptions(['--direct_num_workers', '4'])
> file_pattern = 'part-r-00*
> runner=fn_api_runner.FnApiRunner(
> default_environment=beam_runner_api_pb2.Environment(
> urn=python_urns.SUBPROCESS_SDK,
> payload=b'%s -m apache_beam.runners.worker.sdk_worker_main'
> % sys.executable.encode('ascii')))
> p = beam.Pipeline(runner=runner, options=pipeline_options)
> lines = (p | 'read' >> beam.io.tfrecordio.ReadFromTFRecord(file_pattern)
> | beam.combiners.Count.Globally()
> | beam.io.WriteToText('/tmp/output'))
> p.run()
> {code}
> Only one combination of apache_beam revision / worker type seems to work (I
> refer to https://beam.apache.org/documentation/runners/direct/ for the worker
> types)
> * beam 2.16; neither multithread nor multiprocess achieve high cpu usage on
> multiple cores
> * beam 2.17: able to achieve high cpu usage on all 4 cores
> * beam 2.18: not tested the mulithreaded mode but the multiprocess mode fails
> when trying to serialize the Environment instance most likely because of a
> change from 2.17 to 2.18.
> I also tried briefly SparkRunner with version 2.16 but was no able to achieve
> any throughput.
> What is the recommnended way to achieve what I am trying to ? How can I
> troubleshoot ?
> --
> This is caused by [this
> PR|https://github.com/apache/beam/commit/02f8ad4eee3ec0ea8cbdc0f99c1dad29f00a9f60].
> A [workaround|https://github.com/apache/beam/pull/10729] is tried, which is
> rolling back iobase.py not to use _SDFBoundedSourceWrapper. This confirmed
> that data is distributed to multiple workers, however, there are some
> regressions with SDF wrapper tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[ https://issues.apache.org/jira/browse/BEAM-9343?focusedWorklogId=390259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390259 ] ASF GitHub Bot logged work on BEAM-9343: Author: ASF GitHub Bot Created on: 20/Feb/20 20:52 Start Date: 20/Feb/20 20:52 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10918: [BEAM-9343]Upgrade ZetaSQL to 2020.02.1 URL: https://github.com/apache/beam/pull/10918#issuecomment-589302546 Run SQL Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390259) Time Spent: 1h 20m (was: 1h 10m) > Upgrade ZetaSQL to 2020.02.1 > > > Key: BEAM-9343 > URL: https://issues.apache.org/jira/browse/BEAM-9343 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9286) Create validation tests for metrics based on MonitoringInfo if applicable
[ https://issues.apache.org/jira/browse/BEAM-9286?focusedWorklogId=390255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390255 ] ASF GitHub Bot logged work on BEAM-9286: Author: ASF GitHub Bot Created on: 20/Feb/20 20:50 Start Date: 20/Feb/20 20:50 Worklog Time Spent: 10m Work Description: yifanzou commented on pull request #10823: [BEAM-9286] Create validation runner test for metrics (user counter). URL: https://github.com/apache/beam/pull/10823 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390255) Time Spent: 3.5h (was: 3h 20m) > Create validation tests for metrics based on MonitoringInfo if applicable > - > > Key: BEAM-9286 > URL: https://issues.apache.org/jira/browse/BEAM-9286 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Ruoyun Huang >Assignee: Ruoyun Huang >Priority: Minor > Time Spent: 3.5h > Remaining Estimate: 0h > > Create dedicated validation runner tests for metrics (those based Monitoring > Info). > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7304) Twister2 Beam runner
[ https://issues.apache.org/jira/browse/BEAM-7304?focusedWorklogId=390252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390252 ] ASF GitHub Bot logged work on BEAM-7304: Author: ASF GitHub Bot Created on: 20/Feb/20 20:45 Start Date: 20/Feb/20 20:45 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10888: [BEAM-7304] Twister2 Beam runner URL: https://github.com/apache/beam/pull/10888#issuecomment-589299713 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390252) Time Spent: 6h (was: 5h 50m) > Twister2 Beam runner > > > Key: BEAM-7304 > URL: https://issues.apache.org/jira/browse/BEAM-7304 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Pulasthi Wickramasinghe >Assignee: Pulasthi Wickramasinghe >Priority: Minor > Time Spent: 6h > Remaining Estimate: 0h > > Twister2 is a big data framework which supports both batch and stream > processing [1] [2]. The goal is to develop an beam runner for Twister2. > [1] [https://github.com/DSC-SPIDAL/twister2] > [2] [https://twister2.org/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7304) Twister2 Beam runner
[ https://issues.apache.org/jira/browse/BEAM-7304?focusedWorklogId=390253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390253 ] ASF GitHub Bot logged work on BEAM-7304: Author: ASF GitHub Bot Created on: 20/Feb/20 20:45 Start Date: 20/Feb/20 20:45 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10888: [BEAM-7304] Twister2 Beam runner URL: https://github.com/apache/beam/pull/10888#issuecomment-589299713 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390253) Time Spent: 6h 10m (was: 6h) > Twister2 Beam runner > > > Key: BEAM-7304 > URL: https://issues.apache.org/jira/browse/BEAM-7304 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Pulasthi Wickramasinghe >Assignee: Pulasthi Wickramasinghe >Priority: Minor > Time Spent: 6h 10m > Remaining Estimate: 0h > > Twister2 is a big data framework which supports both batch and stream > processing [1] [2]. The goal is to develop an beam runner for Twister2. > [1] [https://github.com/DSC-SPIDAL/twister2] > [2] [https://twister2.org/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[
https://issues.apache.org/jira/browse/BEAM-9343?focusedWorklogId=390251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390251
]
ASF GitHub Bot logged work on BEAM-9343:
Author: ASF GitHub Bot
Created on: 20/Feb/20 20:41
Start Date: 20/Feb/20 20:41
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #10918:
[BEAM-9343]Upgrade ZetaSQL to 2020.02.1
URL: https://github.com/apache/beam/pull/10918#discussion_r382245781
##
File path: sdks/java/extensions/sql/zetasql/build.gradle
##
@@ -20,12 +20,18 @@ plugins {
id 'org.apache.beam.module'
}
+repositories {
+ maven {
+url "https://oss.sonatype.org/content/repositories/comgooglezetasql-1011;
Review comment:
You need to remove this block before merging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390251)
Time Spent: 1h 10m (was: 1h)
> Upgrade ZetaSQL to 2020.02.1
>
>
> Key: BEAM-9343
> URL: https://issues.apache.org/jira/browse/BEAM-9343
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[ https://issues.apache.org/jira/browse/BEAM-9343?focusedWorklogId=390249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390249 ] ASF GitHub Bot logged work on BEAM-9343: Author: ASF GitHub Bot Created on: 20/Feb/20 20:37 Start Date: 20/Feb/20 20:37 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10918: [BEAM-9343]Upgrade ZetaSQL to 2020.02.1 URL: https://github.com/apache/beam/pull/10918#issuecomment-589285981 R: @apilloud Have verified by running local tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390249) Time Spent: 1h (was: 50m) > Upgrade ZetaSQL to 2020.02.1 > > > Key: BEAM-9343 > URL: https://issues.apache.org/jira/browse/BEAM-9343 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[ https://issues.apache.org/jira/browse/BEAM-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9343 started by Rui Wang. -- > Upgrade ZetaSQL to 2020.02.1 > > > Key: BEAM-9343 > URL: https://issues.apache.org/jira/browse/BEAM-9343 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9343) Upgrade ZetaSQL to 2020.02.1
[ https://issues.apache.org/jira/browse/BEAM-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated BEAM-9343: --- Status: Open (was: Triage Needed) > Upgrade ZetaSQL to 2020.02.1 > > > Key: BEAM-9343 > URL: https://issues.apache.org/jira/browse/BEAM-9343 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9331) The Row object needs better builders
[
https://issues.apache.org/jira/browse/BEAM-9331?focusedWorklogId=390247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390247
]
ASF GitHub Bot logged work on BEAM-9331:
Author: ASF GitHub Bot
Created on: 20/Feb/20 20:32
Start Date: 20/Feb/20 20:32
Worklog Time Spent: 10m
Work Description: alexvanboxel commented on pull request #10883:
[BEAM-9331] Add better Row builders
URL: https://github.com/apache/beam/pull/10883#discussion_r382241904
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
##
@@ -477,6 +477,172 @@ public void testCreateMapWithRowValue() {
assertEquals(data, row.getMap("map"));
}
+ @Test
Review comment:
What about this,
- `addValue`, `getValue` and `getValues` return the input type(s) (so the
logical type)
- `add`, `get`, `detach` returns the base types (is also inline with the
`attach`)
That means a behaviour change only for `getValue` and `getValues`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390247)
Time Spent: 2h 50m (was: 2h 40m)
> The Row object needs better builders
>
>
> Key: BEAM-9331
> URL: https://issues.apache.org/jira/browse/BEAM-9331
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Users should be able to build a Row object by specifying field names. Desired
> syntax:
>
> Row.withSchema(schema)
> .withFieldName("field1", "value)
> .withFieldName("field2.field3", value)
> .build()
>
> Users should also have a builder that allows taking an existing row and
> changing specific fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=390246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390246 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 20/Feb/20 20:25 Start Date: 20/Feb/20 20:25 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10899: [BEAM-8335] Background Caching job URL: https://github.com/apache/beam/pull/10899#issuecomment-589291573 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390246) Time Spent: 70h 10m (was: 70h) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 70h 10m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7304) Twister2 Beam runner
[ https://issues.apache.org/jira/browse/BEAM-7304?focusedWorklogId=390241=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390241 ] ASF GitHub Bot logged work on BEAM-7304: Author: ASF GitHub Bot Created on: 20/Feb/20 20:16 Start Date: 20/Feb/20 20:16 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10888: [BEAM-7304] Twister2 Beam runner URL: https://github.com/apache/beam/pull/10888#issuecomment-589287142 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390241) Time Spent: 5h 40m (was: 5.5h) > Twister2 Beam runner > > > Key: BEAM-7304 > URL: https://issues.apache.org/jira/browse/BEAM-7304 > Project: Beam > Issue Type: New Feature > Components: runner-ideas >Reporter: Pulasthi Wickramasinghe >Assignee: Pulasthi Wickramasinghe >Priority: Minor > Time Spent: 5h 40m > Remaining Estimate: 0h > > Twister2 is a big data framework which supports both batch and stream > processing [1] [2]. The goal is to develop an beam runner for Twister2. > [1] [https://github.com/DSC-SPIDAL/twister2] > [2] [https://twister2.org/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
