[jira] [Created] (BEAM-10218) Add FnApiRunner to cross-language validate runner test suite
Heejong Lee created BEAM-10218: -- Summary: Add FnApiRunner to cross-language validate runner test suite Key: BEAM-10218 URL: https://issues.apache.org/jira/browse/BEAM-10218 Project: Beam Issue Type: Improvement Components: cross-language, runner-direct Reporter: Heejong Lee Assignee: Heejong Lee Add FnApiRunner to cross-language validate runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10208) add cross-language KafkaIO integration test
Heejong Lee created BEAM-10208: -- Summary: add cross-language KafkaIO integration test Key: BEAM-10208 URL: https://issues.apache.org/jira/browse/BEAM-10208 Project: Beam Issue Type: Improvement Components: cross-language, testing Reporter: Heejong Lee add cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10208) add cross-language KafkaIO integration test
[ https://issues.apache.org/jira/browse/BEAM-10208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10208: --- Status: Open (was: Triage Needed) > add cross-language KafkaIO integration test > --- > > Key: BEAM-10208 > URL: https://issues.apache.org/jira/browse/BEAM-10208 > Project: Beam > Issue Type: Improvement > Components: cross-language, testing >Reporter: Heejong Lee >Priority: P2 > > add cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10208) add cross-language KafkaIO integration test
[ https://issues.apache.org/jira/browse/BEAM-10208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-10208: -- Assignee: Heejong Lee > add cross-language KafkaIO integration test > --- > > Key: BEAM-10208 > URL: https://issues.apache.org/jira/browse/BEAM-10208 > Project: Beam > Issue Type: Improvement > Components: cross-language, testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > add cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9869) adding self-contained Kafka service jar for testing
[ https://issues.apache.org/jira/browse/BEAM-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9869. --- Fix Version/s: 2.23.0 Resolution: Fixed > adding self-contained Kafka service jar for testing > --- > > Key: BEAM-9869 > URL: https://issues.apache.org/jira/browse/BEAM-9869 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Labels: stale-assigned > Fix For: 2.23.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-10052) check hash and avoid duplicates when uploading artifact in Python Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-10052. Resolution: Fixed > check hash and avoid duplicates when uploading artifact in Python Dataflow > Runner > - > > Key: BEAM-10052 > URL: https://issues.apache.org/jira/browse/BEAM-10052 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > xlang pipeline could have many duplicated jars. it would be great if we check > hash and avoid duplicate uploads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-10125) adding cross-language KafkaIO integration test
[ https://issues.apache.org/jira/browse/BEAM-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-10125. Fix Version/s: 2.23.0 Resolution: Fixed > adding cross-language KafkaIO integration test > -- > > Key: BEAM-10125 > URL: https://issues.apache.org/jira/browse/BEAM-10125 > Project: Beam > Issue Type: Improvement > Components: cross-language, io-java-kafka >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.23.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > adding cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10052) check hash and avoid duplicates when uploading artifact in Python Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10052: --- Fix Version/s: 2.22.0 > check hash and avoid duplicates when uploading artifact in Python Dataflow > Runner > - > > Key: BEAM-10052 > URL: https://issues.apache.org/jira/browse/BEAM-10052 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > xlang pipeline could have many duplicated jars. it would be great if we check > hash and avoid duplicate uploads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10202) make run_job_server.sh, run_expansion_service.sh OSX-compatible
Heejong Lee created BEAM-10202: -- Summary: make run_job_server.sh, run_expansion_service.sh OSX-compatible Key: BEAM-10202 URL: https://issues.apache.org/jira/browse/BEAM-10202 Project: Beam Issue Type: Improvement Components: cross-language, testing Reporter: Heejong Lee Assignee: Heejong Lee make run_job_server.sh, run_expansion_service.sh compatible with bash 3.2 and add check for flock command line utility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10125) adding cross-language KafkaIO integration test
Heejong Lee created BEAM-10125: -- Summary: adding cross-language KafkaIO integration test Key: BEAM-10125 URL: https://issues.apache.org/jira/browse/BEAM-10125 Project: Beam Issue Type: Improvement Components: io-java-kafka Reporter: Heejong Lee adding cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10125) adding cross-language KafkaIO integration test
[ https://issues.apache.org/jira/browse/BEAM-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-10125: -- Assignee: Heejong Lee > adding cross-language KafkaIO integration test > -- > > Key: BEAM-10125 > URL: https://issues.apache.org/jira/browse/BEAM-10125 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > adding cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10125) adding cross-language KafkaIO integration test
[ https://issues.apache.org/jira/browse/BEAM-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10125: --- Status: Open (was: Triage Needed) > adding cross-language KafkaIO integration test > -- > > Key: BEAM-10125 > URL: https://issues.apache.org/jira/browse/BEAM-10125 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Heejong Lee >Priority: P2 > > adding cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-10077) using filename + hash instead of UUID for staging name
[ https://issues.apache.org/jira/browse/BEAM-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-10077. Resolution: Fixed > using filename + hash instead of UUID for staging name > -- > > Key: BEAM-10077 > URL: https://issues.apache.org/jira/browse/BEAM-10077 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recent change BEAM-9383 disabled the artifact caching logic for GCS by object > names. Changing staging name generation from UUID to filename + hash will > re-enable the artifact caching so we can avoid re-uploading same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) uniquify Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10078: --- Description: After BEAM-9383, Dataflow specific jars (dataflow-worker.jar, windmill_main) could be overwritten when two or more jobs share the same staging location. Since they 1) should have specific predefined names AND 2) should have unique location for avoiding collision, they need special handling when staging artifacts. (was: After BEAM-Dataflow specific jars (dataflow-worker.jar, windmill_main) could be overwritten when two or more jobs share the same staging location. Since they 1) should have specific predefined names AND 2) should have unique location for avoiding collision, they need special handling when staging artifacts.) > uniquify Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > > After BEAM-9383, Dataflow specific jars (dataflow-worker.jar, windmill_main) > could be overwritten when two or more jobs share the same staging location. > Since they 1) should have specific predefined names AND 2) should have unique > location for avoiding collision, they need special handling when staging > artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) uniquify Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10078: --- Description: After BEAM-Dataflow specific jars (dataflow-worker.jar, windmill_main) could be overwritten when two or more jobs share the same staging location. Since they 1) should have specific predefined names AND 2) should have unique location for avoiding collision, they need special handling when staging artifacts. (was: Dataflow specific jars (dataflow-worker.jar, windmill_main) need special handling when staging.) > uniquify Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > > After BEAM-Dataflow specific jars (dataflow-worker.jar, windmill_main) could > be overwritten when two or more jobs share the same staging location. Since > they 1) should have specific predefined names AND 2) should have unique > location for avoiding collision, they need special handling when staging > artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10077) using filename + hash instead of UUID for staging name
[ https://issues.apache.org/jira/browse/BEAM-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10077: --- Description: Recent change BEAM-9383 disabled the artifact caching logic for GCS by object names. Changing staging name generation from UUID to filename + hash will re-enable the artifact caching so we can avoid re-uploading same artifact. (was: change staging name from uuid to filename + hash so we can avoid re-uploading same artifact.) > using filename + hash instead of UUID for staging name > -- > > Key: BEAM-10077 > URL: https://issues.apache.org/jira/browse/BEAM-10077 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > > Recent change BEAM-9383 disabled the artifact caching logic for GCS by object > names. Changing staging name generation from UUID to filename + hash will > re-enable the artifact caching so we can avoid re-uploading same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) uniquify Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10078: --- Fix Version/s: 2.22.0 > uniquify Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > > Dataflow specific jars (dataflow-worker.jar, windmill_main) need special > handling when staging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) uniquify Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10078: --- Summary: uniquify Dataflow specific jars when staging (was: labeling Dataflow specific jars when staging) > uniquify Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Dataflow specific jars (dataflow-worker.jar, windmill_main) need special > handling when staging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10078) labeling Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-10078: -- Assignee: Heejong Lee > labeling Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Dataflow specific jars (dataflow-worker.jar, windmill_main) need special > handling when staging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10078) labeling Dataflow specific jars when staging
Heejong Lee created BEAM-10078: -- Summary: labeling Dataflow specific jars when staging Key: BEAM-10078 URL: https://issues.apache.org/jira/browse/BEAM-10078 Project: Beam Issue Type: Bug Components: runner-dataflow Reporter: Heejong Lee Dataflow specific jars (dataflow-worker.jar, windmill_main) need special handling when staging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) labeling Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10078: --- Status: Open (was: Triage Needed) > labeling Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Dataflow specific jars (dataflow-worker.jar, windmill_main) need special > handling when staging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10077) using filename + hash instead of UUID for staging name
[ https://issues.apache.org/jira/browse/BEAM-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10077: --- Fix Version/s: 2.22.0 > using filename + hash instead of UUID for staging name > -- > > Key: BEAM-10077 > URL: https://issues.apache.org/jira/browse/BEAM-10077 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > > change staging name from uuid to filename + hash so we can avoid re-uploading > same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10077) using filename + hash instead of UUID for staging name
Heejong Lee created BEAM-10077: -- Summary: using filename + hash instead of UUID for staging name Key: BEAM-10077 URL: https://issues.apache.org/jira/browse/BEAM-10077 Project: Beam Issue Type: Bug Components: runner-dataflow Reporter: Heejong Lee Assignee: Heejong Lee change staging name from uuid to filename + hash so we can avoid re-uploading same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10064) Fix google3 import error for BEAM-9383
[ https://issues.apache.org/jira/browse/BEAM-10064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-10064: -- Assignee: Heejong Lee > Fix google3 import error for BEAM-9383 > -- > > Key: BEAM-10064 > URL: https://issues.apache.org/jira/browse/BEAM-10064 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Fix google3 importing error for BEAM-9383 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10064) Fix google3 import error for BEAM-9383
Heejong Lee created BEAM-10064: -- Summary: Fix google3 import error for BEAM-9383 Key: BEAM-10064 URL: https://issues.apache.org/jira/browse/BEAM-10064 Project: Beam Issue Type: Bug Components: test-failures Reporter: Heejong Lee Fix google3 importing error for BEAM-9383 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10064) Fix google3 import error for BEAM-9383
[ https://issues.apache.org/jira/browse/BEAM-10064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10064: --- Status: Open (was: Triage Needed) > Fix google3 import error for BEAM-9383 > -- > > Key: BEAM-10064 > URL: https://issues.apache.org/jira/browse/BEAM-10064 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Heejong Lee >Priority: P2 > > Fix google3 importing error for BEAM-9383 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112909#comment-17112909 ] Heejong Lee commented on BEAM-9383: --- [https://github.com/apache/beam/pull/11771] could mitigate the problem by removing duplicated artifacts from multiple environments. > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10052) check hash and avoid duplicates when uploading artifact in Python Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-10052: --- Status: Open (was: Triage Needed) > check hash and avoid duplicates when uploading artifact in Python Dataflow > Runner > - > > Key: BEAM-10052 > URL: https://issues.apache.org/jira/browse/BEAM-10052 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Priority: P2 > > xlang pipeline could have many duplicated jars. it would be great if we check > hash and avoid duplicate uploads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10052) check hash and avoid duplicates when uploading artifact in Python Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-10052: -- Assignee: Heejong Lee > check hash and avoid duplicates when uploading artifact in Python Dataflow > Runner > - > > Key: BEAM-10052 > URL: https://issues.apache.org/jira/browse/BEAM-10052 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > xlang pipeline could have many duplicated jars. it would be great if we check > hash and avoid duplicate uploads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10052) check hash and avoid duplicates when uploading artifact in Python Dataflow Runner
Heejong Lee created BEAM-10052: -- Summary: check hash and avoid duplicates when uploading artifact in Python Dataflow Runner Key: BEAM-10052 URL: https://issues.apache.org/jira/browse/BEAM-10052 Project: Beam Issue Type: Improvement Components: runner-dataflow Reporter: Heejong Lee xlang pipeline could have many duplicated jars. it would be great if we check hash and avoid duplicate uploads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9383. --- Resolution: Fixed > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable
[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102087#comment-17102087 ] Heejong Lee commented on BEAM-4379: --- This ticket is listed for Google Apache intern project. Temporarily assigned to me and will be reassigned to the intern. > Make ParquetIO Read splittable > -- > > Key: BEAM-4379 > URL: https://issues.apache.org/jira/browse/BEAM-4379 > Project: Beam > Issue Type: Improvement > Components: io-ideas, io-java-parquet >Reporter: Lukasz Gajowy >Priority: Major > > As the title stands - currently it is not splittable which is not optimal for > runners that support splitting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-4379) Make ParquetIO Read splittable
[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-4379: - Assignee: Heejong Lee > Make ParquetIO Read splittable > -- > > Key: BEAM-4379 > URL: https://issues.apache.org/jira/browse/BEAM-4379 > Project: Beam > Issue Type: Improvement > Components: io-ideas, io-java-parquet >Reporter: Lukasz Gajowy >Assignee: Heejong Lee >Priority: Major > > As the title stands - currently it is not splittable which is not optimal for > runners that support splitting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7925) ParquetIO supports neither column projection nor filter predicate
[
https://issues.apache.org/jira/browse/BEAM-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heejong Lee reassigned BEAM-7925:
-
Assignee: Heejong Lee
> ParquetIO supports neither column projection nor filter predicate
> -
>
> Key: BEAM-7925
> URL: https://issues.apache.org/jira/browse/BEAM-7925
> Project: Beam
> Issue Type: Improvement
> Components: io-java-parquet
>Affects Versions: 2.14.0
>Reporter: Neville Li
>Assignee: Heejong Lee
>Priority: Major
>
> Current {{ParquetIO}} supports neither column projection nor filter predicate
> which defeats the performance motivation of using Parquet in the first place.
> That's why we have our own implementation of
> [ParquetIO|https://github.com/spotify/scio/tree/master/scio-parquet/src] in
> Scio.
> Reading Parquet as Avro with column projection has some complications,
> namely, the resulting Avro records may be incomplete and will not survive
> ser/de. A workaround maybe provide a {{TypedRead}} interface that takes a
> {{Function}} that maps invalid Avro {{A}} into user defined type {{B}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9881) refactor KafkaIOIT for reusing pipeline
[ https://issues.apache.org/jira/browse/BEAM-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9881. --- Fix Version/s: Not applicable Resolution: Won't Do > refactor KafkaIOIT for reusing pipeline > --- > > Key: BEAM-9881 > URL: https://issues.apache.org/jira/browse/BEAM-9881 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: Not applicable > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9869) adding self-contained Kafka service jar for testing
[ https://issues.apache.org/jira/browse/BEAM-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-9869: - Assignee: Heejong Lee > adding self-contained Kafka service jar for testing > --- > > Key: BEAM-9869 > URL: https://issues.apache.org/jira/browse/BEAM-9869 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9869) adding self-contained Kafka service jar for testing
[ https://issues.apache.org/jira/browse/BEAM-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9869: -- Status: Open (was: Triage Needed) > adding self-contained Kafka service jar for testing > --- > > Key: BEAM-9869 > URL: https://issues.apache.org/jira/browse/BEAM-9869 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Priority: Major > > adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9869) adding self-contained Kafka service jar for testing
Heejong Lee created BEAM-9869: - Summary: adding self-contained Kafka service jar for testing Key: BEAM-9869 URL: https://issues.apache.org/jira/browse/BEAM-9869 Project: Beam Issue Type: Improvement Components: testing Reporter: Heejong Lee adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9792) BigQuery insertAll ignores retry policy for all errors throwing IOException
[ https://issues.apache.org/jira/browse/BEAM-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9792: -- Status: Open (was: Triage Needed) > BigQuery insertAll ignores retry policy for all errors throwing IOException > --- > > Key: BEAM-9792 > URL: https://issues.apache.org/jira/browse/BEAM-9792 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Heejong Lee >Priority: Major > > BigQuery insertAll ignores retry policy for all errors throwing IOException > (it retries forever with backoff) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9792) BigQuery insertAll ignores retry policy for all errors throwing IOException
Heejong Lee created BEAM-9792: - Summary: BigQuery insertAll ignores retry policy for all errors throwing IOException Key: BEAM-9792 URL: https://issues.apache.org/jira/browse/BEAM-9792 Project: Beam Issue Type: Bug Components: io-java-gcp Reporter: Heejong Lee BigQuery insertAll ignores retry policy for all errors throwing IOException (it retries forever with backoff) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9056) Staging artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9056. --- Fix Version/s: 2.21.0 Resolution: Fixed > Staging artifacts from environment > -- > > Key: BEAM-9056 > URL: https://issues.apache.org/jira/browse/BEAM-9056 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.21.0 > > Time Spent: 9h 50m > Remaining Estimate: 0h > > staging artifacts from artifact information embedded in environment proto. > detail: > https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9455) Environment-sensitive provisioning for Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9455: -- Parent: BEAM-9238 Issue Type: Sub-task (was: Improvement) > Environment-sensitive provisioning for Dataflow > --- > > Key: BEAM-9455 > URL: https://issues.apache.org/jira/browse/BEAM-9455 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Environment-sensitive provisioning for Dataflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9455) Environment-sensitive provisioning for Dataflow
Heejong Lee created BEAM-9455: - Summary: Environment-sensitive provisioning for Dataflow Key: BEAM-9455 URL: https://issues.apache.org/jira/browse/BEAM-9455 Project: Beam Issue Type: Improvement Components: runner-dataflow Reporter: Heejong Lee Assignee: Heejong Lee Environment-sensitive provisioning for Dataflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9455) Environment-sensitive provisioning for Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9455: -- Status: Open (was: Triage Needed) > Environment-sensitive provisioning for Dataflow > --- > > Key: BEAM-9455 > URL: https://issues.apache.org/jira/browse/BEAM-9455 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Environment-sensitive provisioning for Dataflow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9229. --- Fix Version/s: 2.20.0 Resolution: Fixed > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.20.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9383: -- Parent: BEAM-9238 Issue Type: Sub-task (was: Improvement) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9415) fix postcommit xlang validate runner
[ https://issues.apache.org/jira/browse/BEAM-9415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9415: -- Description: broken since [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1838/] The proposed PR checks whether the coder is compatible with Java SDK before rehydrating it from expanded components. The coder id renaming is only needed for Java SDK compatible coders so the change is safe. was:broken since [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1838/] > fix postcommit xlang validate runner > > > Key: BEAM-9415 > URL: https://issues.apache.org/jira/browse/BEAM-9415 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > broken since [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1838/] > The proposed PR checks whether the coder is compatible with Java SDK before > rehydrating it from expanded components. The coder id renaming is only needed > for Java SDK compatible coders so the change is safe. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9425) explicitly register default environment instead of assuming the first one as a default
[ https://issues.apache.org/jira/browse/BEAM-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9425: -- Status: Open (was: Triage Needed) > explicitly register default environment instead of assuming the first one as > a default > -- > > Key: BEAM-9425 > URL: https://issues.apache.org/jira/browse/BEAM-9425 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > explicitly register default environment instead of assuming the first one as > a default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9425) explicitly register default environment instead of assuming the first one as a default
Heejong Lee created BEAM-9425: - Summary: explicitly register default environment instead of assuming the first one as a default Key: BEAM-9425 URL: https://issues.apache.org/jira/browse/BEAM-9425 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Heejong Lee Assignee: Heejong Lee explicitly register default environment instead of assuming the first one as a default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9415) fix postcommit xlang validate runner
[ https://issues.apache.org/jira/browse/BEAM-9415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9415: -- Description: broken since [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1838/] > fix postcommit xlang validate runner > > > Key: BEAM-9415 > URL: https://issues.apache.org/jira/browse/BEAM-9415 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > broken since [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1838/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9415) fix postcommit xlang validate runner
[ https://issues.apache.org/jira/browse/BEAM-9415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9415: -- Status: Open (was: Triage Needed) > fix postcommit xlang validate runner > > > Key: BEAM-9415 > URL: https://issues.apache.org/jira/browse/BEAM-9415 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9415) fix postcommit xlang validate runner
Heejong Lee created BEAM-9415: - Summary: fix postcommit xlang validate runner Key: BEAM-9415 URL: https://issues.apache.org/jira/browse/BEAM-9415 Project: Beam Issue Type: Bug Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9383) Staging Dataflow artifacts from environment
Heejong Lee created BEAM-9383: - Summary: Staging Dataflow artifacts from environment Key: BEAM-9383 URL: https://issues.apache.org/jira/browse/BEAM-9383 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9348) Enable CrossLanguageValidateRunner test for direct runner
[ https://issues.apache.org/jira/browse/BEAM-9348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9348: -- Status: Open (was: Triage Needed) > Enable CrossLanguageValidateRunner test for direct runner > - > > Key: BEAM-9348 > URL: https://issues.apache.org/jira/browse/BEAM-9348 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Enable CrossLanguageValidateRunner test for direct runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9348) Enable CrossLanguageValidateRunner test for direct runner
Heejong Lee created BEAM-9348: - Summary: Enable CrossLanguageValidateRunner test for direct runner Key: BEAM-9348 URL: https://issues.apache.org/jira/browse/BEAM-9348 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee Enable CrossLanguageValidateRunner test for direct runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9230. --- Fix Version/s: 2.20.0 Resolution: Fixed > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-7961. --- Fix Version/s: 2.19.0 Resolution: Fixed > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.19.0 > > Time Spent: 23h 20m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9338) add postcommit XVR spark badge
[ https://issues.apache.org/jira/browse/BEAM-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9338. --- Fix Version/s: Not applicable Resolution: Fixed > add postcommit XVR spark badge > -- > > Key: BEAM-9338 > URL: https://issues.apache.org/jira/browse/BEAM-9338 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: Not applicable > > Time Spent: 0.5h > Remaining Estimate: 0h > > add postcommit xvr spark badges -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9341) postcommit xvr flink, spark failure
[ https://issues.apache.org/jira/browse/BEAM-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9341. --- Fix Version/s: Not applicable Resolution: Fixed > postcommit xvr flink, spark failure > --- > > Key: BEAM-9341 > URL: https://issues.apache.org/jira/browse/BEAM-9341 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > started from [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1738/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9341) postcommit xvr flink, spark failure
[ https://issues.apache.org/jira/browse/BEAM-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9341: -- Description: started from [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1738/] > postcommit xvr flink, spark failure > --- > > Key: BEAM-9341 > URL: https://issues.apache.org/jira/browse/BEAM-9341 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > started from [https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1738/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9341) postcommit xvr flink, spark failure
[ https://issues.apache.org/jira/browse/BEAM-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9341: -- Status: Open (was: Triage Needed) > postcommit xvr flink, spark failure > --- > > Key: BEAM-9341 > URL: https://issues.apache.org/jira/browse/BEAM-9341 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9341) postcommit xvr flink, spark failure
Heejong Lee created BEAM-9341: - Summary: postcommit xvr flink, spark failure Key: BEAM-9341 URL: https://issues.apache.org/jira/browse/BEAM-9341 Project: Beam Issue Type: Bug Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9338) add postcommit XVR spark badge
[ https://issues.apache.org/jira/browse/BEAM-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9338: -- Summary: add postcommit XVR spark badge (was: add postcommit XVR spark tickers) > add postcommit XVR spark badge > -- > > Key: BEAM-9338 > URL: https://issues.apache.org/jira/browse/BEAM-9338 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > add postcommit xvr spark tickers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9338) add postcommit XVR spark badge
[ https://issues.apache.org/jira/browse/BEAM-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9338: -- Description: add postcommit xvr spark badges (was: add postcommit xvr spark tickers) > add postcommit XVR spark badge > -- > > Key: BEAM-9338 > URL: https://issues.apache.org/jira/browse/BEAM-9338 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > add postcommit xvr spark badges -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9338) add postcommit XVR spark tickers
Heejong Lee created BEAM-9338: - Summary: add postcommit XVR spark tickers Key: BEAM-9338 URL: https://issues.apache.org/jira/browse/BEAM-9338 Project: Beam Issue Type: Bug Components: website Reporter: Heejong Lee Assignee: Heejong Lee add postcommit xvr spark tickers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9338) add postcommit XVR spark tickers
[ https://issues.apache.org/jira/browse/BEAM-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9338: -- Status: Open (was: Triage Needed) > add postcommit XVR spark tickers > > > Key: BEAM-9338 > URL: https://issues.apache.org/jira/browse/BEAM-9338 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > add postcommit xvr spark tickers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9338) add postcommit XVR spark tickers
[ https://issues.apache.org/jira/browse/BEAM-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9338: -- Issue Type: Improvement (was: Bug) > add postcommit XVR spark tickers > > > Key: BEAM-9338 > URL: https://issues.apache.org/jira/browse/BEAM-9338 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > add postcommit xvr spark tickers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9335) update hard-coded coder id when translating Java external transforms
Heejong Lee created BEAM-9335: - Summary: update hard-coded coder id when translating Java external transforms Key: BEAM-9335 URL: https://issues.apache.org/jira/browse/BEAM-9335 Project: Beam Issue Type: Bug Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee hard-coded coder id needs to be updated when translating Java external transforms. Otherwise pipeline will fail if coder id is reused. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9335) update hard-coded coder id when translating Java external transforms
[ https://issues.apache.org/jira/browse/BEAM-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9335: -- Status: Open (was: Triage Needed) > update hard-coded coder id when translating Java external transforms > > > Key: BEAM-9335 > URL: https://issues.apache.org/jira/browse/BEAM-9335 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > hard-coded coder id needs to be updated when translating Java external > transforms. Otherwise pipeline will fail if coder id is reused. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9237) Environment-sensitive provisioning
[ https://issues.apache.org/jira/browse/BEAM-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9237: -- Parent: BEAM-9238 Issue Type: Sub-task (was: Improvement) > Environment-sensitive provisioning > -- > > Key: BEAM-9237 > URL: https://issues.apache.org/jira/browse/BEAM-9237 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > * Extending provision service > # Extending provision request proto to add an environment ID parameter > # Committing manifest per environment > # Passing an environment ID to SDK harness > # Modifying Job API to keep multiple retrieval tokens > # Retrieve the metadata only relevant to the given environment from SDK > harness with an environment ID -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9056) Staging artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9056: -- Parent: BEAM-9238 Issue Type: Sub-task (was: Improvement) > Staging artifacts from environment > -- > > Key: BEAM-9056 > URL: https://issues.apache.org/jira/browse/BEAM-9056 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > staging artifacts from artifact information embedded in environment proto. > detail: > https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9229: -- Parent: BEAM-9238 Issue Type: Sub-task (was: Improvement) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9238) Cross-language pipeline dependency management
[ https://issues.apache.org/jira/browse/BEAM-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9238: -- Status: Open (was: Triage Needed) > Cross-language pipeline dependency management > - > > Key: BEAM-9238 > URL: https://issues.apache.org/jira/browse/BEAM-9238 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Meta-issue for tracking cross-language pipeline dependency management -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9238) Cross-language pipeline dependency management
Heejong Lee created BEAM-9238: - Summary: Cross-language pipeline dependency management Key: BEAM-9238 URL: https://issues.apache.org/jira/browse/BEAM-9238 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Meta-issue for tracking cross-language pipeline dependency management -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9238) Cross-language pipeline dependency management
[ https://issues.apache.org/jira/browse/BEAM-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-9238: - Assignee: Heejong Lee > Cross-language pipeline dependency management > - > > Key: BEAM-9238 > URL: https://issues.apache.org/jira/browse/BEAM-9238 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Meta-issue for tracking cross-language pipeline dependency management -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9237) Environment-sensitive provisioning
Heejong Lee created BEAM-9237: - Summary: Environment-sensitive provisioning Key: BEAM-9237 URL: https://issues.apache.org/jira/browse/BEAM-9237 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee * Extending provision service # Extending provision request proto to add an environment ID parameter # Committing manifest per environment # Passing an environment ID to SDK harness # Modifying Job API to keep multiple retrieval tokens # Retrieve the metadata only relevant to the given environment from SDK harness with an environment ID -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9237) Environment-sensitive provisioning
[ https://issues.apache.org/jira/browse/BEAM-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9237: -- Status: Open (was: Triage Needed) > Environment-sensitive provisioning > -- > > Key: BEAM-9237 > URL: https://issues.apache.org/jira/browse/BEAM-9237 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > * Extending provision service > # Extending provision request proto to add an environment ID parameter > # Committing manifest per environment > # Passing an environment ID to SDK harness > # Modifying Job API to keep multiple retrieval tokens > # Retrieve the metadata only relevant to the given environment from SDK > harness with an environment ID -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
Heejong Lee created BEAM-9230: - Summary: Enable CrossLanguageValidateRunner test for Spark runner Key: BEAM-9230 URL: https://issues.apache.org/jira/browse/BEAM-9230 Project: Beam Issue Type: Improvement Components: runner-spark Reporter: Heejong Lee Assignee: Heejong Lee Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9230: -- Status: Open (was: Triage Needed) > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9229: -- Status: Open (was: Triage Needed) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9229) Adding dependency information to Environment proto
Heejong Lee created BEAM-9229: - Summary: Adding dependency information to Environment proto Key: BEAM-9229 URL: https://issues.apache.org/jira/browse/BEAM-9229 Project: Beam Issue Type: Improvement Components: beam-model Reporter: Heejong Lee Assignee: Heejong Lee Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014725#comment-17014725
]
Heejong Lee edited comment on BEAM-9093 at 1/13/20 11:24 PM:
-
I think this is WAI not a bug. There are two options {{\-\-no_use_public_ips}}
and {{\-\-use_public_ips}} which turn on and off the flag {{use_public_ips}}.
When users use the second parameter of PipelineOptions, it's supposed to
directly override flags not options.
was (Author: heejong):
I think this is WAI not a bug. There are two options '--no_use_public_ips' and
'--use_public_ips' which turn on and off the flag 'use_public_ips'. When users
use the second parameter of PipelineOptions, it's supposed to directly override
flags not options.
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014725#comment-17014725
]
Heejong Lee commented on BEAM-9093:
---
I think this is WAI not a bug. There are two options '--no_use_public_ips' and
'--use_public_ips' which turn on and off the flag 'use_public_ips'. When users
use the second parameter of PipelineOptions, it's supposed to directly override
flags not options.
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9056) Staging artifacts from environment
Heejong Lee created BEAM-9056: - Summary: Staging artifacts from environment Key: BEAM-9056 URL: https://issues.apache.org/jira/browse/BEAM-9056 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee staging artifacts from artifact information embedded in environment proto. detail: https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9056) Staging artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9056: -- Status: Open (was: Triage Needed) > Staging artifacts from environment > -- > > Key: BEAM-9056 > URL: https://issues.apache.org/jira/browse/BEAM-9056 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > staging artifacts from artifact information embedded in environment proto. > detail: > https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9048) Matching compile-time and run-time type information in Java external transforms
Heejong Lee created BEAM-9048: - Summary: Matching compile-time and run-time type information in Java external transforms Key: BEAM-9048 URL: https://issues.apache.org/jira/browse/BEAM-9048 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: Heejong Lee Output type annotation by Java external transform API only checks compile-time type correctness. Expansion service is solely responsible for run-time type correctness. It would be great if we match both information at pipeline construction time so that preventing pipeline failing in runners after 10 hours. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9048) Matching compile-time and run-time type information in Java external transforms
[ https://issues.apache.org/jira/browse/BEAM-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee reassigned BEAM-9048: - Assignee: Heejong Lee > Matching compile-time and run-time type information in Java external > transforms > --- > > Key: BEAM-9048 > URL: https://issues.apache.org/jira/browse/BEAM-9048 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > Output type annotation by Java external transform API only checks > compile-time type correctness. Expansion service is solely responsible for > run-time type correctness. It would be great if we match both information at > pipeline construction time so that preventing pipeline failing in runners > after 10 hours. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9048) Matching compile-time and run-time type information in Java external transforms
[ https://issues.apache.org/jira/browse/BEAM-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9048: -- Status: Open (was: Triage Needed) > Matching compile-time and run-time type information in Java external > transforms > --- > > Key: BEAM-9048 > URL: https://issues.apache.org/jira/browse/BEAM-9048 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Priority: Major > > Output type annotation by Java external transform API only checks > compile-time type correctness. Expansion service is solely responsible for > run-time type correctness. It would be great if we match both information at > pipeline construction time so that preventing pipeline failing in runners > after 10 hours. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9047) Java TestPipeline doesn't support remote job endpoint
[ https://issues.apache.org/jira/browse/BEAM-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9047: -- Status: Open (was: Triage Needed) > Java TestPipeline doesn't support remote job endpoint > - > > Key: BEAM-9047 > URL: https://issues.apache.org/jira/browse/BEAM-9047 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Priority: Major > > Java TestPipeline doesn't support remote job endpoint. It does neither wait > until pipeline finishes nor check the output status. Workaround would be > manually checking output status in test teardown function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9047) Java TestPipeline doesn't support remote job endpoint
Heejong Lee created BEAM-9047: - Summary: Java TestPipeline doesn't support remote job endpoint Key: BEAM-9047 URL: https://issues.apache.org/jira/browse/BEAM-9047 Project: Beam Issue Type: Improvement Components: testing Reporter: Heejong Lee Java TestPipeline doesn't support remote job endpoint. It does neither wait until pipeline finishes nor check the output status. Workaround would be manually checking output status in test teardown function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9034) Update environment_id for ExternalTransform in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-9034. --- Fix Version/s: 2.19.0 Resolution: Fixed > Update environment_id for ExternalTransform in Python SDK > - > > Key: BEAM-9034 > URL: https://issues.apache.org/jira/browse/BEAM-9034 > Project: Beam > Issue Type: Bug > Components: beam-model, java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.19.0 > > Time Spent: 50m > Remaining Estimate: 0h > > ExternalTransform in Python SDK also needs environment_id field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8903) handling --jar_packages experimental flag in PortableRunner for staging external dependencies used in expanded transforms
[ https://issues.apache.org/jira/browse/BEAM-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-8903. --- Fix Version/s: 2.18.0 Resolution: Fixed > handling --jar_packages experimental flag in PortableRunner for staging > external dependencies used in expanded transforms > - > > Key: BEAM-8903 > URL: https://issues.apache.org/jira/browse/BEAM-8903 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.18.0 > > Time Spent: 50m > Remaining Estimate: 0h > > handling --jar_packages experimental flag in PortableRunner for staging > external dependencies used in expanded transforms. could be removed after > proper dependency management for cross-language pipelines is implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8545) don't docker pull before docker run
[ https://issues.apache.org/jira/browse/BEAM-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-8545. --- Fix Version/s: 2.17.0 Resolution: Won't Fix > don't docker pull before docker run > > > Key: BEAM-8545 > URL: https://issues.apache.org/jira/browse/BEAM-8545 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.17.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Since 'docker run' automatically pulls when the image doesn't exist locally, > I think it's safe to remove explicit 'docker pull' before 'docker run'. > Without 'docker pull', we won't update the local image with the remote image > (for the same tag) but it shouldn't be a problem in prod that the unique tag > is assumed for each released version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8905) matching Java PCollectionTuple translation naming convention in expansion service with index only
[
https://issues.apache.org/jira/browse/BEAM-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heejong Lee resolved BEAM-8905.
---
Fix Version/s: 2.18.0
Resolution: Fixed
> matching Java PCollectionTuple translation naming convention in expansion
> service with index only
> -
>
> Key: BEAM-8905
> URL: https://issues.apache.org/jira/browse/BEAM-8905
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Python PCollectionTuple is translated to an index-keyed map e.g.
> \{0->pcollection1, 1->pcollection2}, however Java PCollectionTuple is
> translated to slightly different formats such as \{output_0->pcollection1,
> output_1->pcollection2}. We need to match these naming conventions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8904) properly update output pcollections from expanded transforms
[ https://issues.apache.org/jira/browse/BEAM-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-8904. --- Fix Version/s: 2.18.0 Resolution: Fixed > properly update output pcollections from expanded transforms > > > Key: BEAM-8904 > URL: https://issues.apache.org/jira/browse/BEAM-8904 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.18.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > currently output pcollections from expanded transforms are ignored. we need > to properly update output pcollections when it's returned to the caller of > expansion service. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8902) parameterize input type of Java external transform
[ https://issues.apache.org/jira/browse/BEAM-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-8902. --- Fix Version/s: 2.18.0 Resolution: Fixed > parameterize input type of Java external transform > -- > > Key: BEAM-8902 > URL: https://issues.apache.org/jira/browse/BEAM-8902 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.18.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > parameterize input type of Java external transform to allow multiple > pcollections like PCollectionTuple or PCollectionList. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9034) Update environment_id for ExternalTransform in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9034: -- Summary: Update environment_id for ExternalTransform in Python SDK (was: Update environment for ExternalTransform in Python SDK) > Update environment_id for ExternalTransform in Python SDK > - > > Key: BEAM-9034 > URL: https://issues.apache.org/jira/browse/BEAM-9034 > Project: Beam > Issue Type: Bug > Components: beam-model, java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > ExternalTransform in Python SDK also needs environment_id field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9034) Update environment for ExternalTransform in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee updated BEAM-9034: -- Status: Open (was: Triage Needed) > Update environment for ExternalTransform in Python SDK > -- > > Key: BEAM-9034 > URL: https://issues.apache.org/jira/browse/BEAM-9034 > Project: Beam > Issue Type: Bug > Components: beam-model, java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > > ExternalTransform in Python SDK also needs environment_id field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9034) Update environment for ExternalTransform in Python SDK
Heejong Lee created BEAM-9034: - Summary: Update environment for ExternalTransform in Python SDK Key: BEAM-9034 URL: https://issues.apache.org/jira/browse/BEAM-9034 Project: Beam Issue Type: Bug Components: beam-model, java-fn-execution Reporter: Heejong Lee Assignee: Heejong Lee ExternalTransform in Python SDK also needs environment_id field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8901) add experimental flag for reusing flink local environment
[ https://issues.apache.org/jira/browse/BEAM-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heejong Lee resolved BEAM-8901. --- Fix Version/s: 2.19.0 Resolution: Workaround > add experimental flag for reusing flink local environment > - > > Key: BEAM-8901 > URL: https://issues.apache.org/jira/browse/BEAM-8901 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Fix For: 2.19.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Flink job server launches a new mini cluster every time we run the pipeline > on Flink local environment. To prevent OOM, we need to reuse existing Flink > local environment if possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
