[jira] [Updated] (BEAM-6928) Make Python SDK custom Sink the default Sink for BigQuery
[ https://issues.apache.org/jira/browse/BEAM-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6928: -- Fix Version/s: (was: 2.19.0) > Make Python SDK custom Sink the default Sink for BigQuery > - > > Key: BEAM-6928 > URL: https://issues.apache.org/jira/browse/BEAM-6928 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > > This is for 2.14.0 - please bump version to 2.14.0 when doing 2.13.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6892) Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS if not specified by user.
[ https://issues.apache.org/jira/browse/BEAM-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6892: -- Fix Version/s: (was: 2.19.0) > Use temp_location for BQ FILE_LOADS on DirectRunner, and autocreate it in GCS > if not specified by user. > --- > > Key: BEAM-6892 > URL: https://issues.apache.org/jira/browse/BEAM-6892 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Pablo Estrada >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8992) ./gradlew :sdks:go:examples:goVet fails
[
https://issues.apache.org/jira/browse/BEAM-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-8992:
--
Fix Version/s: (was: 2.19.0)
> ./gradlew :sdks:go:examples:goVet fails
> ---
>
> Key: BEAM-8992
> URL: https://issues.apache.org/jira/browse/BEAM-8992
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Affects Versions: 2.19.0
>Reporter: Udi Meiri
>Assignee: Robert Burke
>Priority: Major
>
> {code}
> > Task :sdks:go:examples:resolveBuildDependencies
> Resolving
> ./github.com/apache/beam/sdks/go@/usr/local/google/home/ehudm/src/beam-release/sdks/go
> .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2:
> cannot find package "github.com/linkedin/goavro" in any of:
>
> /usr/local/google/home/ehudm/src/beam-release/sdks/go/examples/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro
> (vendor tree)
>
> /usr/local/google/home/ehudm/.gradle/go/binary/1.12/go/src/github.com/linkedin/goavro
> (from $GOROOT)
>
> /usr/local/google/home/ehudm/src/beam-release/sdks/go/examples/.gogradle/project_gopath/src/github.com/linkedin/goavro
> (from $GOPATH)
> > Task :sdks:go:examples:goVet FAILED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9012) Include `-> None` on Pipeline and PipelineOptions `__init__` methods for pytype compatibility
[
https://issues.apache.org/jira/browse/BEAM-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-9012:
--
Fix Version/s: (was: 2.19.0)
> Include `-> None` on Pipeline and PipelineOptions `__init__` methods for
> pytype compatibility
> -
>
> Key: BEAM-9012
> URL: https://issues.apache.org/jira/browse/BEAM-9012
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> mypy [made a decision|https://github.com/python/mypy/issues/604] to allow
> init methods to omit {{\-> None}} return type annotations, but pytype has no
> such feature. I think we should include {{\-> None}} annotations for pytype
> compatibility.
> cc: [~chadrik]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9012) Include `-> None` on Pipeline and PipelineOptions `__init__` methods for pytype compatibility
[
https://issues.apache.org/jira/browse/BEAM-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011517#comment-17011517
]
Kenneth Knowles commented on BEAM-9012:
---
Peanut gallery: Seems like satisfying both static checkers is worth it in this
case. I'm employed by Google, but my heart belongs to types/verification, so
I'm not just saying this because of my paycheck.
> Include `-> None` on Pipeline and PipelineOptions `__init__` methods for
> pytype compatibility
> -
>
> Key: BEAM-9012
> URL: https://issues.apache.org/jira/browse/BEAM-9012
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> mypy [made a decision|https://github.com/python/mypy/issues/604] to allow
> init methods to omit {{\-> None}} return type annotations, but pytype has no
> such feature. I think we should include {{\-> None}} annotations for pytype
> compatibility.
> cc: [~chadrik]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9013) Multi-output TestStream breaks the DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9013: -- Fix Version/s: (was: 2.19.0) > Multi-output TestStream breaks the DataflowRunner > - > > Key: BEAM-9013 > URL: https://issues.apache.org/jira/browse/BEAM-9013 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers
[ https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9049: -- Fix Version/s: (was: 2.19.0) > MemoryMonitor thrashing detection is too aggressive for batch workers > - > > Key: BEAM-9049 > URL: https://issues.apache.org/jira/browse/BEAM-9049 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the streaming dataflow worker we've implemented push-back so that we will > reduce parallelism when there is memory pressure. Since we cannot do this on > the batch worker, it doesn't make sense for our thrashing detection to be so > aggressive. We should increase the thresholds used for thrashing detection > when running on Batch workloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9013) Multi-output TestStream breaks the DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011514#comment-17011514 ] Kenneth Knowles commented on BEAM-9013: --- Can you update the "Affects Version" and "Fix Version" fields to match what release notes should display? > Multi-output TestStream breaks the DataflowRunner > - > > Key: BEAM-9013 > URL: https://issues.apache.org/jira/browse/BEAM-9013 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9068) Use local docker image if available for Java and Go
[ https://issues.apache.org/jira/browse/BEAM-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9068: -- Fix Version/s: (was: 2.19.0) > Use local docker image if available for Java and Go > --- > > Key: BEAM-9068 > URL: https://issues.apache.org/jira/browse/BEAM-9068 > Project: Beam > Issue Type: Task > Components: build-system >Affects Versions: 2.17.0 >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > > Python already implemented in this way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7861: -- Fix Version/s: (was: 2.19.0) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9030: -- Fix Version/s: (was: 2.19.0) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9030: -- Issue Type: Improvement (was: Bug) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8989) Backwards incompatible change in ParDo.getSideInputs (caught by failure when running Apache Nemo quickstart)
[
https://issues.apache.org/jira/browse/BEAM-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-8989:
--
Fix Version/s: (was: 2.19.0)
> Backwards incompatible change in ParDo.getSideInputs (caught by failure when
> running Apache Nemo quickstart)
>
>
> Key: BEAM-8989
> URL: https://issues.apache.org/jira/browse/BEAM-8989
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.16.0, 2.17.0, 2.18.0
>Reporter: Luke Cwik
>Assignee: Reuven Lax
>Priority: Critical
>
> [PR/9275|https://github.com/apache/beam/pull/9275] changed
> *ParDo.getSideInputs* from *List* to *Map PCollectionView>* which is backwards incompatible change and was released as
> part of Beam 2.16.0 erroneously.
> Running the Apache Nemo Quickstart fails with:
>
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Translator private
> static void
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
> have failed to translate
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27Exception in thread
> "main" java.lang.RuntimeException: Translator private static void
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
> have failed to translate
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27 at
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:113)
> at
> org.apache.nemo.compiler.frontend.beam.PipelineVisitor.visitPrimitiveTransform(PipelineVisitor.java:46)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
> at
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) at
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:80) at
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:31) at
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) at
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at
> org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185) at
> org.apache.beam.examples.WordCount.main(WordCount.java:192)Caused by:
> java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:109)
> ... 14 moreCaused by: java.lang.NoSuchMethodError:
> org.apache.beam.sdk.transforms.ParDo$MultiOutput.getSideInputs()Ljava/util/List;
> at
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(PipelineTranslator.java:236)
> ... 19 more{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9075) Join re-ordering doesn't work for ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9075?focusedWorklogId=368817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368817 ] ASF GitHub Bot logged work on BEAM-9075: Author: ASF GitHub Bot Created on: 09/Jan/20 06:57 Start Date: 09/Jan/20 06:57 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10542: [BEAM-9075] Disable JoinCommuteRule for ZetaSQL planner URL: https://github.com/apache/beam/pull/10542#issuecomment-572419480 LGTM It would be great if we can test by internal test suite before merging it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368817) Time Spent: 40m (was: 0.5h) > Join re-ordering doesn't work for ZetaSQL > - > > Key: BEAM-9075 > URL: https://issues.apache.org/jira/browse/BEAM-9075 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Affects Versions: 2.17.0 >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Join re-ordering can actually break queries in the ZetaSQL planner because > Calcite's JoinCommuteRule doesn't work when schemas have nested fields. It > only adjusts RexInputRef nodes and not RexFieldAccess. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9075) Join re-ordering doesn't work for ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9075?focusedWorklogId=368818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368818 ] ASF GitHub Bot logged work on BEAM-9075: Author: ASF GitHub Bot Created on: 09/Jan/20 06:57 Start Date: 09/Jan/20 06:57 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10542: [BEAM-9075] Disable JoinCommuteRule for ZetaSQL planner URL: https://github.com/apache/beam/pull/10542#issuecomment-572419480 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368818) Time Spent: 50m (was: 40m) > Join re-ordering doesn't work for ZetaSQL > - > > Key: BEAM-9075 > URL: https://issues.apache.org/jira/browse/BEAM-9075 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Affects Versions: 2.17.0 >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Join re-ordering can actually break queries in the ZetaSQL planner because > Calcite's JoinCommuteRule doesn't work when schemas have nested fields. It > only adjusts RexInputRef nodes and not RexFieldAccess. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9076) [QST] How to make python sdk wheel package?
[
https://issues.apache.org/jira/browse/BEAM-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keunhyun Oh updated BEAM-9076:
--
Description:
How to make python sdk wheel package?
I've modified and tested beam's python sdks.
I've tried to run command to upload local built wheel package to my local
repository like below.
{code:java}
$> python setup.py sdist bdist_wheel upload -r local{code}
Below files are not included in the package made by me.
{code:java}
./apache_beam/coders: coder_impl.cpython-36m-x86_64-linux-gnu.so
./apache_beam/coders: stream.cpython-36m-x86_64-linux-gnu.so
./apache_beam/metrics: execution.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners: common.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: logger.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: opcounters.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: operations.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: statesampler_fast.cpython-36m-x86_64-linux-gnu.so
./apache_beam/transforms: cy_combiners.cpython-36m-x86_64-linux-gnu.so
./apache_beam/transforms:
cy_dataflow_distribution_counter.cpython-36m-x86_64-linux-gnu.so
./apache_beam/utils: counters.cpython-36m-x86_64-linux-gnu.so
./apache_beam/utils: windowed_value.cpython-36m-x86_64-linux-gnu.so
{code}
Please, let me know what is right command to make the wheel package?
was:
How to make python sdk wheel package?
I've modified and tested beam's python sdks.
I've tried to run command to upload local built wheel package to my local
repository like below.
{code:java}
$> python setup.py sdist bdist_wheel upload -r local{code}
Below files are not included in the package made by me.
{code:java}
./apache_beam/coders: coder_impl.cpython-36m-x86_64-linux-gnu.so
./apache_beam/coders: stream.cpython-36m-x86_64-linux-gnu.so
./apache_beam/metrics: execution.cpython-36m-x86_64-linux-gnu.soOnly in
./apache_beam/runners: common.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: logger.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: opcounters.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: operations.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: statesampler_fast.cpython-36m-x86_64-linux-gnu.so
./apache_beam/transforms: cy_combiners.cpython-36m-x86_64-linux-gnu.so
./apache_beam/transforms:
cy_dataflow_distribution_counter.cpython-36m-x86_64-linux-gnu.so
./apache_beam/utils: counters.cpython-36m-x86_64-linux-gnu.soOnly in
./apache_beam/utils: windowed_value.cpython-36m-x86_64-linux-gnu.so
{code}
Please, let me know what is right command to make the wheel package?
> [QST] How to make python sdk wheel package?
> ---
>
> Key: BEAM-9076
> URL: https://issues.apache.org/jira/browse/BEAM-9076
> Project: Beam
> Issue Type: Task
> Components: build-system
>Affects Versions: 2.16.0
>Reporter: Keunhyun Oh
>Priority: Major
> Fix For: Not applicable
>
>
> How to make python sdk wheel package?
> I've modified and tested beam's python sdks.
> I've tried to run command to upload local built wheel package to my local
> repository like below.
> {code:java}
> $> python setup.py sdist bdist_wheel upload -r local{code}
> Below files are not included in the package made by me.
> {code:java}
> ./apache_beam/coders: coder_impl.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/coders: stream.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/metrics: execution.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/runners: common.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/runners/worker: logger.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/runners/worker: opcounters.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/runners/worker: operations.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/runners/worker:
> statesampler_fast.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/transforms: cy_combiners.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/transforms:
> cy_dataflow_distribution_counter.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/utils: counters.cpython-36m-x86_64-linux-gnu.so
> ./apache_beam/utils: windowed_value.cpython-36m-x86_64-linux-gnu.so
> {code}
> Please, let me know what is right command to make the wheel package?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9076) [QST] How to make python sdk wheel package?
Keunhyun Oh created BEAM-9076:
-
Summary: [QST] How to make python sdk wheel package?
Key: BEAM-9076
URL: https://issues.apache.org/jira/browse/BEAM-9076
Project: Beam
Issue Type: Task
Components: build-system
Affects Versions: 2.16.0
Reporter: Keunhyun Oh
Fix For: Not applicable
How to make python sdk wheel package?
I've modified and tested beam's python sdks.
I've tried to run command to upload local built wheel package to my local
repository like below.
{code:java}
$> python setup.py sdist bdist_wheel upload -r local{code}
Below files are not included in the package made by me.
{code:java}
./apache_beam/coders: coder_impl.cpython-36m-x86_64-linux-gnu.so
./apache_beam/coders: stream.cpython-36m-x86_64-linux-gnu.so
./apache_beam/metrics: execution.cpython-36m-x86_64-linux-gnu.soOnly in
./apache_beam/runners: common.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: logger.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: opcounters.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: operations.cpython-36m-x86_64-linux-gnu.so
./apache_beam/runners/worker: statesampler_fast.cpython-36m-x86_64-linux-gnu.so
./apache_beam/transforms: cy_combiners.cpython-36m-x86_64-linux-gnu.so
./apache_beam/transforms:
cy_dataflow_distribution_counter.cpython-36m-x86_64-linux-gnu.so
./apache_beam/utils: counters.cpython-36m-x86_64-linux-gnu.soOnly in
./apache_beam/utils: windowed_value.cpython-36m-x86_64-linux-gnu.so
{code}
Please, let me know what is right command to make the wheel package?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368806 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 05:07 Start Date: 09/Jan/20 05:07 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364561109 ## File path: sdks/python/gen_protos.py ## @@ -47,6 +50,164 @@ ] +def generate_urn_files(log, out_dir): Review comment: Note that the parent package is excluded from pylint, so style is not an issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368806) Time Spent: 41h 10m (was: 41h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 41h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368796
]
ASF GitHub Bot logged work on BEAM-7746:
Author: ASF GitHub Bot
Created on: 09/Jan/20 03:18
Start Date: 09/Jan/20 03:18
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10367: [BEAM-7746]
Add python type hints (part 2)
URL: https://github.com/apache/beam/pull/10367#discussion_r364543781
##
File path: sdks/python/apache_beam/transforms/sideinputs.py
##
@@ -58,6 +59,16 @@ def map_via_end(source_window):
return map_via_end
+def get_sideinput_index(tag):
+ # type: (str) -> int
+ match = re.match('side([0-9]+)(-.*)?$', tag,
+ re.DOTALL)
Review comment:
I made two assumptions with this change:
- that all these instances were supposed to be doing the same thing
- that the author who used `re.DOTALL` did so deliberately
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368796)
Time Spent: 41h (was: 40h 50m)
> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 41h
> Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484
> type hints so that I can clearly see what types are required, get completion
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here: [https://github.com/apache/beam/pull/9056]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368794
]
ASF GitHub Bot logged work on BEAM-9059:
Author: ASF GitHub Bot
Created on: 09/Jan/20 03:10
Start Date: 09/Jan/20 03:10
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10521:
[BEAM-9059] Use string constants in PTransformTranslation instead of getUrn()
URL: https://github.com/apache/beam/pull/10521#discussion_r364542265
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
##
@@ -45,7 +44,7 @@
* to associate each input with its output. The nonce is represented as an
opaque set of bytes.
*/
public class WindowMappingFnRunner {
- static final String URN =
BeamUrns.getUrn(StandardPTransforms.Primitives.MAP_WINDOWS);
+ static final String URN = PTransformTranslation.MERGE_WINDOWS_TRANSFORM_URN;
Review comment:
should be `MAP_WINDOWS_TRANSFORM_URN`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368794)
Time Spent: 3h 10m (was: 3h)
> Migrate PTransformTranslation to use string constants
> -
>
> Key: BEAM-9059
> URL: https://issues.apache.org/jira/browse/BEAM-9059
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Trivial
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This allows for the values to be used within switch case statements.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9075) Join re-ordering doesn't work for ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9075?focusedWorklogId=368792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368792 ] ASF GitHub Bot logged work on BEAM-9075: Author: ASF GitHub Bot Created on: 09/Jan/20 03:04 Start Date: 09/Jan/20 03:04 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10542: [BEAM-9075] Disable JoinCommuteRule for ZetaSQL planner URL: https://github.com/apache/beam/pull/10542#issuecomment-572361171 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368792) Time Spent: 0.5h (was: 20m) > Join re-ordering doesn't work for ZetaSQL > - > > Key: BEAM-9075 > URL: https://issues.apache.org/jira/browse/BEAM-9075 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Affects Versions: 2.17.0 >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Join re-ordering can actually break queries in the ZetaSQL planner because > Calcite's JoinCommuteRule doesn't work when schemas have nested fields. It > only adjusts RexInputRef nodes and not RexFieldAccess. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9075) Join re-ordering doesn't work for ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9075?focusedWorklogId=368791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368791 ] ASF GitHub Bot logged work on BEAM-9075: Author: ASF GitHub Bot Created on: 09/Jan/20 03:04 Start Date: 09/Jan/20 03:04 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10542: [BEAM-9075] Disable JoinCommuteRule for ZetaSQL planner URL: https://github.com/apache/beam/pull/10542#issuecomment-572361149 R: @amaliujia CC: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368791) Time Spent: 20m (was: 10m) > Join re-ordering doesn't work for ZetaSQL > - > > Key: BEAM-9075 > URL: https://issues.apache.org/jira/browse/BEAM-9075 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Affects Versions: 2.17.0 >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Join re-ordering can actually break queries in the ZetaSQL planner because > Calcite's JoinCommuteRule doesn't work when schemas have nested fields. It > only adjusts RexInputRef nodes and not RexFieldAccess. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9075) Join re-ordering doesn't work for ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9075?focusedWorklogId=368790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368790 ] ASF GitHub Bot logged work on BEAM-9075: Author: ASF GitHub Bot Created on: 09/Jan/20 03:03 Start Date: 09/Jan/20 03:03 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #10542: [BEAM-9075] Disable JoinCommuteRule for ZetaSQL planner URL: https://github.com/apache/beam/pull/10542 Remove the `JoinCommuteRule` from rulesets used in the ZetaSQL planner. It currently doesn't work without struct flattening. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8481) Python 3.7 Postcommit test -- frequent timeouts
[ https://issues.apache.org/jira/browse/BEAM-8481?focusedWorklogId=368781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368781 ] ASF GitHub Bot logged work on BEAM-8481: Author: ASF GitHub Bot Created on: 09/Jan/20 02:51 Start Date: 09/Jan/20 02:51 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10378: [BEAM-8481] Fix a race condition in proto stubs generation. URL: https://github.com/apache/beam/pull/10378#issuecomment-572358372 Thanks, @ibzib for the fix for BEAM-9070! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368781) Time Spent: 5h 50m (was: 5h 40m) > Python 3.7 Postcommit test -- frequent timeouts > --- > > Key: BEAM-8481 > URL: https://issues.apache.org/jira/browse/BEAM-8481 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ahmet Altay >Assignee: Valentyn Tymofieiev >Priority: Critical > Time Spent: 5h 50m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_PostCommit_Python37/] – this suite > seemingly frequently timing out. Other suites are not affected by these > timeouts. From the history, the issues started before Oct 10 and we cannot > pinpoint because history is lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8481) Python 3.7 Postcommit test -- frequent timeouts
[ https://issues.apache.org/jira/browse/BEAM-8481?focusedWorklogId=368780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368780 ] ASF GitHub Bot logged work on BEAM-8481: Author: ASF GitHub Bot Created on: 09/Jan/20 02:51 Start Date: 09/Jan/20 02:51 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10378: [BEAM-8481] Fix a race condition in proto stubs generation. URL: https://github.com/apache/beam/pull/10378#issuecomment-572358229 PTAL @udim or @ibzib This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368780) Time Spent: 5h 40m (was: 5.5h) > Python 3.7 Postcommit test -- frequent timeouts > --- > > Key: BEAM-8481 > URL: https://issues.apache.org/jira/browse/BEAM-8481 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ahmet Altay >Assignee: Valentyn Tymofieiev >Priority: Critical > Time Spent: 5h 40m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_PostCommit_Python37/] – this suite > seemingly frequently timing out. Other suites are not affected by these > timeouts. From the history, the issues started before Oct 10 and we cannot > pinpoint because history is lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9075) Join re-ordering doesn't work for ZetaSQL
Brian Hulette created BEAM-9075: --- Summary: Join re-ordering doesn't work for ZetaSQL Key: BEAM-9075 URL: https://issues.apache.org/jira/browse/BEAM-9075 Project: Beam Issue Type: Bug Components: dsl-sql-zetasql Affects Versions: 2.17.0 Reporter: Brian Hulette Assignee: Brian Hulette Join re-ordering can actually break queries in the ZetaSQL planner because Calcite's JoinCommuteRule doesn't work when schemas have nested fields. It only adjusts RexInputRef nodes and not RexFieldAccess. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9074) Make it visible on Beam website that Beam signed a pledge to sunset Python 2 offering in 2020 and future versions of Beam will not support Python 2.
[ https://issues.apache.org/jira/browse/BEAM-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev updated BEAM-9074: -- Status: Open (was: Triage Needed) > Make it visible on Beam website that Beam signed a pledge to sunset Python 2 > offering in 2020 and future versions of Beam will not support Python 2. > - > > Key: BEAM-9074 > URL: https://issues.apache.org/jira/browse/BEAM-9074 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Valentyn Tymofieiev >Assignee: Rose Nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368778
]
ASF GitHub Bot logged work on BEAM-9059:
Author: ASF GitHub Bot
Created on: 09/Jan/20 02:31
Start Date: 09/Jan/20 02:31
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10521: [BEAM-9059]
Use string constants in PTransformTranslation instead of getUrn()
URL: https://github.com/apache/beam/pull/10521#discussion_r364535090
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
##
@@ -45,7 +44,7 @@
* to associate each input with its output. The nonce is represented as an
opaque set of bytes.
*/
public class WindowMappingFnRunner {
- static final String URN =
BeamUrns.getUrn(StandardPTransforms.Primitives.MAP_WINDOWS);
+ static final String URN = PTransformTranslation.MERGE_WINDOWS_TRANSFORM_URN;
Review comment:
Is this change intended?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368778)
Time Spent: 3h (was: 2h 50m)
> Migrate PTransformTranslation to use string constants
> -
>
> Key: BEAM-9059
> URL: https://issues.apache.org/jira/browse/BEAM-9059
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Trivial
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This allows for the values to be used within switch case statements.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368775 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:26 Start Date: 09/Jan/20 02:26 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#issuecomment-572352389 btw, I made some edits to my answers to clarify them, so you should continue the review via github rather than email. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368775) Time Spent: 40h 50m (was: 40h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368773 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:25 Start Date: 09/Jan/20 02:25 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364532942 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -907,7 +907,8 @@ def encode_to_stream(self, value, out, nested): buffer = create_OutputStream() if (self._write_state is not None and out.size() - start_size > self._write_state_threshold): -tail = (value_iter[index + 1:] if isinstance(value, (list, tuple)) +tail = (value_iter[index + 1:] +if isinstance(value_iter, (list, tuple)) Review comment: It protects against this error: ``` apache_beam/coders/coder_impl.py:910: error: Value of type "Union[Iterator[Any], List[Any], Tuple[Any, ...]]" is not indexable [index] ``` The original code was using the type of `value` to implicitly determine the type of `value_iter`. This confuses humans and linters :) By using `isinstance(value_iter, ...)` instead, mypy knows to reduce its type from `Union[Iterator[Any], List[Any], Tuple[Any, ...]]` down to `Union[List[Any], Tuple[Any, ...]]` before the `__getitem__` code is checked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368773) Time Spent: 40.5h (was: 40h 20m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40.5h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368774 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:25 Start Date: 09/Jan/20 02:25 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364532942 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -907,7 +907,8 @@ def encode_to_stream(self, value, out, nested): buffer = create_OutputStream() if (self._write_state is not None and out.size() - start_size > self._write_state_threshold): -tail = (value_iter[index + 1:] if isinstance(value, (list, tuple)) +tail = (value_iter[index + 1:] +if isinstance(value_iter, (list, tuple)) Review comment: It protects against this error: ``` apache_beam/coders/coder_impl.py:910: error: Value of type "Union[Iterator[Any], List[Any], Tuple[Any, ...]]" is not indexable [index] ``` The original code was using the type of `value` to implicitly determine the type of `value_iter`. This confuses humans and linters :) By using `isinstance` against `value_iter` instead, mypy knows to reduce its type from `Union[Iterator[Any], List[Any], Tuple[Any, ...]]` down to `Union[List[Any], Tuple[Any, ...]]` before the `__getitem__` code is checked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368774) Time Spent: 40h 40m (was: 40.5h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368772 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:25 Start Date: 09/Jan/20 02:25 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364532942 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -907,7 +907,8 @@ def encode_to_stream(self, value, out, nested): buffer = create_OutputStream() if (self._write_state is not None and out.size() - start_size > self._write_state_threshold): -tail = (value_iter[index + 1:] if isinstance(value, (list, tuple)) +tail = (value_iter[index + 1:] +if isinstance(value_iter, (list, tuple)) Review comment: It protects against this error: ``` apache_beam/coders/coder_impl.py:910: error: Value of type "Union[Iterator[Any], List[Any], Tuple[Any, ...]]" is not indexable [index] ``` The original code was using the type of `value` to implicitly determine the type of `value_iter`. This confuses humans and linters :) By using `isinstance(value_iter, ...)` mypy knows to reduce its type from `Union[Iterator[Any], List[Any], Tuple[Any, ...]]` down to `Union[List[Any], Tuple[Any, ...]]` before the `__getitem__` code is checked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368772) Time Spent: 40h 20m (was: 40h 10m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40h 20m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368771 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:24 Start Date: 09/Jan/20 02:24 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364532942 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -907,7 +907,8 @@ def encode_to_stream(self, value, out, nested): buffer = create_OutputStream() if (self._write_state is not None and out.size() - start_size > self._write_state_threshold): -tail = (value_iter[index + 1:] if isinstance(value, (list, tuple)) +tail = (value_iter[index + 1:] +if isinstance(value_iter, (list, tuple)) Review comment: It protects against this error: ``` apache_beam/coders/coder_impl.py:910: error: Value of type "Union[Iterator[Any], List[Any], Tuple[Any, ...]]" is not indexable [index] ``` The original code was using the type of `value` to implicitly determine the type of `value_iter`. This confuses humans and linters :) By using `value_iter` mypy knows to reduce `Union[Iterator[Any], List[Any], Tuple[Any, ...]]` down to `Union[List[Any], Tuple[Any, ...]]` before the `__getitem__` code is checked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368771) Time Spent: 40h 10m (was: 40h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368770 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:23 Start Date: 09/Jan/20 02:23 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#issuecomment-572351739 @udim thanks for the review! very good questions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368770) Time Spent: 40h (was: 39h 50m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368769=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368769 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:20 Start Date: 09/Jan/20 02:20 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364533360 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -879,6 +892,7 @@ def process(self, windowed_value): return self.do_fn_invoker.invoke_process(windowed_value) except BaseException as exn: self._reraise_augmented(exn) + return None Review comment: same :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368769) Time Spent: 39h 50m (was: 39h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 39h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368768 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:20 Start Date: 09/Jan/20 02:20 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364533315 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -793,6 +804,8 @@ def current_element_progress(self): restriction_tracker = self.threadsafe_restriction_tracker if restriction_tracker: return restriction_tracker.current_progress() +else: + return None Review comment: yup, required This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368768) Time Spent: 39h 40m (was: 39.5h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 39h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9074) Make it visible on Beam website that Beam signed a pledge to sunset Python 2 offering in 2020 and future versions of Beam will not support Python 2.
Valentyn Tymofieiev created BEAM-9074: - Summary: Make it visible on Beam website that Beam signed a pledge to sunset Python 2 offering in 2020 and future versions of Beam will not support Python 2. Key: BEAM-9074 URL: https://issues.apache.org/jira/browse/BEAM-9074 Project: Beam Issue Type: Improvement Components: website Reporter: Valentyn Tymofieiev Assignee: Rose Nguyen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368767 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:19 Start Date: 09/Jan/20 02:19 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364532942 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -907,7 +907,8 @@ def encode_to_stream(self, value, out, nested): buffer = create_OutputStream() if (self._write_state is not None and out.size() - start_size > self._write_state_threshold): -tail = (value_iter[index + 1:] if isinstance(value, (list, tuple)) +tail = (value_iter[index + 1:] +if isinstance(value_iter, (list, tuple)) Review comment: It protects against this error: ``` apache_beam/coders/coder_impl.py:910: error: Value of type "Union[Iterator[Any], List[Any], Tuple[Any, ...]]" is not indexable [index] ``` The code is using the type of `value` to implicitly determine the type of `value_iter`. This confuses humans and linters :) By using `value_iter` mypy knows to reduce `Union[Iterator[Any], List[Any], Tuple[Any, ...]]` down to `Union[List[Any], Tuple[Any, ...]]` before the `__getitem__` code is checked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368767) Time Spent: 39.5h (was: 39h 20m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 39.5h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9073) PipelineVisitor is topologically-order dependent
[ https://issues.apache.org/jira/browse/BEAM-9073?focusedWorklogId=368765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368765 ] ASF GitHub Bot logged work on BEAM-9073: Author: ASF GitHub Bot Created on: 09/Jan/20 02:15 Start Date: 09/Jan/20 02:15 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #10541: [BEAM-9073] Fixes order-dependence in PipelineVisitor URL: https://github.com/apache/beam/pull/10541 The Python PipelineVisitor is topologically-order dependent and can visit the same transform multiple times. The fix is to journal each transform we visit. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9073) PipelineVisitor is topologically-order dependent
[ https://issues.apache.org/jira/browse/BEAM-9073?focusedWorklogId=368766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368766 ] ASF GitHub Bot logged work on BEAM-9073: Author: ASF GitHub Bot Created on: 09/Jan/20 02:15 Start Date: 09/Jan/20 02:15 Worklog Time Spent: 10m Work Description: rohdesamuel commented on issue #10541: [BEAM-9073] Fixes order-dependence in PipelineVisitor URL: https://github.com/apache/beam/pull/10541#issuecomment-572349674 R: @lukecwik can you review this please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368766) Time Spent: 20m (was: 10m) > PipelineVisitor is topologically-order dependent > > > Key: BEAM-9073 > URL: https://issues.apache.org/jira/browse/BEAM-9073 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The Python PipelineVisitor is topologically-order dependent and can visit the > same transform multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368764 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:13 Start Date: 09/Jan/20 02:13 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364531963 ## File path: sdks/python/apache_beam/pvalue.py ## @@ -82,7 +82,7 @@ class PValue(object): """ def __init__(self, - pipeline, # type: Pipeline + pipeline, # type: Optional[Pipeline] Review comment: Yes, here: ```python class PCollection(PValue, Generic[T]): [...] @staticmethod def from_runner_api(proto, context): # type: (beam_runner_api_pb2.PCollection, PipelineContext) -> PCollection # Producer and tag will be filled in later, the key point is that the # same object is returned for the same pcollection id. return PCollection( None, element_type=context.element_type_from_coder_id(proto.coder_id), windowing=context.windowing_strategies.get_by_id( proto.windowing_strategy_id), is_bounded=proto.is_bounded == beam_runner_api_pb2.IsBounded.BOUNDED) ``` It would be nice if we could get rid of this case somehow, because by making this optional we have to deal with the possibility of `pvalue.pipeline` being `None` throughout the code base. I went back and forth on whether to make the arg optional or simply ignore the error in the method above, but I think I decided that the method above was a common case and thus we needed the protection against None-values throughout the code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368764) Time Spent: 39h 20m (was: 39h 10m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 39h 20m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9073) PipelineVisitor is topologically-order dependent
Sam Rohde created BEAM-9073: --- Summary: PipelineVisitor is topologically-order dependent Key: BEAM-9073 URL: https://issues.apache.org/jira/browse/BEAM-9073 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Sam Rohde Assignee: Sam Rohde The Python PipelineVisitor is topologically-order dependent and can visit the same transform multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368762 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 02:07 Start Date: 09/Jan/20 02:07 Worklog Time Spent: 10m Work Description: angoenka commented on issue #10190: [BEAM-8575] Added two unit tests to CombineTest class to test that Co… URL: https://github.com/apache/beam/pull/10190#issuecomment-572347542 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368762) Time Spent: 43h 20m (was: 43h 10m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 43h 20m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368761 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 02:06 Start Date: 09/Jan/20 02:06 Worklog Time Spent: 10m Work Description: angoenka commented on issue #10190: [BEAM-8575] Added two unit tests to CombineTest class to test that Co… URL: https://github.com/apache/beam/pull/10190#issuecomment-572347325 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368761) Time Spent: 43h 10m (was: 43h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 43h 10m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[ https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368757 ] ASF GitHub Bot logged work on BEAM-9059: Author: ASF GitHub Bot Created on: 09/Jan/20 02:05 Start Date: 09/Jan/20 02:05 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10521: [BEAM-9059] Use string constants in PTransformTranslation instead of getUrn() URL: https://github.com/apache/beam/pull/10521#issuecomment-572347043 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368757) Time Spent: 2.5h (was: 2h 20m) > Migrate PTransformTranslation to use string constants > - > > Key: BEAM-9059 > URL: https://issues.apache.org/jira/browse/BEAM-9059 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Trivial > Time Spent: 2.5h > Remaining Estimate: 0h > > This allows for the values to be used within switch case statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[ https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368758 ] ASF GitHub Bot logged work on BEAM-9059: Author: ASF GitHub Bot Created on: 09/Jan/20 02:05 Start Date: 09/Jan/20 02:05 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10521: [BEAM-9059] Use string constants in PTransformTranslation instead of getUrn() URL: https://github.com/apache/beam/pull/10521#issuecomment-572347074 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368758) Time Spent: 2h 40m (was: 2.5h) > Migrate PTransformTranslation to use string constants > - > > Key: BEAM-9059 > URL: https://issues.apache.org/jira/browse/BEAM-9059 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Trivial > Time Spent: 2h 40m > Remaining Estimate: 0h > > This allows for the values to be used within switch case statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[ https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368760 ] ASF GitHub Bot logged work on BEAM-9059: Author: ASF GitHub Bot Created on: 09/Jan/20 02:05 Start Date: 09/Jan/20 02:05 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10521: [BEAM-9059] Use string constants in PTransformTranslation instead of getUrn() URL: https://github.com/apache/beam/pull/10521#issuecomment-572346917 LGTM. +1 to merge once the jenkins turns green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368760) Time Spent: 2h 50m (was: 2h 40m) > Migrate PTransformTranslation to use string constants > - > > Key: BEAM-9059 > URL: https://issues.apache.org/jira/browse/BEAM-9059 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Trivial > Time Spent: 2h 50m > Remaining Estimate: 0h > > This allows for the values to be used within switch case statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[ https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368756 ] ASF GitHub Bot logged work on BEAM-9059: Author: ASF GitHub Bot Created on: 09/Jan/20 02:05 Start Date: 09/Jan/20 02:05 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10521: [BEAM-9059] Use string constants in PTransformTranslation instead of getUrn() URL: https://github.com/apache/beam/pull/10521#issuecomment-572346993 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368756) Time Spent: 2h 20m (was: 2h 10m) > Migrate PTransformTranslation to use string constants > - > > Key: BEAM-9059 > URL: https://issues.apache.org/jira/browse/BEAM-9059 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Trivial > Time Spent: 2h 20m > Remaining Estimate: 0h > > This allows for the values to be used within switch case statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants
[ https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=368755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368755 ] ASF GitHub Bot logged work on BEAM-9059: Author: ASF GitHub Bot Created on: 09/Jan/20 02:04 Start Date: 09/Jan/20 02:04 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10521: [BEAM-9059] Use string constants in PTransformTranslation instead of getUrn() URL: https://github.com/apache/beam/pull/10521#issuecomment-572346917 LGTM. +1 to merge once the jenkins turns green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368755) Time Spent: 2h 10m (was: 2h) > Migrate PTransformTranslation to use string constants > - > > Key: BEAM-9059 > URL: https://issues.apache.org/jira/browse/BEAM-9059 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Trivial > Time Spent: 2h 10m > Remaining Estimate: 0h > > This allows for the values to be used within switch case statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368754 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 02:04 Start Date: 09/Jan/20 02:04 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364518511 ## File path: sdks/python/apache_beam/utils/profiler.py ## @@ -105,6 +105,7 @@ def create_profiler(profile_id, **kwargs): if random.random() < options.profile_sample_rate: return Profile(profile_id, options.profile_location, **kwargs) return create_profiler +return None Review comment: if a function returns an optional result (i.e. `Union[None, X]`) then mypy requires that you explicitly return `None` for the optional case(s). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368754) Time Spent: 39h 10m (was: 39h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 39h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368753
]
ASF GitHub Bot logged work on BEAM-7746:
Author: ASF GitHub Bot
Created on: 09/Jan/20 02:02
Start Date: 09/Jan/20 02:02
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10367: [BEAM-7746]
Add python type hints (part 2)
URL: https://github.com/apache/beam/pull/10367#discussion_r364529725
##
File path: sdks/python/apache_beam/transforms/sideinputs.py
##
@@ -58,6 +59,16 @@ def map_via_end(source_window):
return map_via_end
+def get_sideinput_index(tag):
+ # type: (str) -> int
+ match = re.match('side([0-9]+)(-.*)?$', tag,
+ re.DOTALL)
Review comment:
There were three usages of this regex. One used `re.DOTALL` so I went with
the most permissive.
- transforms.core:
```python
int(re.match('side([0-9]+)(-.*)?$', tag).group(1))
```
- runners.worker.bundle_processor:
```python
int(re.match('side([0-9]+)(-.*)?$', tag, re.DOTALL).group(1))
```
- pipeline:
```python
int(re.match('side([0-9]+)(-.*)?$', tag).group(1))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368753)
Time Spent: 39h (was: 38h 50m)
> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 39h
> Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484
> type hints so that I can clearly see what types are required, get completion
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here: [https://github.com/apache/beam/pull/9056]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368752
]
ASF GitHub Bot logged work on BEAM-7746:
Author: ASF GitHub Bot
Created on: 09/Jan/20 02:01
Start Date: 09/Jan/20 02:01
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10367: [BEAM-7746]
Add python type hints (part 2)
URL: https://github.com/apache/beam/pull/10367#discussion_r364529725
##
File path: sdks/python/apache_beam/transforms/sideinputs.py
##
@@ -58,6 +59,16 @@ def map_via_end(source_window):
return map_via_end
+def get_sideinput_index(tag):
+ # type: (str) -> int
+ match = re.match('side([0-9]+)(-.*)?$', tag,
+ re.DOTALL)
Review comment:
There were three usages of this regex. One used `re.DOTALL` so I went with
the most permissive.
- transforms.core:
```python
int(re.match('side([0-9]+)(-.*)?$', tag).group(1))
```
- runners.worker.bundle_processor:
```python
int(re.match('side([0-9]+)(-.*)?$', tag, re.DOTALL).group(1))
```
- pipeline:
```python
int(re.match('side([0-9]+)(-.*)?$', tag).group(1))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368752)
Time Spent: 38h 50m (was: 38h 40m)
> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 38h 50m
> Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484
> type hints so that I can clearly see what types are required, get completion
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here: [https://github.com/apache/beam/pull/9056]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368747 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 01:52 Start Date: 09/Jan/20 01:52 Worklog Time Spent: 10m Work Description: bumblebee-coming commented on issue #10190: [BEAM-8575] Added two unit tests to CombineTest class to test that Co… URL: https://github.com/apache/beam/pull/10190#issuecomment-572344103 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368747) Time Spent: 43h (was: 42h 50m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 43h > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368746 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 01:50 Start Date: 09/Jan/20 01:50 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364527420 ## File path: sdks/python/apache_beam/portability/__init__.py ## @@ -17,3 +17,18 @@ """For internal use only; no backwards-compatibility guarantees.""" from __future__ import absolute_import + +from typing import TYPE_CHECKING +from typing import NamedTuple + +if TYPE_CHECKING: + from apache_beam.portability.api import metrics_pb2 + + +PropertiesFromEnumValue = NamedTuple( Review comment: yeah, I typically follow that convention as well. In this case, putting `PropertiesFromEnumValue ` in common_urns.py would cause an import cycle, because it imports the _urn.py files which need access to `PropertiesFromEnumValue`. I considered putting `PropertiesFromEnumValue` in a new module, but it felt like overkill. Happy to do so, if it's preferable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368746) Time Spent: 38h 40m (was: 38.5h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 38h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368745 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 01:50 Start Date: 09/Jan/20 01:50 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364527420 ## File path: sdks/python/apache_beam/portability/__init__.py ## @@ -17,3 +17,18 @@ """For internal use only; no backwards-compatibility guarantees.""" from __future__ import absolute_import + +from typing import TYPE_CHECKING +from typing import NamedTuple + +if TYPE_CHECKING: + from apache_beam.portability.api import metrics_pb2 + + +PropertiesFromEnumValue = NamedTuple( Review comment: yeah, I typically follow that convention as well. In this case, common_urns.py would cause an import cycle, because it imports the _urn.py files which need access to `PropertiesFromEnumValue`. I considered putting `PropertiesFromEnumValue` in a new module, but it felt like overkill. Happy to do so, if it's preferable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368745) Time Spent: 38.5h (was: 38h 20m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 38.5h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368744 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 09/Jan/20 01:43 Start Date: 09/Jan/20 01:43 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-572341918 Thank you for all of your review and merging! I have bring up the discussion about gRPC Vendor Release, the detail can be found in [1]. We can open the second half of the upgrade PR once the release is out. [1] https://lists.apache.org/thread.html/7e4de745a224efc24fd5c82737160ef050cbd18a99b45c4e3434fd56%40%3Cdev.beam.apache.org%3E This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368744) Time Spent: 6h (was: 5h 50m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 6h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368743 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 01:41 Start Date: 09/Jan/20 01:41 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10277: [BEAM-8575] Reenable passing VR tests. URL: https://github.com/apache/beam/pull/10277#issuecomment-572341387 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368743) Time Spent: 42h 50m (was: 42h 40m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 42h 50m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9051) BigQueryUtils toBeamRow to support GenericRecords with millisecond precision timestamps
[ https://issues.apache.org/jira/browse/BEAM-9051?focusedWorklogId=368742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368742 ] ASF GitHub Bot logged work on BEAM-9051: Author: ASF GitHub Bot Created on: 09/Jan/20 01:35 Start Date: 09/Jan/20 01:35 Worklog Time Spent: 10m Work Description: RyanBerti commented on issue #10540: [BEAM-9051] BigQueryUtils toBeamRow support the Avro timestamp-millis logical type URL: https://github.com/apache/beam/pull/10540#issuecomment-572339923 R: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368742) Time Spent: 20m (was: 10m) > BigQueryUtils toBeamRow to support GenericRecords with millisecond precision > timestamps > --- > > Key: BEAM-9051 > URL: https://issues.apache.org/jira/browse/BEAM-9051 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Ryan Berti >Assignee: Ryan Berti >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Currently, BigQueryUtils assumes all Avro GenericRecords which utilize > timestamp fields include timestamp data in microseconds when converting from > GenericRecords to Beam Rows > ([https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L568]). > The implementation should also support timestamp data in milliseconds; this > should be feasible by referencing the LogicalType associated with the avro > field to determine if the underlying value is represented in millis or micros. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9051) BigQueryUtils toBeamRow to support GenericRecords with millisecond precision timestamps
[ https://issues.apache.org/jira/browse/BEAM-9051?focusedWorklogId=368741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368741 ] ASF GitHub Bot logged work on BEAM-9051: Author: ASF GitHub Bot Created on: 09/Jan/20 01:34 Start Date: 09/Jan/20 01:34 Worklog Time Spent: 10m Work Description: RyanBerti commented on pull request #10540: [BEAM-9051] BigQueryUtils toBeamRow support the Avro timestamp-millis logical type URL: https://github.com/apache/beam/pull/10540 This change allows users with Avro GenericRecords that utilize non-microsecond timestamp values to utilize BigQueryUtils.toBeamRow to convert those GenericRecords into Beam Rows. This functionality was required for our implementation, as were generating GenericRecords from Scala case classes and didn't have direct access to the implementation which chose the precision of the timestamp value to be generated based on Instant fields (these Instant fields were converted to milliseconds in our case). Testing shows that the functionality works for nested rows and arrays. The implementation should not affect any existing use cases as it preserves the original API. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
[jira] [Commented] (BEAM-8496) remove SDF translators in flink streaming transform translator
[ https://issues.apache.org/jira/browse/BEAM-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011288#comment-17011288 ] Kai Jiang commented on BEAM-8496: - For failed nexmark on flink job, it was fixed by [https://github.com/apache/beam/commit/4af507f567605723af940c2af3b2912d0948adf5] actually. > remove SDF translators in flink streaming transform translator > -- > > Key: BEAM-8496 > URL: https://issues.apache.org/jira/browse/BEAM-8496 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > > Since URN of SDF has been moved to runners-core-construction-java, we need to > remove it. > Otherwise, in failed nexmark Jenkins > [job|https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/4128/console], > it causes duplicated transformer registered in > [PTransformTranslation.KnownTransformPayloadTranslator()|https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java#L290] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=368734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368734 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 09/Jan/20 01:16 Start Date: 09/Jan/20 01:16 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #10536: [BEAM-7861] Add direct_running_mode option for direct runners to switch between multi_threading and multi_processing easily URL: https://github.com/apache/beam/pull/10536#discussion_r364520622 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -438,6 +438,12 @@ def _add_argparse_args(cls, parser): type=int, default=1, help='number of parallel running workers.') +parser.add_argument( +'--direct_running_mode', +default=None, Review comment: Would it make sense to add another choice for `EmbeddedWorkerHandler` and change the default to that, instead of defaulting to `None`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368734) Time Spent: 40m (was: 0.5h) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 40m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368636 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 01:07 Start Date: 09/Jan/20 01:07 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364518632 ## File path: sdks/python/gen_protos.py ## @@ -47,6 +50,164 @@ ] +def generate_urn_files(log, out_dir): + """ + Create a pyi stub for common_urns Review comment: ah yes. sorry. will also add an explanation of how this works. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368636) Time Spent: 38h 20m (was: 38h 10m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 38h 20m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368634=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368634 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 01:06 Start Date: 09/Jan/20 01:06 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364518511 ## File path: sdks/python/apache_beam/utils/profiler.py ## @@ -105,6 +105,7 @@ def create_profiler(profile_id, **kwargs): if random.random() < options.profile_sample_rate: return Profile(profile_id, options.profile_location, **kwargs) return create_profiler +return None Review comment: if a function returns an optional result (i.e. `Union[None, X]`) then mypy requires that you explicitly return `None` for the optional case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368634) Time Spent: 38h 10m (was: 38h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 38h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8882) Allow Dataflow to automatically choose portability or not.
[ https://issues.apache.org/jira/browse/BEAM-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011180#comment-17011180 ] Robert Bradshaw commented on BEAM-8882: --- I would word this as "beam.Create no longer preserves order unless reshuffle=False is passed in as an argument." It's still odd to say, as PCollections are, generically, unordered, but in this case people may have assumed order + fusion. > Allow Dataflow to automatically choose portability or not. > -- > > Key: BEAM-8882 > URL: https://issues.apache.org/jira/browse/BEAM-8882 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Critical > Fix For: 2.18.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > We would like the Dataflow service to be able to automatically choose whether > to run pipelines in a portable way. In order to do this, we need to provide > more information even if portability is not explicitly requested. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368537 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:56 Start Date: 09/Jan/20 00:56 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364516322 ## File path: sdks/python/gen_protos.py ## @@ -47,6 +50,164 @@ ] +def generate_urn_files(log, out_dir): Review comment: Here is `beam_job_api_pb2_urn.py`: ```python from builtins import object from .. import PropertiesFromEnumValue from . import metrics_pb2 EMPTY_MONITORING_INFO_LABEL_PROPS = metrics_pb2.MonitoringInfoLabelProps() EMPTY_MONITORING_INFO_SPEC = metrics_pb2.MonitoringInfoSpec() class JobMessage(object): class MessageImportance(object): MESSAGE_IMPORTANCE_UNSPECIFIED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) JOB_MESSAGE_DEBUG = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) JOB_MESSAGE_DETAILED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) JOB_MESSAGE_BASIC = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) JOB_MESSAGE_WARNING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) JOB_MESSAGE_ERROR = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) class JobState(object): class Enum(object): UNSPECIFIED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) STOPPED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) RUNNING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) DONE = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) FAILED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) CANCELLED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) UPDATED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) DRAINING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) DRAINED = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) STARTING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) CANCELLING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) UPDATING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) class PipelineOptionType(object): class Enum(object): STRING = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) BOOLEAN = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) INTEGER = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) NUMBER = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) ARRAY = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) OBJECT = PropertiesFromEnumValue(u'', u'', EMPTY_MONITORING_INFO_SPEC, EMPTY_MONITORING_INFO_LABEL_PROPS) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368537) Time Spent: 38h (was: 37h 50m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 38h > Remaining Estimate: 0h > > As a developer of the beam source code, I
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368534 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 09/Jan/20 00:50 Start Date: 09/Jan/20 00:50 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368534) Time Spent: 5h 40m (was: 5.5h) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368535 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 09/Jan/20 00:50 Start Date: 09/Jan/20 00:50 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-572328778 Linkage checker reported no errors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368535) Time Spent: 5h 50m (was: 5h 40m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9065) Spark runner accumulates metrics (incorrectly) between runs
[ https://issues.apache.org/jira/browse/BEAM-9065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011100#comment-17011100 ] Udi Meiri commented on BEAM-9065: - 2.18.0 has been cut 5 weeks ago. Let's push this fix to 2.19.0 and this as a Known Issue to the release notes. > Spark runner accumulates metrics (incorrectly) between runs > --- > > Key: BEAM-9065 > URL: https://issues.apache.org/jira/browse/BEAM-9065 > Project: Beam > Issue Type: Bug > Components: runner-spark >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Fix For: 2.18.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When pipeline.run() is called, MetricsAccumulator (wrapper of > MetricsContainerStepMap spark accumulator) is initialized. Spark needs this > class to be a singleton for failover. The problem is that when several > pipelines are run inside the same JVM, the initialization of > MetricsAccumulator singleton does not reset the underlying spark accumulator > causing metrics to be accumulated between runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8882) Allow Dataflow to automatically choose portability or not.
[ https://issues.apache.org/jira/browse/BEAM-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011099#comment-17011099 ] Udi Meiri commented on BEAM-8882: - I think the best thing here is to add a release note telling: Users running pipeline with beam.Create using DirectRunner (such as in tests) may need to add the keyword reshuffle=False to Create() if they're seeing weird behavior. Reshuffle use in Create has the effect of randomizing the order of the resulting PCollection. > Allow Dataflow to automatically choose portability or not. > -- > > Key: BEAM-8882 > URL: https://issues.apache.org/jira/browse/BEAM-8882 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Critical > Fix For: 2.18.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > We would like the Dataflow service to be able to automatically choose whether > to run pipelines in a portable way. In order to do this, we need to provide > more information even if portability is not explicitly requested. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8951) Stop using nose in load tests
[ https://issues.apache.org/jira/browse/BEAM-8951?focusedWorklogId=368527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368527 ] ASF GitHub Bot logged work on BEAM-8951: Author: ASF GitHub Bot Created on: 09/Jan/20 00:35 Start Date: 09/Jan/20 00:35 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #10435: [BEAM-8951] Stop using nose in load tests URL: https://github.com/apache/beam/pull/10435#issuecomment-572325201 Run Load Tests Java Combine Portable Flink Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368527) Time Spent: 6h (was: 5h 50m) > Stop using nose in load tests > - > > Key: BEAM-8951 > URL: https://issues.apache.org/jira/browse/BEAM-8951 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Fix For: Not applicable > > Time Spent: 6h > Remaining Estimate: 0h > > The community is considering moving away from nose to pytest: > https://issues.apache.org/jira/browse/BEAM-3713. We should change the way of > running Python load tests: instead of being subclasses of > `unittest.TestCase`, they could be plain Python scripts, just like wordcount > examples. This will bring one additional benefit: _LOAD_TEST_ENABLED_ guard > will be no longer needed and could be safely removed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368524 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 00:34 Start Date: 09/Jan/20 00:34 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#issuecomment-572324905 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368524) Time Spent: 42.5h (was: 42h 20m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 42.5h > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368523 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 00:34 Start Date: 09/Jan/20 00:34 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10190: [BEAM-8575] Added two unit tests to CombineTest class to test that Co… URL: https://github.com/apache/beam/pull/10190#issuecomment-572324861 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368523) Time Spent: 42h 20m (was: 42h 10m) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 42h 20m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368526 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 00:34 Start Date: 09/Jan/20 00:34 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#issuecomment-572324976 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368526) Time Spent: 42h 40m (was: 42.5h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 42h 40m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368517 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364492849 ## File path: sdks/python/apache_beam/utils/profiler.py ## @@ -105,6 +105,7 @@ def create_profiler(profile_id, **kwargs): if random.random() < options.profile_sample_rate: return Profile(profile_id, options.profile_location, **kwargs) return create_profiler +return None Review comment: Isn't this redundant? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368517) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368513 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364402749 ## File path: sdks/python/gen_protos.py ## @@ -47,6 +50,164 @@ ] +def generate_urn_files(log, out_dir): + """ + Create a pyi stub for common_urns Review comment: Same comment from parent PR: Please update the docstring. IIUC, this does not generate a pyi stub, but several _urns.py files that are imported by common_urns. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368513) Time Spent: 37h 10m (was: 37h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368514
]
ASF GitHub Bot logged work on BEAM-7746:
Author: ASF GitHub Bot
Created on: 09/Jan/20 00:31
Start Date: 09/Jan/20 00:31
Worklog Time Spent: 10m
Work Description: udim commented on pull request #10367: [BEAM-7746] Add
python type hints (part 2)
URL: https://github.com/apache/beam/pull/10367#discussion_r364359100
##
File path: sdks/python/apache_beam/transforms/sideinputs.py
##
@@ -58,6 +59,16 @@ def map_via_end(source_window):
return map_via_end
+def get_sideinput_index(tag):
+ # type: (str) -> int
+ match = re.match('side([0-9]+)(-.*)?$', tag,
+ re.DOTALL)
Review comment:
Why `re.DOTALL`? It's not present in the code this function is replacing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368514)
Time Spent: 37h 20m (was: 37h 10m)
> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 37h 20m
> Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484
> type hints so that I can clearly see what types are required, get completion
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here: [https://github.com/apache/beam/pull/9056]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368516 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364397975 ## File path: sdks/python/apache_beam/portability/__init__.py ## @@ -17,3 +17,18 @@ """For internal use only; no backwards-compatibility guarantees.""" from __future__ import absolute_import + +from typing import TYPE_CHECKING +from typing import NamedTuple + +if TYPE_CHECKING: + from apache_beam.portability.api import metrics_pb2 + + +PropertiesFromEnumValue = NamedTuple( Review comment: We typically don't put anything beyond import statements in `__init__.py` files. Could this be put in common_urns.py? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368516) Time Spent: 37h 40m (was: 37.5h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368520 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364483283 ## File path: sdks/python/apache_beam/pvalue.py ## @@ -82,7 +82,7 @@ class PValue(object): """ def __init__(self, - pipeline, # type: Pipeline + pipeline, # type: Optional[Pipeline] Review comment: Is (1) above incorrect? Are some PValues initialized without a pipeline? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368520) Time Spent: 37h 50m (was: 37h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368519 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364501177 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -879,6 +892,7 @@ def process(self, windowed_value): return self.do_fn_invoker.invoke_process(windowed_value) except BaseException as exn: self._reraise_augmented(exn) + return None Review comment: Same This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368519) Time Spent: 37h 50m (was: 37h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368515 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364472090 ## File path: sdks/python/gen_protos.py ## @@ -47,6 +50,164 @@ ] +def generate_urn_files(log, out_dir): Review comment: Could you post somewhere the output files? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368515) Time Spent: 37.5h (was: 37h 20m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37.5h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368518 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364501015 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -793,6 +804,8 @@ def current_element_progress(self): restriction_tracker = self.threadsafe_restriction_tracker if restriction_tracker: return restriction_tracker.current_progress() +else: + return None Review comment: Again, isn't this redundant? Does mypy require it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368518) Time Spent: 37h 50m (was: 37h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=368512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368512 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 09/Jan/20 00:31 Start Date: 09/Jan/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10367: [BEAM-7746] Add python type hints (part 2) URL: https://github.com/apache/beam/pull/10367#discussion_r364352387 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -907,7 +907,8 @@ def encode_to_stream(self, value, out, nested): buffer = create_OutputStream() if (self._write_state is not None and out.size() - start_size > self._write_state_threshold): -tail = (value_iter[index + 1:] if isinstance(value, (list, tuple)) +tail = (value_iter[index + 1:] +if isinstance(value_iter, (list, tuple)) Review comment: The replacement of value with value_iter seems to be logically equivalent, so what was mypy complaining about here? Was the change made to let mypy reason about the type of `tail`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368512) Time Spent: 37h (was: 36h 50m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 37h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests
[ https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=368507=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368507 ] ASF GitHub Bot logged work on BEAM-8575: Author: ASF GitHub Bot Created on: 09/Jan/20 00:09 Start Date: 09/Jan/20 00:09 Worklog Time Spent: 10m Work Description: bumblebee-coming commented on issue #10159: [BEAM-8575] Added a unit test to CombineTest class to test that Combi… URL: https://github.com/apache/beam/pull/10159#issuecomment-572318605 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368507) Time Spent: 42h 10m (was: 42h) > Add more Python validates runner tests > -- > > Key: BEAM-8575 > URL: https://issues.apache.org/jira/browse/BEAM-8575 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: wendy liu >Assignee: wendy liu >Priority: Major > Time Spent: 42h 10m > Remaining Estimate: 0h > > This is the umbrella issue to track the work of adding more Python tests to > improve test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9071) Ignore pytype untill the typing feature is fully developed
[ https://issues.apache.org/jira/browse/BEAM-9071?focusedWorklogId=368505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368505 ] ASF GitHub Bot logged work on BEAM-9071: Author: ASF GitHub Bot Created on: 08/Jan/20 23:56 Start Date: 08/Jan/20 23:56 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10533: [BEAM-9071] Ignore pytype before typing is fully developed. URL: https://github.com/apache/beam/pull/10533 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368505) Time Spent: 1.5h (was: 1h 20m) > Ignore pytype untill the typing feature is fully developed > -- > > Key: BEAM-9071 > URL: https://issues.apache.org/jira/browse/BEAM-9071 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Boyuan Zhang >Assignee: Udi Meiri >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Please remove all # pytype: skip-file after feature fully supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result
[ https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=368497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368497 ] ASF GitHub Bot logged work on BEAM-9027: Author: ASF GitHub Bot Created on: 08/Jan/20 23:38 Start Date: 08/Jan/20 23:38 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10539: [BEAM-9027] Unparse DOY/DOW/WEEK Enums properly for ZetaSQL URL: https://github.com/apache/beam/pull/10539#issuecomment-572310694 R: @robinyqiu cc: @amaliujia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368497) Time Spent: 3h 20m (was: 3h 10m) > [SQL] ZetaSQL unparsing should produce valid result > --- > > Key: BEAM-9027 > URL: https://issues.apache.org/jira/browse/BEAM-9027 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > * ZetaSQL does not recognize keyword INTERVAL > * Calcite cannot unparse RexNode back to bytes literal > * Calcite cannot unparse some floating point literals correctly > * Calcite cannot unparse some string literals correctly > * Calcite cannot unparse types correctly for CAST function -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result
[ https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=368496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368496 ] ASF GitHub Bot logged work on BEAM-9027: Author: ASF GitHub Bot Created on: 08/Jan/20 23:38 Start Date: 08/Jan/20 23:38 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10538: [BEAM-9027] Backport BigQuerySqlDialect fixes URL: https://github.com/apache/beam/pull/10538#issuecomment-572310661 R: @robinyqiu cc: @amaliujia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368496) Time Spent: 3h 10m (was: 3h) > [SQL] ZetaSQL unparsing should produce valid result > --- > > Key: BEAM-9027 > URL: https://issues.apache.org/jira/browse/BEAM-9027 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > * ZetaSQL does not recognize keyword INTERVAL > * Calcite cannot unparse RexNode back to bytes literal > * Calcite cannot unparse some floating point literals correctly > * Calcite cannot unparse some string literals correctly > * Calcite cannot unparse types correctly for CAST function -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8630) Prototype of BeamSQL Calc using ZetaSQL Expression Evaluator
[ https://issues.apache.org/jira/browse/BEAM-8630?focusedWorklogId=368493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368493 ] ASF GitHub Bot logged work on BEAM-8630: Author: ASF GitHub Bot Created on: 08/Jan/20 23:37 Start Date: 08/Jan/20 23:37 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10537: [BEAM-8630] Use column numbers for BeamZetaSqlCalRel URL: https://github.com/apache/beam/pull/10537#issuecomment-572310483 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368493) Time Spent: 5h 50m (was: 5h 40m) > Prototype of BeamSQL Calc using ZetaSQL Expression Evaluator > > > Key: BEAM-8630 > URL: https://issues.apache.org/jira/browse/BEAM-8630 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result
[ https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=368489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368489 ] ASF GitHub Bot logged work on BEAM-9027: Author: ASF GitHub Bot Created on: 08/Jan/20 23:35 Start Date: 08/Jan/20 23:35 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #10539: [BEAM-9027] Unparse DOY/DOW/WEEK Enums properly for ZetaSQL URL: https://github.com/apache/beam/pull/10539 Fixes enums for unparsing, passes compliance tests with ZetaSQL Calc. Unit tests aren't possible because of mismatches of these types being 0 or 1 indexed. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result
[ https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=368485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368485 ] ASF GitHub Bot logged work on BEAM-9027: Author: ASF GitHub Bot Created on: 08/Jan/20 23:34 Start Date: 08/Jan/20 23:34 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #10538: [BEAM-9027] Backport BigQuerySqlDialect apache/calcite@e63c252c URL: https://github.com/apache/beam/pull/10538 Pull in fixes from apache/calcite@e63c252c Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8630) Prototype of BeamSQL Calc using ZetaSQL Expression Evaluator
[ https://issues.apache.org/jira/browse/BEAM-8630?focusedWorklogId=368484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368484 ] ASF GitHub Bot logged work on BEAM-8630: Author: ASF GitHub Bot Created on: 08/Jan/20 23:33 Start Date: 08/Jan/20 23:33 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #10537: [BEAM-8630] Use column numbers for BeamZetaSqlCalRel URL: https://github.com/apache/beam/pull/10537 Use column numbers for BeamZetaSqlCalRel, this fixes multiple test failures. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Commented] (BEAM-1589) Add OnWindowExpiration method to Stateful DoFn
[ https://issues.apache.org/jira/browse/BEAM-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011071#comment-17011071 ] Reuven Lax commented on BEAM-1589: -- output timestamp support is now in (Flink support pending). I think we should just use the same timer that's used for garbage collection to fire OnWindowExpiration callbacks. In order to make this work, we need to modify this timer to have an output timestamp inside the window (maybe the end of the window?). > Add OnWindowExpiration method to Stateful DoFn > -- > > Key: BEAM-1589 > URL: https://issues.apache.org/jira/browse/BEAM-1589 > Project: Beam > Issue Type: New Feature > Components: runner-core, sdk-java-core >Reporter: Jingsong Lee >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > See BEAM-1517 > This allows the user to do some work before the state's garbage collection. > It seems kind of annoying, but on the other hand forgetting to set a final > timer to flush state is probably data loss most of the time. > FlinkRunner does this work very simply, but other runners, such as > DirectRunner, need to traverse all the states to do this, and maybe it's a > little hard. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9071) Ignore pytype untill the typing feature is fully developed
[ https://issues.apache.org/jira/browse/BEAM-9071?focusedWorklogId=368481=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368481 ] ASF GitHub Bot logged work on BEAM-9071: Author: ASF GitHub Bot Created on: 08/Jan/20 23:27 Start Date: 08/Jan/20 23:27 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10533: [BEAM-9071] Ignore pytype before typing is fully developed. URL: https://github.com/apache/beam/pull/10533#issuecomment-572307858 I did some manual check and changes. Waiting for test passing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368481) Time Spent: 1h 20m (was: 1h 10m) > Ignore pytype untill the typing feature is fully developed > -- > > Key: BEAM-9071 > URL: https://issues.apache.org/jira/browse/BEAM-9071 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Boyuan Zhang >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Please remove all # pytype: skip-file after feature fully supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8933) BigQuery IO should support read/write in Arrow format
[ https://issues.apache.org/jira/browse/BEAM-8933?focusedWorklogId=368482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368482 ] ASF GitHub Bot logged work on BEAM-8933: Author: ASF GitHub Bot Created on: 08/Jan/20 23:27 Start Date: 08/Jan/20 23:27 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #10369: [BEAM-8933] BigQueryIO Arrow for read URL: https://github.com/apache/beam/pull/10369#discussion_r364494539 ## File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageQuerySource.java ## @@ -57,6 +59,32 @@ priority, location, kmsKey, +format, +parseFn, +outputCoder, +bqServices); + } + + public static BigQueryStorageQuerySource create( Review comment: Good catch. The whole class is still experimental, so changes should be OK to make. On the other hand, I wonder if the sources should be marked `@Internal` or even made non-public with only the transform being public. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368482) Time Spent: 4h 50m (was: 4h 40m) > BigQuery IO should support read/write in Arrow format > - > > Key: BEAM-8933 > URL: https://issues.apache.org/jira/browse/BEAM-8933 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > As of right now BigQuery uses Avro format for reading and writing. > We should add a config to BigQueryIO to specify which format to use: Arrow or > Avro (with Avro as default). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8717) Beam Dependency Update Request: org.apache.commons:commons-lang3
[ https://issues.apache.org/jira/browse/BEAM-8717?focusedWorklogId=368478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368478 ] ASF GitHub Bot logged work on BEAM-8717: Author: ASF GitHub Bot Created on: 08/Jan/20 23:24 Start Date: 08/Jan/20 23:24 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10524: [BEAM-8717] Update commons-lang3 to version 3.9 URL: https://github.com/apache/beam/pull/10524#issuecomment-572305568 Should we be vendoring commons-lang3 instead of using the shaded classes from beam-sdks-java-core? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368478) Time Spent: 2h 20m (was: 2h 10m) > Beam Dependency Update Request: org.apache.commons:commons-lang3 > > > Key: BEAM-8717 > URL: https://issues.apache.org/jira/browse/BEAM-8717 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Ismaël Mejía >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:43:43.060362 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:11:02.203215 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:17:32.152530 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:16:47.060229 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:17:09.857528 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:12:21.614448 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:15:59.144846 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8717) Beam Dependency Update Request: org.apache.commons:commons-lang3
[
https://issues.apache.org/jira/browse/BEAM-8717?focusedWorklogId=368477=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368477
]
ASF GitHub Bot logged work on BEAM-8717:
Author: ASF GitHub Bot
Created on: 08/Jan/20 23:23
Start Date: 08/Jan/20 23:23
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10524: [BEAM-8717]
Update commons-lang3 to version 3.9
URL: https://github.com/apache/beam/pull/10524#discussion_r364493129
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -432,7 +432,7 @@ class BeamModulePlugin implements Plugin {
commons_compress:
"org.apache.commons:commons-compress:1.19",
commons_csv :
"org.apache.commons:commons-csv:1.4",
commons_io :
"commons-io:commons-io:2.6",
-commons_lang3 :
"org.apache.commons:commons-lang3:3.6",
+commons_lang3 :
"org.apache.commons:commons-lang3:3.9",
Review comment:
You could remove this from this list and have the full maven coordinates
appear in sdks/java/core/build.gradle since it is no longer used elsewhere.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 368477)
Time Spent: 2h 10m (was: 2h)
> Beam Dependency Update Request: org.apache.commons:commons-lang3
>
>
> Key: BEAM-8717
> URL: https://issues.apache.org/jira/browse/BEAM-8717
> Project: Beam
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Ismaël Mejía
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> - 2019-11-15 19:43:43.060362
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current version is 3.6. The latest version is 3.9
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-11-19 21:11:02.203215
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current version is 3.6. The latest version is 3.9
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-02 12:17:32.152530
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current version is 3.6. The latest version is 3.9
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-09 12:16:47.060229
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current version is 3.6. The latest version is 3.9
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-23 12:17:09.857528
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current version is 3.6. The latest version is 3.9
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-30 14:12:21.614448
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current version is 3.6. The latest version is 3.9
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2020-01-06 12:15:59.144846
> -
> Please consider upgrading the dependency
> org.apache.commons:commons-lang3.
> The current
[jira] [Work logged] (BEAM-8717) Beam Dependency Update Request: org.apache.commons:commons-lang3
[ https://issues.apache.org/jira/browse/BEAM-8717?focusedWorklogId=368476=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368476 ] ASF GitHub Bot logged work on BEAM-8717: Author: ASF GitHub Bot Created on: 08/Jan/20 23:19 Start Date: 08/Jan/20 23:19 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10524: [BEAM-8717] Update commons-lang3 to version 3.9 URL: https://github.com/apache/beam/pull/10524#issuecomment-572305568 Should we be vendoring commons-lang3 instead of using the shaded classes from beam-sdks-java-core? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368476) Time Spent: 2h (was: 1h 50m) > Beam Dependency Update Request: org.apache.commons:commons-lang3 > > > Key: BEAM-8717 > URL: https://issues.apache.org/jira/browse/BEAM-8717 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Ismaël Mejía >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > - 2019-11-15 19:43:43.060362 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:11:02.203215 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:17:32.152530 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:16:47.060229 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:17:09.857528 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:12:21.614448 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:15:59.144846 > - > Please consider upgrading the dependency > org.apache.commons:commons-lang3. > The current version is 3.6. The latest version is 3.9 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=368469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368469 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 08/Jan/20 23:04 Start Date: 08/Jan/20 23:04 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10536: [BEAM-7861] Add direct_running_mode option for direct runners to switch between multi_threading and multi_processing easily URL: https://github.com/apache/beam/pull/10536#issuecomment-572300418 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368469) Time Spent: 0.5h (was: 20m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=368467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368467 ] ASF GitHub Bot logged work on BEAM-5605: Author: ASF GitHub Bot Created on: 08/Jan/20 22:58 Start Date: 08/Jan/20 22:58 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10535: [BEAM-5605] Add support for executing pair with restriction, split restriction, split and size restriction, process element and restriction and process sized element and restriction within the Java SDK harness. URL: https://github.com/apache/beam/pull/10535#issuecomment-572298643 R: @youngoli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368467) Time Spent: 4.5h (was: 4h 20m) > Support Portable SplittableDoFn for batch > - > > Key: BEAM-5605 > URL: https://issues.apache.org/jira/browse/BEAM-5605 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Assignee: Luke Cwik >Priority: Major > Labels: portability > Time Spent: 4.5h > Remaining Estimate: 0h > > Roll-up item tracking work towards supporting portable SplittableDoFn for > batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=368465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368465 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 08/Jan/20 22:55 Start Date: 08/Jan/20 22:55 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10536: [BEAM-7861] Add direct_running_mode option for direct runners to switch between multi_threading and multi_processing easily URL: https://github.com/apache/beam/pull/10536#issuecomment-572297690 R: @robertwb Cc: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368465) Time Spent: 20m (was: 10m) > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 20m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hannah Jiang updated BEAM-7861: --- Fix Version/s: 2.19.0 > Make it easy to change between multi-process and multi-thread mode for Python > Direct runners > > > Key: BEAM-7861 > URL: https://issues.apache.org/jira/browse/BEAM-7861 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 10m > Remaining Estimate: 0h > > BEAM-3645 makes it possible to run a map task parallel. > However, users need to change runner when switch between multithreading and > multiprocessing mode. > We want to add a flag (ex: --use-multiprocess) to make the switch easy > without changing the runner each time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
