[jira] [Updated] (BEAM-4381) Provide DoFn to convert a Row (record with schema) to a javax.json.JsonObject
[ https://issues.apache.org/jira/browse/BEAM-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4381: Labels: stale-P2 (was: ) > Provide DoFn to convert a Row (record with schema) to a javax.json.JsonObject > - > > Key: BEAM-4381 > URL: https://issues.apache.org/jira/browse/BEAM-4381 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Priority: P2 > Labels: stale-P2 > > This task is about ensuring the generic record of beam can be converted to a > more mainstream format (JSON). The usage os javax.json.JsonObject is to > ensure it is portable and implementation independent and integrates th emost > probably with client libraries (most implementations have a bridge to the jsr > anyway) > > It will likely lead to provide a Dofn Row -> JsonObject and the opposite. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6840) PostRelease failing on Dataflow due to java.lang.NoSuchMethodError: DoFnSignature$ProcessElementMethod.getRowParameter(): DoFnSignature$Parameter$RowParameter
[
https://issues.apache.org/jira/browse/BEAM-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-6840:
Labels: (was: stale-assigned)
> PostRelease failing on Dataflow due to java.lang.NoSuchMethodError:
> DoFnSignature$ProcessElementMethod.getRowParameter():
> DoFnSignature$Parameter$RowParameter
> --
>
> Key: BEAM-6840
> URL: https://issues.apache.org/jira/browse/BEAM-6840
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Kenneth Knowles
>Priority: P1
>
> https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/551/console
> {code}
> java.lang.NoSuchMethodError:
> org.apache.beam.sdk.transforms.reflect.DoFnSignature$ProcessElementMethod.getRowParameter()Lorg/apache/beam/sdk/transforms/reflect/DoFnSignature$Parameter$RowParameter;
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.(SimpleDoFnRunner.java:175)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.DoFnRunners.simpleRunner(DoFnRunners.java:63)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.SimpleDoFnRunnerFactory.createRunner(SimpleDoFnRunnerFactory.java:52)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:288)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:312)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:409)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:378)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
> 04:18:48 at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
> 04:18:48 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 04:18:48 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 04:18:48 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 04:18:48 at java.lang.Thread.run(Thread.java:745)
> {code}
> This means we need a new worker. The postrelease tests verify that the
> Dataflow worker image works when we _don't_ stage the worker jar.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-6654) Support SDF within ElasticsearchIO
[ https://issues.apache.org/jira/browse/BEAM-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131431#comment-17131431 ] Beam JIRA Bot commented on BEAM-6654: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Support SDF within ElasticsearchIO > -- > > Key: BEAM-6654 > URL: https://issues.apache.org/jira/browse/BEAM-6654 > Project: Beam > Issue Type: New Feature > Components: io-java-elasticsearch >Reporter: Tony Moulton >Priority: P3 > > Similar to HbaseIO.ReadAll, it would be very useful to support a SDF based > transform which can compose queries based off of input elements. > Implementation would split the work similarly to how the current Source API > implementation works (leveraging slices within Elasticsearch). > This work can facilitate a transition of ElasticsearchIO to SDF when more > runners have implemented support. Until then, I propose leaving the read() > implementation alone and add a readAll method and associated transform to be > implemented as an SDF. Once enough runner support is ready, the existing > Read transform would be updated to use the same SDF as the ReadAll transform > with a null input element. > A prototype implementation currently exists (albeit developed in a bubble) at > [https://github.com/0xdecaf/beam/tree/feature/elasticsearchio.readall] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4786) Distinct has bad parallelism characteristics
[ https://issues.apache.org/jira/browse/BEAM-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131586#comment-17131586 ] Beam JIRA Bot commented on BEAM-4786: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Distinct has bad parallelism characteristics > > > Key: BEAM-4786 > URL: https://issues.apache.org/jira/browse/BEAM-4786 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Pablo Estrada >Priority: P2 > Labels: ccoss2019 > > Distinct groups first and then drops extra elements. Should drop elements in > the mappers, and later in reducers. > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java#L100 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-6382) SamzaRunner: add an option to read configs using a user-defined factory
[ https://issues.apache.org/jira/browse/BEAM-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-6382: --- Assignee: (was: Xinyu Liu) > SamzaRunner: add an option to read configs using a user-defined factory > --- > > Key: BEAM-6382 > URL: https://issues.apache.org/jira/browse/BEAM-6382 > Project: Beam > Issue Type: Improvement > Components: runner-samza >Reporter: Xinyu Liu >Priority: P2 > Labels: stale-assigned > Time Spent: 1.5h > Remaining Estimate: 0h > > We need an option to read configs from a factory which is useful in Yarn as > well as user-defined file format. By default this config factory is to read > property file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3393) Empty flattens cannot be used as side inputs.
[
https://issues.apache.org/jira/browse/BEAM-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131796#comment-17131796
]
Beam JIRA Bot commented on BEAM-3393:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Empty flattens cannot be used as side inputs.
> -
>
> Key: BEAM-3393
> URL: https://issues.apache.org/jira/browse/BEAM-3393
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Robert Bradshaw
>Priority: P2
> Labels: stale-P2
>
> {{with beam.Pipeline() as p:
> main = p | "CM" >> beam.Create([])
> side1 = p | "C1" >> beam.Create([])
> side2 = p | "C2" >> beam.Create([])
> side = (side1, side2) | beam.Flatten()
> res = main | beam.Map(lambda x, side: x, beam.pvalue.AsList(side))
> }}
> results in
> {{
> File
> "/Users/robertwb/Work/beam/incubator-beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
> line 84, in add_values
> assert not view.has_result
> }}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-4459) Schemas across pipeline modifications
[ https://issues.apache.org/jira/browse/BEAM-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-4459: --- Assignee: (was: Reuven Lax) > Schemas across pipeline modifications > - > > Key: BEAM-4459 > URL: https://issues.apache.org/jira/browse/BEAM-4459 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Reuven Lax >Priority: P2 > Labels: stale-assigned > > As per the snapshot/update proposal, we want to be able to update pipelines > without damaging the in-flight state. Since schema fields might get reordered > on update, we must ensure that the old mappings are preserved. This will > require us to have two ids - the logical ids the user interfaces with (which > might change), and the physical index where we store the schema. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8413) test_streaming_pipeline_returns_expected_user_metrics_fnapi_it failed on latest PostCommit Py36
[ https://issues.apache.org/jira/browse/BEAM-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8413: --- Assignee: (was: Ankur Goenka) > test_streaming_pipeline_returns_expected_user_metrics_fnapi_it failed on > latest PostCommit Py36 > - > > Key: BEAM-8413 > URL: https://issues.apache.org/jira/browse/BEAM-8413 > Project: Beam > Issue Type: New Feature > Components: test-failures >Reporter: Boyuan Zhang >Priority: P2 > Labels: stale-assigned > > https://builds.apache.org/job/beam_PostCommit_Python36/731/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8140) Python API: PTransform should be immutable and reusable
[
https://issues.apache.org/jira/browse/BEAM-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131254#comment-17131254
]
Beam JIRA Bot commented on BEAM-8140:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> Python API: PTransform should be immutable and reusable
> ---
>
> Key: BEAM-8140
> URL: https://issues.apache.org/jira/browse/BEAM-8140
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Chris Suchanek
>Priority: P2
>
> While the Java API seems fine the Python API is (at least) counterintuitive.
> Let's see the following example:
> {code:python}
> p1 = beam.Pipeline()
> p2 = beam.Pipeline()
> node = 'ReadTrainData' >> beam.io.ReadFromText("/tmp/aaa.txt")
> p1 | node
> p2 | node //fails here {code}
> The code above will fail because the _node_ somehow remembers that it was
> already attached to _p1_. In fact, unlike in Java, the | (apply) method is
> defined on the _PTransform_.
> If any, only the pipeline object should be mutable here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8154) IOTypeHints: merge None and ((), {}) into one value
[
https://issues.apache.org/jira/browse/BEAM-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131247#comment-17131247
]
Beam JIRA Bot commented on BEAM-8154:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> IOTypeHints: merge None and ((), {}) into one value
> ---
>
> Key: BEAM-8154
> URL: https://issues.apache.org/jira/browse/BEAM-8154
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: P2
>
> See https://github.com/apache/beam/pull/9283/files#r317829250 for discussion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9702) Update Java KinesisIO to support AWS SDK v2
[ https://issues.apache.org/jira/browse/BEAM-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-9702: --- Assignee: (was: Jonothan Farr) > Update Java KinesisIO to support AWS SDK v2 > --- > > Key: BEAM-9702 > URL: https://issues.apache.org/jira/browse/BEAM-9702 > Project: Beam > Issue Type: Improvement > Components: io-java-kinesis >Reporter: Jonothan Farr >Priority: P3 > Labels: stale-assigned > Time Spent: 1h 10m > Remaining Estimate: 0h > > As a contributor to the Java Kinesis IO, I would like a version of KinesisIO > that is based on the v2 AWS SDK so that support for enhanced fanout can be > added. I would also like this to be a separate work item from adding the > enhanced fanout support so that users of KinesisIO can potentially benefit > from updating to the v2 AWS SDK while enhanced fanout support is still being > worked on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9168) AppliedPTransform.from_runner_api fails on unexpected non-ParDo class with PAR_DO.urn
[ https://issues.apache.org/jira/browse/BEAM-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9168: Labels: (was: stale-assigned) > AppliedPTransform.from_runner_api fails on unexpected non-ParDo class with > PAR_DO.urn > - > > Key: BEAM-9168 > URL: https://issues.apache.org/jira/browse/BEAM-9168 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Priority: P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > This is failing on a google-internal test. > Unexpected class is apache_beam.transforms.core.RunnerAPIPTransformHolder. > Failed assertion: > https://github.com/apache/beam/blob/a59f897a64b0006ef3fcbe5a750d5f46499cfe61/sdks/python/apache_beam/pipeline.py#L1052-L1053 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8441) Python 3 pipeline fails with errors in StockUnpickler.find_class() during loading a main session.
[
https://issues.apache.org/jira/browse/BEAM-8441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-8441:
---
Assignee: (was: Valentyn Tymofieiev)
> Python 3 pipeline fails with errors in StockUnpickler.find_class() during
> loading a main session.
> --
>
> Key: BEAM-8441
> URL: https://issues.apache.org/jira/browse/BEAM-8441
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Jannik Franz
>Priority: P2
> Labels: stale-assigned
>
> When running Apache Beam with Python3 on Google Cloud Dataflow the pipeline
> fails during pickler.load_session(session_file):
> StockUnpickler.find_class(self, module, name) AttributeError: Can't get
> attribute 'SomeAttribute' on '/usr/local/lib/python3.6/site-packages/dataflow_worker/start.py'>
> Note that this is different from BEAM-8651, since the error happens in a
> Batch Pipeline on a Dataflow runner and the error happens consistently.
> When testing it in the local/direct runner there seems to be no issue.
>
> {code:java}
> class FlattenCustomActions(beam.PTransform):
> """ Transforms Facebook Day ActionsOnly retains actions with
> custom_conversions
> Flattens the actions
> Adds custom conversions names using a side input
> """
> def __init__(self, conversions):
> super(FlattenCustomActions, self).__init__()
> self.conversions = conversionsdef expand(self, input_or_inputs):
> return (
> input_or_inputs
> | "FlattenActions" >> beam.ParDo(flatten_filter_actions)
> | "AddConversionName" >> beam.Map(add_conversion_name,
> self.conversions)
> )
> # ...
> # in run():
> pipeline_options = PipelineOptions(pipeline_args)
> pipeline_options.view_as(SetupOptions).save_main_session = True
> p = beam.Pipeline(options=pipeline_options)
> conversions_output = (
> p
> | "ReadConversions" >> ReadFromText(known_args.input_conversions,
> coder=JsonCoder())
> | TransformConversionMetadata()
> )(
> conversions_output
> | "WriteConversions"
> >> WriteCoerced(
> known_args.output_conversions,
> known_args.output_type,
> schema_path=BIGQUERY_SCHEMA_CONVERSIONS_PATH,
> )
> )(
> p
> | ReadFacebookJson(known_args.input, retain_root_fields=True)
> | FlattenCustomActions(beam.pvalue.AsList(conversions_output))
> | "WriteActions"
> >> WriteCoerced(
> known_args.output, known_args.output_type,
> schema_path=BIGQUERY_SCHEMA_ACTIONS_PATH
> )
> ){code}
>
> I receive the following Traceback in Dataflow:
> {code:java}
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line
> 773, in run self._load_main_session(self.local_staging_directory)
> File
> "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line
> 489, in _load_main_session pickler.load_session(session_file)
> File
> "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py",
> line 287, in load_session return dill.load_session(file_path)
> File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 410, in
> load_session module = unpickler.load()
> File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 474, in
> find_class return StockUnpickler.find_class(self, module, name)
> AttributeError: Can't get attribute 'FlattenCustomActions' on 'dataflow_worker.start' from
> '/usr/local/lib/python3.6/site-packages/dataflow_worker/start.py'>
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9335) update hard-coded coder id when translating Java external transforms
[ https://issues.apache.org/jira/browse/BEAM-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9335: Labels: (was: stale-assigned) > update hard-coded coder id when translating Java external transforms > > > Key: BEAM-9335 > URL: https://issues.apache.org/jira/browse/BEAM-9335 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Priority: P2 > Time Spent: 1h > Remaining Estimate: 0h > > hard-coded coder id needs to be updated when translating Java external > transforms. Otherwise pipeline will fail if coder id is reused. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7934) Dataflow Python SDK logging: step_id is always empty string
[
https://issues.apache.org/jira/browse/BEAM-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7934:
Labels: (was: stale-assigned)
> Dataflow Python SDK logging: step_id is always empty string
> ---
>
> Key: BEAM-7934
> URL: https://issues.apache.org/jira/browse/BEAM-7934
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-py-core
>Affects Versions: 2.13.0
>Reporter: James Hutchison
>Priority: P2
>
> Using the dataflow runner, log messages always show up in stackdriver with
> the step_id as the empty string, so filtering log messages for a step doesn't
> work.
> {code:java}
> resource: {
> labels: {
> job_id: ""
> job_name: ""
> project_id: ""
> region: ""
> step_id: ""
> }
> type: "dataflow_step"
> }{code}
> Another user seems to have posted in the old github repo and appears to be
> seeing the same problem based on their output:
> [https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/62]
> From what I can tell is only affecting streaming pipelines
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-7611) Python BigTableIO IT is not running in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-7611: --- Assignee: (was: Solomon Duskis) > Python BigTableIO IT is not running in any test suites > -- > > Key: BEAM-7611 > URL: https://issues.apache.org/jira/browse/BEAM-7611 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, testing >Reporter: Chamikara Madhusanka Jayalath >Priority: P1 > Labels: stale-assigned > > We added an integration test here: [https://github.com/apache/beam/pull/7367] > > But this currently does not get picked up by any test suites (and get skipped > by some due to missing dependencies) hence BigTable sink is largely untested. > > First attempt to enable it: [https://github.com/apache/beam/pull/8886] > > Solomon assigning to you since I cannot find Juan's (PR author) Jira ID. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8487) Python typehints: support forward references
[
https://issues.apache.org/jira/browse/BEAM-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-8487:
---
Assignee: (was: Udi Meiri)
> Python typehints: support forward references
>
>
> Key: BEAM-8487
> URL: https://issues.apache.org/jira/browse/BEAM-8487
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: P2
> Labels: stale-assigned
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Typehints may be given as string literals:
> https://www.python.org/dev/peps/pep-0484/#forward-references
> These are currently not evaluated and result in errors.
> Example 1:
> {code}
> def test_typed_callable_string_hints(self):
> def do_fn(element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > return issubclass(sub, base)
> E TypeError: issubclass() arg 2 must be a class or tuple of classes
> typehints.py:1168: TypeError
> {code}
> Example 2:
> {code}
> def test_typed_dofn_string_hints(self):
> class MyDoFn(beam.DoFn):
> def process(self, element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(MyDoFn())
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > raise ValueError('%s is not iterable' % type_hint)
> E ValueError: typehints.List[str] is not iterable
> typehints.py:1194: ValueError
> {code}
> where the non-iterable entity the error refers to is a string literal
> ("typehints.List[str]").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8018) Detect unexported fields in unregistered types for better error messages
[ https://issues.apache.org/jira/browse/BEAM-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8018: --- Assignee: (was: Robert Burke) > Detect unexported fields in unregistered types for better error messages > > > Key: BEAM-8018 > URL: https://issues.apache.org/jira/browse/BEAM-8018 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Robert Burke >Priority: P3 > Labels: stale-assigned > Time Spent: 50m > Remaining Estimate: 0h > > Unregistered types with unexported fields cause pipeline execution time > panics rather than construction time panics/errors, which would be preferable. > Improved error messages encouraging users to register their types already > exists, but isn't currently triggered in this instance, so it's a matter of > returning an error in this case. > eg. for using time.Time as a value in a DoFn... > panic: > > reflect.StructOf: StructOf does not allow unexported fields [recovered] > panic: reflect.StructOf: StructOf does not allow unexported fields > goroutine 195 [running]: > panic(0x7e0a060, 0x84beb70) > third_party/go/gc/src/runtime/panic.go:567 +0x2da fp=0xc000dc1178 > sp=0xc000dc10c0 pc=0xee24daa > testing.tRunner.func1(0xc000d6ec00) > third_party/go/gc/src/testing/testing.go:830 +0x388 fp=0xc000dc11f8 > sp=0xc000dc1178 pc=0xfafda58 > runtime.call32(0x0, 0x833e1e0, 0xc000caaab0, 0x80008) > third_party/go/gc/src/runtime/asm_amd64.s:519 +0x3b fp=0xc000dc1228 > sp=0xc000dc11f8 pc=0xee53acb > panic(0x7e0a060, 0x84beb70) > third_party/go/gc/src/runtime/panic.go:522 +0x1b5 fp=0xc000dc12e0 > sp=0xc000dc1228 pc=0xee24c85 > reflect.runtimeStructField(0xc000e267e0, 0x4, 0xc000e267e4, 0x4, 0x8541880, > 0x7e0a060, 0x0, 0x0, 0x0, 0xc000e26990, ...) > third_party/go/gc/src/reflect/type.go:2765 +0x1c2 fp=0xc000dc1348 > sp=0xc000dc12e0 pc=0xee80ea2 > reflect.StructOf(0xc0006dd040, 0x3, 0x4, 0x0, 0x0) > third_party/go/gc/src/reflect/type.go:2371 +0x21f6 fp=0xc000dc1b28 > sp=0xc000dc1348 pc=0xee7f816 > [google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48800|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48800], > 0xc000dc1e68, 0xee3c4f2, 0xc000e26920, 0x0) > > third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/serialize.go:558 > +0xa6c fp=0xc000dc1df8 sp=0xc000dc1b28 pc=0xf53a7ac > [google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48780|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48780], > 0x0, 0x0, 0x0, 0x1) > > third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/serialize.go:533 > +0x471 fp=0xc000dc20c8 sp=0xc000dc1df8 pc=0xf53a1b1 > [google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48680|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48680], > 0xc000369bf0, 0xc000b24a40, 0x3e, 0xc000961728) > > third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/serialize.go:542 > +0x6c5 fp=0xc000dc2398 sp=0xc000dc20c8 pc=0xf53a405 > [google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48600|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48600], > 0xc0001af340, 0x1, 0x1, 0x2) > > third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/serialize.go:583 > +0x12fc fp=0xc000dc2668 sp=0xc000dc2398 pc=0xf53b03c > [google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48480|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48480], > 0xc0001af2d0, 0x1, 0x1, 0x2) > > third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/serialize.go:542 > +0x6c5 fp=0xc000dc2938 sp=0xc000dc2668 pc=0xf53a405 > [google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48280|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/graphx.decodeType(0xc000c48280], > 0x20, 0xc000e265c0, 0x0, 0x1) > > third_party/golang/apache_beam/pkg/beam/core/runtime/graphx/serialize.go:542 > +0x6c5 fp=0xc000dc2c08 sp=0xc000dc2938 pc=0xf53a405 >
[jira] [Assigned] (BEAM-8961) GBK Python Load test on Flink fails in Jenkins jobs
[ https://issues.apache.org/jira/browse/BEAM-8961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8961: --- Assignee: (was: Kamil Wasilewski) > GBK Python Load test on Flink fails in Jenkins jobs > --- > > Key: BEAM-8961 > URL: https://issues.apache.org/jira/browse/BEAM-8961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Priority: P2 > Labels: stale-assigned > > [https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/53/console] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9549) Flaky portableWordCountBatch and portableWordCountStreaming tests
[
https://issues.apache.org/jira/browse/BEAM-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131025#comment-17131025
]
Beam JIRA Bot commented on BEAM-9549:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> Flaky portableWordCountBatch and portableWordCountStreaming tests
> -
>
> Key: BEAM-9549
> URL: https://issues.apache.org/jira/browse/BEAM-9549
> Project: Beam
> Issue Type: Test
> Components: test-failures
>Reporter: Ning Kang
>Priority: P2
> Attachments: Sr5cNnx8sAW.png
>
>
> The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and
> :sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky,
> sometimes throws grpc errrors.
> Stacktrace
> !Sr5cNnx8sAW.png|width=2049,height=1001!
> In text:
> {code:java}
> INFO:root:Using Python SDK docker image:
> apache/beam_python2.7_sdk:2.21.0.dev. If the image is not available at local,
> we will try to pull from
> hub.docker.comINFO:apache_beam.runners.portability.fn_api_runner_transforms:
>
> INFO:apache_beam.utils.subprocess_server:Starting service
> with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v'
> '/var/run/docker.sock:/var/run/docker.sock' '--network=host'
> 'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost'
> '--job-port' '58753' '--artifact-port' '60175' '--expansion-port'
> '33067']INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> ArtifactStagingService started on
> localhost:60175INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
> ExpansionService started on
> localhost:33067INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> JobService started on localhost:58753ERROR:grpc._common:Exception
> deserializing message!Traceback (most recent call last): File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",
> line 84, in _transformreturn transformer(message)DecodeError: Error
> parsing messageTraceback (most recent call last): File
> "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py",
> line 72, in _run_codeexec code in run_globals File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
> line 142, in run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
> line 121, in runresult = p.run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 495, in runself._options).run(False) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 508, in runreturn self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 401, in run_pipelinejob_service_handle.submit(proto_pipeline) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 102, in submitprepare_response = self.prepare(proto_pipeline) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 179, in preparetimeout=self.timeout) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 826, in __call__return _end_unary_response_blocking(state, call,
> False, None) File
>
[jira] [Updated] (BEAM-9425) explicitly register default environment instead of assuming the first one as a default
[ https://issues.apache.org/jira/browse/BEAM-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9425: Labels: (was: stale-assigned) > explicitly register default environment instead of assuming the first one as > a default > -- > > Key: BEAM-9425 > URL: https://issues.apache.org/jira/browse/BEAM-9425 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Heejong Lee >Priority: P2 > > explicitly register default environment instead of assuming the first one as > a default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8587) Add TestStream support for Dataflow runner
[ https://issues.apache.org/jira/browse/BEAM-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8587: --- Assignee: (was: Andrew Crites) > Add TestStream support for Dataflow runner > -- > > Key: BEAM-8587 > URL: https://issues.apache.org/jira/browse/BEAM-8587 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, testing >Reporter: Andrew Crites >Priority: P2 > Labels: stale-assigned > Time Spent: 1h 50m > Remaining Estimate: 0h > > TestStream support needed to test features like late data and processing time > triggers on local Dataflow runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8212) StatefulParDoFn creates GC timers for every record
[ https://issues.apache.org/jira/browse/BEAM-8212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8212: --- Assignee: (was: Akshay Iyangar) > StatefulParDoFn creates GC timers for every record > --- > > Key: BEAM-8212 > URL: https://issues.apache.org/jira/browse/BEAM-8212 > Project: Beam > Issue Type: Bug > Components: runner-core >Reporter: Akshay Iyangar >Priority: P2 > Labels: stale-assigned > > Hi > So currently the StatefulParDoFn create timers for all the records. > [https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java#L211] > This becomes a problem if you are using GlobalWindows for streaming where > these timers get created and never get closed since the window will never > close. > This is a problem especially if your memory bound in rocksDB where these > timers take up potential space and sloe the pipelines considerably. > Was wondering that if the pipeline runs in global windows we should avoid > adding timers to it at all? > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8322) Log when artifacts can't be fetched on startup
[ https://issues.apache.org/jira/browse/BEAM-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8322: --- Assignee: (was: Kyle Weaver) > Log when artifacts can't be fetched on startup > -- > > Key: BEAM-8322 > URL: https://issues.apache.org/jira/browse/BEAM-8322 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: P3 > Labels: stale-assigned > > I noticed when I was testing my new artifact retrieval service: > While I'm sure artifact retrieval is the cause of error, there is nothing in > the logs at all to indicate that. > > EDIT: I realized the root cause might be because I wasn't returning an > artifact response, making the client wait forever. There should probably be > timeouts for that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7738) Support PubSubIO to be configured externally for use with other SDKs
[ https://issues.apache.org/jira/browse/BEAM-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-7738: --- Assignee: (was: Chad Dombrova) > Support PubSubIO to be configured externally for use with other SDKs > > > Key: BEAM-7738 > URL: https://issues.apache.org/jira/browse/BEAM-7738 > Project: Beam > Issue Type: New Feature > Components: cross-language, io-java-gcp, runner-flink, sdk-py-core >Reporter: Chad Dombrova >Priority: P2 > Labels: portability, stale-assigned > Time Spent: 9h 50m > Remaining Estimate: 0h > > Now that KafkaIO is supported via the external transform API (BEAM-7029) we > should add support for PubSub. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7992) Unhandled type_constraint in apache_beam.io.gcp.bigquery_write_it_test.BigQueryWriteIntegrationTests.test_big_query_write_new_types
[
https://issues.apache.org/jira/browse/BEAM-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-7992:
---
Assignee: (was: Udi Meiri)
> Unhandled type_constraint in
> apache_beam.io.gcp.bigquery_write_it_test.BigQueryWriteIntegrationTests.test_big_query_write_new_types
> ---
>
> Key: BEAM-7992
> URL: https://issues.apache.org/jira/browse/BEAM-7992
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: P2
> Labels: stale-assigned
>
> {code}
> root: DEBUG: Unhandled type_constraint: Union[]
> root: DEBUG: Unhandled type_constraint: Union[]
> root: DEBUG: Unhandled type_constraint: Any
> root: DEBUG: Unhandled type_constraint: Any
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37_PR/20/testReport/junit/apache_beam.io.gcp.bigquery_write_it_test/BigQueryWriteIntegrationTests/test_big_query_write_new_types/
> These log entries are from opcode.py's _unpack_lists.
> They might be pointing to a bug or missing feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9073) PipelineVisitor is topologically-order dependent
[ https://issues.apache.org/jira/browse/BEAM-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9073: Labels: (was: stale-assigned) > PipelineVisitor is topologically-order dependent > > > Key: BEAM-9073 > URL: https://issues.apache.org/jira/browse/BEAM-9073 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Sam Rohde >Priority: P2 > Time Spent: 1h 10m > Remaining Estimate: 0h > > The Python PipelineVisitor is topologically-order dependent and can visit the > same transform multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7964) Multiple triggering SQL pipeline
[ https://issues.apache.org/jira/browse/BEAM-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131277#comment-17131277 ] Beam JIRA Bot commented on BEAM-7964: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Multiple triggering SQL pipeline > > > Key: BEAM-7964 > URL: https://issues.apache.org/jira/browse/BEAM-7964 > Project: Beam > Issue Type: New Feature > Components: dsl-sql, sdk-java-core >Reporter: Rui Wang >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9117) Clustering coder does not get used for BQ multi-partitioned tables
[ https://issues.apache.org/jira/browse/BEAM-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-9117: --- Assignee: (was: Chamikara Madhusanka Jayalath) > Clustering coder does not get used for BQ multi-partitioned tables > -- > > Key: BEAM-9117 > URL: https://issues.apache.org/jira/browse/BEAM-9117 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: stale-assigned > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8843) Limit bigquery.py exponential backoff to 60s
[ https://issues.apache.org/jira/browse/BEAM-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131153#comment-17131153 ] Beam JIRA Bot commented on BEAM-8843: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Limit bigquery.py exponential backoff to 60s > > > Key: BEAM-8843 > URL: https://issues.apache.org/jira/browse/BEAM-8843 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Priority: P2 > > Seen here: > https://issues.apache.org/jira/browse/BEAM-8842?focusedCommentId=16983995=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16983995 > retry delays grow quickly into the 1000s range. > cc [~chamikara] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9071) Ignore pytype untill the typing feature is fully developed
[ https://issues.apache.org/jira/browse/BEAM-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-9071: --- Assignee: (was: Udi Meiri) > Ignore pytype untill the typing feature is fully developed > -- > > Key: BEAM-9071 > URL: https://issues.apache.org/jira/browse/BEAM-9071 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Boyuan Zhang >Priority: P2 > Labels: stale-assigned > Time Spent: 1.5h > Remaining Estimate: 0h > > Please remove all # pytype: skip-file after feature fully supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9131) cleanup bug in run_rc_validation.sh
[
https://issues.apache.org/jira/browse/BEAM-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-9131:
Labels: (was: stale-assigned)
> cleanup bug in run_rc_validation.sh
> ---
>
> Key: BEAM-9131
> URL: https://issues.apache.org/jira/browse/BEAM-9131
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Udi Meiri
>Priority: P2
>
> Saw this in the console:
> {code}
> -Cleaning up BigQuery & Pubsub-
> /usr/lib/google-cloud-sdk/platform/bq/bq.py:45: DeprecationWarning: the imp
> module is deprecated in favour of importlib; see the module's documentation
> for alternative uses
> import imp
> FATAL Flags parsing error: Unknown command line flag 'rf'
> Run 'bq.py help' to get help
> Deleted topic [projects/XXX/topics/leader_board-ehudm-java-topic-0115_19681].
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8580) Request Python API to support windows ClosingBehavior
[ https://issues.apache.org/jira/browse/BEAM-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8580: --- Assignee: (was: Yichi Zhang) > Request Python API to support windows ClosingBehavior > - > > Key: BEAM-8580 > URL: https://issues.apache.org/jira/browse/BEAM-8580 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: wendy liu >Priority: P2 > Labels: stale-assigned > > Beam Python should have an API to support windows ClosingBehavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8023: Labels: (was: stale-assigned) > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Priority: P3 > Time Spent: 2h 50m > Remaining Estimate: 0h > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7471) DirectRunner GameStats fails, ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462"
[
https://issues.apache.org/jira/browse/BEAM-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7471:
Labels: (was: stale-assigned)
> DirectRunner GameStats fails, ValueError: PCollection of size 2 with more
> than one element accessed as a singleton view. First two elements encountered
> are "181.611538462", "9.1". [while running
> 'CalculateSpammyUsers/ProcessAndFilter']
> ---
>
> Key: BEAM-7471
> URL: https://issues.apache.org/jira/browse/BEAM-7471
> Project: Beam
> Issue Type: Improvement
> Components: runner-gearpump
>Affects Versions: 2.13.0
>Reporter: Ankur Goenka
>Priority: P2
>
>
> GameStats example is failing with following logs
> {noformat}
> *
> * Running GameStats with DirectRunner
> *
> INFO:root:Missing pipeline option (runner). Executing pipeline using the
> default runner: DirectRunner.
> /usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1137:
> BeamDeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported
> method_to_use = self._compute_method(p, p.options)
> INFO:root:Running pipeline with DirectRunner.
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Created table
> google.com:clouddfe.goenka_python_validations.game_stats_teams with schema
> fields: [ fields: []
> mode: u'NULLABLE'
> name: u'processing_time'
> type: u'STRING'>, fields: []
> mode: u'NULLABLE'
> name: u'total_score'
> type: u'INTEGER'>, fields: []
> mode: u'NULLABLE'
> name: u'window_start'
> type: u'STRING'>, fields: []
> mode: u'NULLABLE'
> name: u'team'
> type: u'STRING'>]>. Result: creationTime: 1559341252301
> etag: u'tEQeN0Sr4FvttBRZuXn7zw=='
> id: u'google.com:clouddfe:goenka_python_validations.game_stats_teams'
> kind: u'bigquery#table'
> lastModifiedTime: 1559341252336
> location: u'US'
> numBytes: 0
> numLongTermBytes: 0
> numRows: 0
> schema: fields: [ fields: []
> mode: u'NULLABLE'
> name: u'processing_time'
> type: u'STRING'>, fields: []
> mode: u'NULLABLE'
> name: u'total_score'
> type: u'INTEGER'>, fields: []
> mode: u'NULLABLE'
> name: u'window_start'
> type: u'STRING'>, fields: []
> mode: u'NULLABLE'
> name: u'team'
> type: u'STRING'>]>
> selfLink:
> u'https://www.googleapis.com/bigquery/v2/projects/google.com:clouddfe/datasets/goenka_python_validations/tables/game_stats_teams'
> tableReference: datasetId: u'goenka_python_validations'
> projectId: u'google.com:clouddfe'
> tableId: u'game_stats_teams'>
> type: u'TABLE'>.
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> ERROR:root:Exception at bundle
> ,
> due to an exception.
> Traceback (most recent call last):
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
> line 343, in call
> finish_state)
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
> line 380, in attempt_call
> evaluator.process_element(value)
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
> line 633, in process_element
> self.runner.process(element)
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 753, in process
> self._reraise_augmented(exn)
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 807, in _reraise_augmented
> raise_with_traceback(new_exn)
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 751, in process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File
> "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
> line 564, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
> File
>
[jira] [Updated] (BEAM-9624) Combine operation should support only converting to accumulators
[ https://issues.apache.org/jira/browse/BEAM-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9624: Labels: (was: stale-assigned) > Combine operation should support only converting to accumulators > > > Key: BEAM-9624 > URL: https://issues.apache.org/jira/browse/BEAM-9624 > Project: Beam > Issue Type: Improvement > Components: runner-core >Reporter: Andrew Crites >Priority: P2 > Time Spent: 2h 40m > Remaining Estimate: 0h > > For streaming pipelines, we want to be able to lift the combiner into the > MergeBuckets without having to also do a PartialGroupByKey before the > shuffle. We don't want to do the PGBK since it could cause non-deterministic > results when used with some triggers. > We propose adding a new URN for doing just the convert to accumulators step > and adding support for it in Java/Python/Go. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9134) RedisIO.ReadAll incorrectly alters windowing behavior
[ https://issues.apache.org/jira/browse/BEAM-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9134: Labels: (was: stale-assigned) > RedisIO.ReadAll incorrectly alters windowing behavior > - > > Key: BEAM-9134 > URL: https://issues.apache.org/jira/browse/BEAM-9134 > Project: Beam > Issue Type: Bug > Components: io-java-redis >Reporter: Ismaël Mejía >Priority: P3 > > When using RedisIO.ReadAll as an intermediate step of a pipeline that reads > data from an unboundedsource it does not produces results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8541) Beam directRunner documentation java tab has python information
[ https://issues.apache.org/jira/browse/BEAM-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8541: --- Assignee: (was: Brent Worden) > Beam directRunner documentation java tab has python information > --- > > Key: BEAM-8541 > URL: https://issues.apache.org/jira/browse/BEAM-8541 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Collins >Priority: P2 > Labels: stale-assigned > > [https://beam.apache.org/documentation/runners/direct/] > > Clicking on the "Java SDK" tab still has python examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7925) ParquetIO supports neither column projection nor filter predicate
[
https://issues.apache.org/jira/browse/BEAM-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-7925:
---
Assignee: (was: Heejong Lee)
> ParquetIO supports neither column projection nor filter predicate
> -
>
> Key: BEAM-7925
> URL: https://issues.apache.org/jira/browse/BEAM-7925
> Project: Beam
> Issue Type: Improvement
> Components: io-java-parquet
>Affects Versions: 2.14.0
>Reporter: Neville Li
>Priority: P2
> Labels: stale-assigned
>
> Current {{ParquetIO}} supports neither column projection nor filter predicate
> which defeats the performance motivation of using Parquet in the first place.
> That's why we have our own implementation of
> [ParquetIO|https://github.com/spotify/scio/tree/master/scio-parquet/src] in
> Scio.
> Reading Parquet as Avro with column projection has some complications,
> namely, the resulting Avro records may be incomplete and will not survive
> ser/de. A workaround maybe provide a {{TypedRead}} interface that takes a
> {{Function}} that maps invalid Avro {{A}} into user defined type {{B}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-7968) UDF case insensitive
[ https://issues.apache.org/jira/browse/BEAM-7968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-7968: --- Assignee: (was: Andrew Pilloud) > UDF case insensitive > > > Key: BEAM-7968 > URL: https://issues.apache.org/jira/browse/BEAM-7968 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Yang Zhang >Priority: P2 > Labels: stale-assigned > > Currently, in Beam SQL, UDF is case sensitive. Is there a plan to make UDF > case insensitive? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8307) NPE in Calcite dialect when input PCollection has logical type in schema, from JdbcIO Transform
[ https://issues.apache.org/jira/browse/BEAM-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8307: --- Assignee: (was: Rahul Patwari) > NPE in Calcite dialect when input PCollection has logical type in schema, > from JdbcIO Transform > --- > > Key: BEAM-8307 > URL: https://issues.apache.org/jira/browse/BEAM-8307 > Project: Beam > Issue Type: Bug > Components: dsl-sql, io-java-jdbc >Affects Versions: 2.15.0 >Reporter: Rahul Patwari >Priority: P2 > Labels: stale-assigned > Attachments: exception_sqlTransform.txt > > Time Spent: 2h 20m > Remaining Estimate: 0h > > An exception is thrown when SqlTransform is used with JdbcIO. > Schema of the Source Table: > _CREATE TABLE actor (_ > _actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,_ > _first_name VARCHAR(45) NOT NULL,_ > _last_name VARCHAR(45) NOT NULL,_ > _last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE > CURRENT_TIMESTAMP,_ > _PRIMARY KEY (actor_id),_ > _KEY idx_actor_last_name (last_name)_ > _)ENGINE=InnoDB DEFAULT CHARSET=utf8;_ > > Many of the Jdbc Types are converted to Logical_Types in Beam. > Refer: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java#L102] > The initial observation is that this code: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L207] > is returning null. > > PFA [^exception_sqlTransform.txt] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9750) Streaming Word Count Example Documents is out of date (Python)
[ https://issues.apache.org/jira/browse/BEAM-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9750: Labels: (was: stale-assigned) > Streaming Word Count Example Documents is out of date (Python) > -- > > Key: BEAM-9750 > URL: https://issues.apache.org/jira/browse/BEAM-9750 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, website >Reporter: Ahmet Altay >Priority: P2 > > Flink runners are listed as "This runner is not yet available for the Python > SDK." This is not accurate., Flink runner supports streaming with python. > Link: > https://beam.apache.org/get-started/wordcount-example/#streamingwordcount-example > /cc [~ibzib] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8125) SchemaCoder/RowCoder.verifyDeterministic throws NPE when schema has a logical type
[ https://issues.apache.org/jira/browse/BEAM-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8125: Labels: (was: stale-assigned) > SchemaCoder/RowCoder.verifyDeterministic throws NPE when schema has a logical > type > -- > > Key: BEAM-8125 > URL: https://issues.apache.org/jira/browse/BEAM-8125 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Affects Versions: 2.15.0 >Reporter: Brian Hulette >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7983) Template parameters don't work if they are only used in DoFns
[
https://issues.apache.org/jira/browse/BEAM-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-7983:
---
Assignee: (was: Luke Cwik)
> Template parameters don't work if they are only used in DoFns
> -
>
> Key: BEAM-7983
> URL: https://issues.apache.org/jira/browse/BEAM-7983
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Yunqing Zhou
>Priority: P3
> Labels: stale-assigned
>
> Template parameters don't work if they are only used in DoFns but not
> anywhere else in main.
> Sample pipeline:
>
> {code:java}
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.options.ValueProvider;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.ParDo;
> public class BugPipeline {
> public interface Options extends PipelineOptions {
> ValueProvider getFoo();
> void setFoo(ValueProvider foo);
> }
> public static void main(String[] args) throws Exception {
> Options options = PipelineOptionsFactory.fromArgs(args).as(Options.class);
> Pipeline p = Pipeline.create(options);
> p.apply(Create.of(1)).apply(ParDo.of(new DoFn() {
> @ProcessElement
> public void processElement(ProcessContext context) {
>
> System.out.println(context.getPipelineOptions().as(Options.class).getFoo());
> }
> }));
> p.run();
>
>
>
>
> }
> }
> {code}
> Option "foo" is not used anywhere else than the DoFn. So to reproduce the
> problem:
> {code:bash}
> $java BugPipeline --project=$PROJECT --stagingLocation=$STAGING
> --templateLocation=$TEMPLATE --runner=DataflowRunner
> $gcloud dataflow jobs run $NAME --gcs-location=$TEMPLATE --parameters=foo=bar
> {code}
> it will fail w/ this error:
> {code}
> ERROR: (gcloud.dataflow.jobs.run) INVALID_ARGUMENT: (2621bec26c2488b7): The
> workflow could not be created. Causes: (2621bec26c248dba): Found unexpected
> parameters: ['foo' (perhaps you meant 'zone')]
> - '@type': type.googleapis.com/google.rpc.DebugInfo
> detail: "(2621bec26c2488b7): The workflow could not be created. Causes:
> (2621bec26c248dba):\
> \ Found unexpected parameters: ['foo' (perhaps you meant 'zone')]"
> {code}
> The underlying problem is that ProxyInvocationHandler.java only populate
> options which are "invoked" to the pipeline option map in the job object:
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L159
> One way to solve it is to save all ValueProvider type of params in the
> pipelineoptions section. Alternatively, some registration mechanism can be
> introduced.
> A current workaround is to annotate the parameter with
> {code}@Validation.Required{code}, which will call invoke() behind the scene.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8823) Make FnApiRunner work by executing ready elements instead of stages
[ https://issues.apache.org/jira/browse/BEAM-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-8823: --- Assignee: (was: Pablo Estrada) > Make FnApiRunner work by executing ready elements instead of stages > --- > > Key: BEAM-8823 > URL: https://issues.apache.org/jira/browse/BEAM-8823 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Priority: P2 > Labels: stale-assigned > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8681) Beam Dependency Update Request: com.google.api.grpc:proto-google-cloud-pubsub-v1
[ https://issues.apache.org/jira/browse/BEAM-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131164#comment-17131164 ] Beam JIRA Bot commented on BEAM-8681: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Beam Dependency Update Request: > com.google.api.grpc:proto-google-cloud-pubsub-v1 > > > Key: BEAM-8681 > URL: https://issues.apache.org/jira/browse/BEAM-8681 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Time Spent: 2h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:38:51.277928 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.83.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:09.290732 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.83.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:11.792298 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.83.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:21.703111 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:22.824001 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:03:51.651991 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:27.071879 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7984) [python] The coder returned for typehints.List should be IterableCoder
[ https://issues.apache.org/jira/browse/BEAM-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7984: Labels: (was: stale-assigned) > [python] The coder returned for typehints.List should be IterableCoder > -- > > Key: BEAM-7984 > URL: https://issues.apache.org/jira/browse/BEAM-7984 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Priority: P2 > Time Spent: 4h 10m > Remaining Estimate: 0h > > IterableCoder encodes a list and decodes to list, but > typecoders.registry.get_coder(typehints.List[bytes]) returns a > FastPrimitiveCoder. I don't see any reason why this would be advantageous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9780) Add a DICOM IO Connector for Google Cloud Healthcare API
[ https://issues.apache.org/jira/browse/BEAM-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-9780: --- Assignee: (was: David Anderson) > Add a DICOM IO Connector for Google Cloud Healthcare API > > > Key: BEAM-9780 > URL: https://issues.apache.org/jira/browse/BEAM-9780 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: David Anderson >Priority: P3 > Labels: stale-assigned > > Add IO Transforms for the DICOM store in the [Google Cloud Healthcare > API|https://cloud.google.com/healthcare/docs/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8572) tox environment: assert on Cython source file presence
[ https://issues.apache.org/jira/browse/BEAM-8572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8572: Labels: (was: stale-assigned) > tox environment: assert on Cython source file presence > -- > > Key: BEAM-8572 > URL: https://issues.apache.org/jira/browse/BEAM-8572 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Udi Meiri >Priority: P2 > > Add an assertion somewhere that checks if Cythonized files are present in the > sdist tarball in use. That is for "tox -e py27" assert that these files are > not present, for "tox -e py27-cython" assert that they are present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8571) Use SDK version in default Go container tag
[ https://issues.apache.org/jira/browse/BEAM-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8571: Labels: (was: stale-assigned) > Use SDK version in default Go container tag > --- > > Key: BEAM-8571 > URL: https://issues.apache.org/jira/browse/BEAM-8571 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Kyle Weaver >Priority: P2 > > Currently, the Go SDK uses container `apachebeam/go_sdk:latest` by default > [1]. This causes confusion when using locally built containers [2], > especially since images are automatically pulled, meaning the release image > is used instead of the developer's own image (BEAM-8545). > [1] > [https://github.com/apache/beam/blob/473377ef8f51949983508f70663e75ef0ee24a7f/sdks/go/pkg/beam/options/jobopts/options.go#L111] > [[2] > https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E|https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8613) Add environment variable support to Docker environment
[ https://issues.apache.org/jira/browse/BEAM-8613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131171#comment-17131171 ] Beam JIRA Bot commented on BEAM-8613: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Add environment variable support to Docker environment > -- > > Key: BEAM-8613 > URL: https://issues.apache.org/jira/browse/BEAM-8613 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-core, runner-direct >Reporter: Nathan Rusch >Priority: P4 > Time Spent: 1.5h > Remaining Estimate: 0h > > The Process environment allows specifying environment variables via a map > field on its payload message. The Docker environment should support this same > pattern, and forward the contents of the map through to the container runtime. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8510) SamzaJobServerDriver to extend JobServerDriver
[ https://issues.apache.org/jira/browse/BEAM-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131186#comment-17131186 ] Beam JIRA Bot commented on BEAM-8510: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > SamzaJobServerDriver to extend JobServerDriver > -- > > Key: BEAM-8510 > URL: https://issues.apache.org/jira/browse/BEAM-8510 > Project: Beam > Issue Type: Improvement > Components: runner-samza >Reporter: Daniel Chen >Priority: P2 > > Should extend JobServerDriver to take advantage of the default service > initialization -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131203#comment-17131203 ] Beam JIRA Bot commented on BEAM-8388: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Priority: P2 > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8809) AvroWriteRequest should have a public constructor
[ https://issues.apache.org/jira/browse/BEAM-8809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131156#comment-17131156 ] Beam JIRA Bot commented on BEAM-8809: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > AvroWriteRequest should have a public constructor > - > > Key: BEAM-8809 > URL: https://issues.apache.org/jira/browse/BEAM-8809 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Steve Niemitz >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > AvroWriteRequest currently has an internal constructor, which prevents users > from unit testing their avro format functions for the BQ writer. This > constructor should be public. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8276) Document Flink pipeline jars on website
[ https://issues.apache.org/jira/browse/BEAM-8276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131224#comment-17131224 ] Beam JIRA Bot commented on BEAM-8276: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Document Flink pipeline jars on website > --- > > Key: BEAM-8276 > URL: https://issues.apache.org/jira/browse/BEAM-8276 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Kyle Weaver >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8277) Make docker build quicker
[ https://issues.apache.org/jira/browse/BEAM-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8277: Labels: (was: stale-assigned) > Make docker build quicker > - > > Key: BEAM-8277 > URL: https://issues.apache.org/jira/browse/BEAM-8277 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Kyle Weaver >Priority: P2 > > Building the Python SDK harness container takes minutes on my machine. > ``` > ./gradlew :sdks:python:container:buildAll > BUILD SUCCESSFUL in 9m 33s > ``` > Possible lead: "We spend mins pulling cmd/beamctl deps." > [https://github.com/apache/beam/blob/47feeafb21023e2a60ae51737cc4000a2033719c/sdks/python/container/build.gradle#L38] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8912) PreCommit_Python2_PVR_Flink_Commit flaky
[
https://issues.apache.org/jira/browse/BEAM-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131142#comment-17131142
]
Beam JIRA Bot commented on BEAM-8912:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> PreCommit_Python2_PVR_Flink_Commit flaky
>
>
> Key: BEAM-8912
> URL: https://issues.apache.org/jira/browse/BEAM-8912
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink
>Reporter: Ahmet Altay
>Priority: P1
>
> cc: [~angoenka]
> Logs:
> [https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Commit/1687/console]
> Error:
> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:external:v1:0:beam:metric:sampled_byte_size:v1
> \{PCOLLECTION=ref_PCollection_PCollection_27}: DistributionResult\{sum=59,
> count=1, min=59, max=59}))*19:44:02* [flink-runner-job-invoker] WARN
> org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService - Failed
> to remove job staging directory for token
> \{"sessionId":"job_b05dbc6c-00d8-4df1-9bf1-efdf35899fa6","basePath":"/tmp/flinktestdPbkyj"}:
> {}*19:44:02* java.io.FileNotFoundException:
> /tmp/flinktestdPbkyj/job_b05dbc6c-00d8-4df1-9bf1-efdf35899fa6/MANIFEST (No
> such file or directory)*19:44:02* at
> java.io.FileInputStream.open0(Native Method)*19:44:02* at
> java.io.FileInputStream.open(FileInputStream.java:195)*19:44:02* at
> java.io.FileInputStream.(FileInputStream.java:138)*19:44:02* at
> org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)*19:44:02*
> at
> org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)*19:44:02*
>at
> org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)*19:44:02*
> at
> org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)*19:44:02*
>at
> org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)*19:44:02*
> at
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)*19:44:02*
>at
> org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)*19:44:02*
> at
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)*19:44:02*
>at
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)*19:44:02*
> at
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)*19:44:02*
> at
> org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)*19:44:02*
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)*19:44:02*
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*19:44:02*
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*19:44:02*
> at java.lang.Thread.run(Thread.java:748)*19:44:02*
> INFO:apache_beam.runners.portability.portable_runner:Job state changed to
> DONE*19:44:02* .INFO:__main__:removing conf dir:
> /tmp/flinktest-confGy17Mj*19:44:02*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7391) Colab examples for other transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131351#comment-17131351 ] Beam JIRA Bot commented on BEAM-7391: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Colab examples for other transforms (Python) > > > Key: BEAM-7391 > URL: https://issues.apache.org/jira/browse/BEAM-7391 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Priority: P3 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7888) Test Multi Process Direct Runner With Largish Data
[ https://issues.apache.org/jira/browse/BEAM-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-7888: --- Assignee: (was: Hannah Jiang) > Test Multi Process Direct Runner With Largish Data > -- > > Key: BEAM-7888 > URL: https://issues.apache.org/jira/browse/BEAM-7888 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: P2 > Labels: stale-assigned > > Filing this as a tracker. > We can test multiprocess runner with a largish amount of data to the extend > that we can do this on Jenkins. This will serve 2 purposes: > - Find out issues related to multi processing. It would be easier to find > rare issues when running over non-trivial data. > - Serve as a baseline (if not a benchmark) to understand the limits of the > multiprocess runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8364) SchemaCoder inconsistent equality behavior for POJO
[
https://issues.apache.org/jira/browse/BEAM-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131207#comment-17131207
]
Beam JIRA Bot commented on BEAM-8364:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> SchemaCoder inconsistent equality behavior for POJO
> ---
>
> Key: BEAM-8364
> URL: https://issues.apache.org/jira/browse/BEAM-8364
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, sdk-java-core
>Affects Versions: 2.16.0
>Reporter: Neville Li
>Priority: P3
>
> One can create a {{SchemaCoder}} for arbitrary type {{T}} with
> {{SchemaCoder.of(schema, toRowFunction, fromRowFunction)}}. However, in cases
> where {{T}} lacks proper equality behavior, i.e. POJO, the result coder still
> returns true for {{consistentWithEquals}} and {{structuralValue}}s that fail
> equality check.
> This test reproduces the issue.
> {code:java}
> import org.apache.beam.sdk.schemas.Schema;
> import org.apache.beam.sdk.schemas.SchemaCoder;
> import org.apache.beam.sdk.values.Row;
> import org.junit.Test;
> import org.junit.runner.RunWith;
> import org.junit.runners.JUnit4;
> import java.nio.charset.Charset;
> import static org.junit.Assert.*;
> @RunWith(JUnit4.class)
> public class SchemaCoderTest {
> public static class Pojo {
> private final byte[] bytes;
> private final String id;
> public Pojo(byte[] bytes, String id) {
> this.bytes = bytes;
> this.id = id;
> }
> public byte[] getBytes() {
> return bytes;
> }
> public String getId() {
> return id;
> }
> }
> @Test
> public void testCoder() {
> Schema schema =
> Schema.builder().addByteArrayField("bytes").addStringField("id").build();
> SchemaCoder coder = SchemaCoder.of(
> schema,
> t -> Row.withSchema(schema).addValues(t.getBytes(),
> t.getId()).build(),
> r -> new Pojo(r.getBytes("bytes"), r.getString("id")));
> Pojo p1 = new Pojo("hello".getBytes(Charset.forName("UTF-8")), "world");
> Pojo p2 = new Pojo("hello".getBytes(Charset.forName("UTF-8")), "world");
> assertNotEquals(p1, p2); // EXPECTED, p1.equals(p2) == false
> assertFalse(coder.consistentWithEquals()); // FAIL, returns true
> assertEquals(coder.structuralValue(p1), coder.structuralValue(p2)); //
> FAIL
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7765) Add test for snippet accessing_valueprovider_info_after_run
[ https://issues.apache.org/jira/browse/BEAM-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7765: Labels: easy (was: easy stale-assigned) > Add test for snippet accessing_valueprovider_info_after_run > --- > > Key: BEAM-7765 > URL: https://issues.apache.org/jira/browse/BEAM-7765 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Priority: P2 > Labels: easy > Time Spent: 3h > Remaining Estimate: 0h > > This snippet needs a unit test. > It has bugs. For example: > - apache_beam.utils.value_provider doesn't exist > - beam.combiners.Sum doesn't exist > - unused import of: WriteToText > cc: [~pabloem] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8424) Java Dataflow ValidatesRunner tests are timeouting
[ https://issues.apache.org/jira/browse/BEAM-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131200#comment-17131200 ] Beam JIRA Bot commented on BEAM-8424: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Java Dataflow ValidatesRunner tests are timeouting > -- > > Key: BEAM-8424 > URL: https://issues.apache.org/jira/browse/BEAM-8424 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Lukasz Gajowy >Priority: P2 > Fix For: Not applicable > > Time Spent: 2h > Remaining Estimate: 0h > > [https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/] > [https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow/] > these jobs take more than currently set timeout (3h). > > EDIT: currently, after reopening the issue the timeout is set to 4.5h. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8924) Beam Dependency Update Request: org.apache.tika
[ https://issues.apache.org/jira/browse/BEAM-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131137#comment-17131137 ] Beam JIRA Bot commented on BEAM-8924: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Beam Dependency Update Request: org.apache.tika > --- > > Key: BEAM-8924 > URL: https://issues.apache.org/jira/browse/BEAM-8924 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Time Spent: 20m > Remaining Estimate: 0h > > - 2019-12-09 12:20:19.972138 > - > Please consider upgrading the dependency org.apache.tika. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2230) Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined outside of the general ApiSurface class
[ https://issues.apache.org/jira/browse/BEAM-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131951#comment-17131951 ] Beam JIRA Bot commented on BEAM-2230: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined > outside of the general ApiSurface class > > > Key: BEAM-2230 > URL: https://issues.apache.org/jira/browse/BEAM-2230 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currenlty, ApiSurface.getSdkApiSurface() is highly specialized and also not > correct. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2428) Enforce all inputs and outputs are properly declared in Transform Hierarchy
[
https://issues.apache.org/jira/browse/BEAM-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-2428:
Labels: stale-P2 (was: )
> Enforce all inputs and outputs are properly declared in Transform Hierarchy
> ---
>
> Key: BEAM-2428
> URL: https://issues.apache.org/jira/browse/BEAM-2428
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Thomas Groh
>Priority: P2
> Labels: stale-P2
>
> Within a composite transform, each child node may only consume values
> declared as the input to the composite and values produced as output by other
> direct children of the composite node. This can be enforced automatically via
> pipeline traversal, or by the {{TransformHierarchy}} directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-2341) Make our Python pom files virtualenv-compatible.
[ https://issues.apache.org/jira/browse/BEAM-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2341: Labels: stale-P2 (was: ) > Make our Python pom files virtualenv-compatible. > > > Key: BEAM-2341 > URL: https://issues.apache.org/jira/browse/BEAM-2341 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Robert Bradshaw >Priority: P2 > Labels: stale-P2 > > The importance of this may depend on whether we recommend or discourage using > mvn at all for Python, see also > https://issues.apache.org/jira/browse/BEAM-2340 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2470) Inconsistent behavior on the functioning of the dataflow templates?
[
https://issues.apache.org/jira/browse/BEAM-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-2470:
Labels: stale-P2 (was: )
> Inconsistent behavior on the functioning of the dataflow templates?
> ---
>
> Key: BEAM-2470
> URL: https://issues.apache.org/jira/browse/BEAM-2470
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: Damien GOUYETTE
>Priority: P2
> Labels: stale-P2
>
> 0
> down vote
> favorite
> When i create a dataflow template, the characteristics of Runtime parameters
> are not persisted in the template file. At runtime, if i try to pass a value
> for this parameter, i take a 400 error
> I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
> My parameters are the following :
> {code:java}
> trait MyParameters extends PipelineOptions {
> def getInput: ValueProvider[String]
> def setInput(value: ValueProvider[String]): Unit
> }
> {code}
> They are registred with this code
> {code:java}
> val options = PipelineOptionsFactory.fromArgs(cmdlineArgs:
> _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
> PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
> implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
> {code}
> To create the template i call sbt with this parameters :
> {code:java}
> run-main jobs.XmlImportJob--runner=DataflowRunner --project=MyProject
> --templateLocation=gs://myBucket/XmlImportTemplate
> --tempLocation=gs://myBucket/staging --instance=myInstance
> {code}
> If i pass explicitly --input, it becomes a StaticValue instead of
> RuntimeValue, and this time, i can see it in the template file.
> The template is called from a google function watching a bucket storage
> (inspired from
> https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/)
> :
> {code:java}
> ...
> dataflow.projects.templates.create({
> projectId: projectId,
> resource: {
> parameters: {
> input: `gs://${file.bucket}/${file.name}`
> },
> jobName: jobs[job].name,
> gcsPath: 'gs://MyBucket/MyTemplate'
> }
> }
> ...
> {code}
> The 400 error :
> {code:java}
> problem running dataflow template, error was: { Error: (109c1c52dc52fec7):
> The workflow could not be created. Causes: (109c1c52dc52fb8e): Found
> unexpected parameters: ['input' (perhaps you meant 'runner')] at
> Request._callback
> (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
> at Request.self.callback
> (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22)
> at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at
> Request. at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at
> IncomingMessage.
> (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12)
> at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code:
> 400, errors: [ { message: '(109c1c52dc52fec7): The workflow could not be
> created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: [\'input\'
> (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ] }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-2458) Move HashingFn from test -> main to make it more accessible
[ https://issues.apache.org/jira/browse/BEAM-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131920#comment-17131920 ] Beam JIRA Bot commented on BEAM-2458: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Move HashingFn from test -> main to make it more accessible > --- > > Key: BEAM-2458 > URL: https://issues.apache.org/jira/browse/BEAM-2458 > Project: Beam > Issue Type: Bug > Components: io-java-utilities >Reporter: Stephen Sisk >Priority: P2 > Labels: stale-P2 > > HashingFn is currently only available as a test dependency. There's no reason > for that (it could be generally useful to non-test parts of IOs if > necessary), so we should move it over to regular directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3783) Streaming Beam SQL benchmarks on all of our runners
[ https://issues.apache.org/jira/browse/BEAM-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132276#comment-17132276 ] Beam JIRA Bot commented on BEAM-3783: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Streaming Beam SQL benchmarks on all of our runners > --- > > Key: BEAM-3783 > URL: https://issues.apache.org/jira/browse/BEAM-3783 > Project: Beam > Issue Type: New Feature > Components: testing-nexmark >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: P2 > Labels: SQL, bigdata, cloud, gsoc2018, java, stale-assigned > > Beam has a number of classic streaming SQL benchmarks known as "Nexmark" > coded up in both raw Java and also Beam SQL. > So far, expanding functionality has been the focus of Beam SQL so there is > little known about performance - we know only that it is a pretty > straightforward mapping from SQL to Beam that should work OK a lot of the > time. It would be interesting to see where the bottlenecks are when these SQL > benchmarks are translated via Beam SQL into a Beam pipeline and then again > translated to the native capabilities of e.g. Spark and Flink. > This project will require the ability to read, write, and run Java and SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-349) Spark runner should provide a default BoM
[
https://issues.apache.org/jira/browse/BEAM-349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-349:
---
Labels: stale-assigned (was: )
> Spark runner should provide a default BoM
> -
>
> Key: BEAM-349
> URL: https://issues.apache.org/jira/browse/BEAM-349
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: P2
> Labels: stale-assigned
>
> When an user defines a pipeline, he has to define the following dependencies
> in order to work with the Spark runner (in addition of the Spark runner
> artifact itself):
> {code}
>
> org.apache.spark
> spark-core_2.10
> ${spark.version}
>
>
> org.apache.spark
> spark-streaming_2.10
> ${spark.version}
>
>
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
>
>
> com.fasterxml.jackson.core
> jackson-annotations
> ${jackson.version}
>
>
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
>
>
> com.fasterxml.jackson.module
> jackson-module-scala_2.10
> ${jackson.version}
>
> {code}
> These dependencies should come as transitive dependencies from the Spark
> runner (as Flink or dataflow runners do). It means we should change the scope
> of the dependencies in the Spark runner pom to create a BoM (Bill of
> Material).
> I will experiment a bit (with [~amitsela]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-1330) DatastoreIO Writes should flush early when duplicate keys arrive.
[ https://issues.apache.org/jira/browse/BEAM-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1330: Labels: stale-assigned (was: ) > DatastoreIO Writes should flush early when duplicate keys arrive. > - > > Key: BEAM-1330 > URL: https://issues.apache.org/jira/browse/BEAM-1330 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli >Priority: P3 > Labels: stale-assigned > > DatastoreIO writes batches upto 500 entities (rpc limit for Cloud Datastore), > before flushing them out. The writes are non-transactional and thus do not > support duplicate keys in the writes. This can be problem, especially when > using a non global windowing, where multiple windows for the same key end up > in the same batch, and prevents the writes from succeeding. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-92) Data-dependent sinks
[
https://issues.apache.org/jira/browse/BEAM-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132181#comment-17132181
]
Beam JIRA Bot commented on BEAM-92:
---
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Data-dependent sinks
>
>
> Key: BEAM-92
> URL: https://issues.apache.org/jira/browse/BEAM-92
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: P2
> Labels: stale-P2
>
> Current sink API writes all data to a single destination, but there are many
> use cases where different pieces of data need to be routed to different
> destinations where the set of destinations is data-dependent (so can't be
> implemented with a Partition transform).
> One internally discussed proposal was an API of the form:
> {code}
> PCollection PCollection.apply(
> Write.using(DoFn where,
> MapFn> how)
> {code}
> so an item T gets written to a destination (or multiple destinations)
> determined by "where"; and the writing strategy is determined by "how" that
> produces a WriteOperation (current API - global init/write/global finalize
> hooks) for any given destination.
> This API also has other benefits:
> * allows the SinkT to be computed dynamically (in "where"), rather than
> specified at pipeline construction time
> * removes the necessity for a Sink class entirely
> * is sequenceable w.r.t. downstream transforms (you can stick transforms onto
> the returned PCollection, while the current Write.to() returns a PDone)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-4152) Support Go session windowing
[ https://issues.apache.org/jira/browse/BEAM-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4152: Labels: stale-assigned (was: ) > Support Go session windowing > > > Key: BEAM-4152 > URL: https://issues.apache.org/jira/browse/BEAM-4152 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Robert Burke >Priority: P2 > Labels: stale-assigned > Time Spent: 4h 20m > Remaining Estimate: 0h > > Support session windowing and how to handle merging windows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2363) Function composition for SerializableFunction
[ https://issues.apache.org/jira/browse/BEAM-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2363: Labels: stale-assigned triaged (was: triaged) > Function composition for SerializableFunction > - > > Key: BEAM-2363 > URL: https://issues.apache.org/jira/browse/BEAM-2363 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Affects Versions: 2.0.0 >Reporter: Wesley Tanaka >Assignee: Wesley Tanaka >Priority: P3 > Labels: stale-assigned, triaged > Time Spent: 20m > Remaining Estimate: 0h > > https://lists.apache.org/thread.html/0adff14ac37b798726ad2dd4b107ed4095a5204800d501cb88057820@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4070) Disable cython profiling by default
[ https://issues.apache.org/jira/browse/BEAM-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4070: Labels: stale-assigned (was: ) > Disable cython profiling by default > --- > > Key: BEAM-4070 > URL: https://issues.apache.org/jira/browse/BEAM-4070 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Labels: stale-assigned > Time Spent: 2h > Remaining Estimate: 0h > > Enabling cython profiling adds some overhead. > http://cython.readthedocs.io/en/latest/src/tutorial/profiling_tutorial.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4132) Element type inference doesn't work for multi-output DoFns
[
https://issues.apache.org/jira/browse/BEAM-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132236#comment-17132236
]
Beam JIRA Bot commented on BEAM-4132:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Element type inference doesn't work for multi-output DoFns
> --
>
> Key: BEAM-4132
> URL: https://issues.apache.org/jira/browse/BEAM-4132
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.4.0
>Reporter: Chuan Yu Foo
>Assignee: Udi Meiri
>Priority: P2
> Labels: stale-assigned
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> TLDR: if you have a multi-output DoFn, then the non-main PCollections with
> incorrectly have their element types set to None. This affects type checking
> for pipelines involving these PCollections.
> Minimal example:
> {code}
> import apache_beam as beam
> class TripleDoFn(beam.DoFn):
> def process(self, elem):
> yield_elem
> if elem % 2 == 0:
> yield beam.pvalue.TaggedOutput('ten_times', elem * 10)
> if elem % 3 == 0:
> yield beam.pvalue.TaggedOutput('hundred_times', elem * 100)
>
> @beam.typehints.with_input_types(int)
> @beam.typehints.with_output_types(int)
> class MultiplyBy(beam.DoFn):
> def __init__(self, multiplier):
> self._multiplier = multiplier
> def process(self, elem):
> return elem * self._multiplier
>
> def main():
> with beam.Pipeline() as p:
> x, a, b = (
> p
> | 'Create' >> beam.Create([1, 2, 3])
> | 'TripleDo' >> beam.ParDo(TripleDoFn()).with_outputs(
> 'ten_times', 'hundred_times', main='main_output'))
> _ = a | 'MultiplyBy2' >> beam.ParDo(MultiplyBy(2))
> if __name__ == '__main__':
> main()
> {code}
> Running this yields the following error:
> {noformat}
> apache_beam.typehints.decorators.TypeCheckError: Type hint violation for
> 'MultiplyBy2': requires but got None for elem
> {noformat}
> Replacing {{a}} with {{b}} yields the same error. Replacing {{a}} with {{x}}
> instead yields the following error:
> {noformat}
> apache_beam.typehints.decorators.TypeCheckError: Type hint violation for
> 'MultiplyBy2': requires but got Union[TaggedOutput, int] for elem
> {noformat}
> I would expect Beam to correctly infer that {{a}} and {{b}} have element
> types of {{int}} rather than {{None}}, and I would also expect Beam to
> correctly figure out that the element types of {{x}} are compatible with
> {{int}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4114) Allow null as leftNullValue/rightNullValue in Join.fullOuterJoin()
[ https://issues.apache.org/jira/browse/BEAM-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132239#comment-17132239 ] Beam JIRA Bot commented on BEAM-4114: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Allow null as leftNullValue/rightNullValue in Join.fullOuterJoin() > -- > > Key: BEAM-4114 > URL: https://issues.apache.org/jira/browse/BEAM-4114 > Project: Beam > Issue Type: Improvement > Components: extensions-java-join-library >Affects Versions: 2.4.0 >Reporter: Robin Trietsch >Assignee: Robin Trietsch >Priority: P2 > Labels: stale-assigned > > When using the > [Join.fullOuterJoin()|https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org/apache/beam/sdk/extensions/joinlibrary/Join.html#fullOuterJoin-org.apache.beam.sdk.values.PCollection-org.apache.beam.sdk.values.PCollection-V1-V2-], > a checkNotNull() is done for the > [leftNullValue|https://github.com/apache/beam/blob/master/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java#L207] > and > [rightNullValue|https://github.com/apache/beam/blob/master/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java#L208]. > However, it makes more sense to allow null values, since sometimes, if the > key used for the join is not the same, you'd like to see that the value will > become null. This should be decided by the developer, and not by the join > library. > Looking at the source code, this is also supported by > [KV.of()|https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/values/KV.java#L42] > (it allows null values), which is used in Join.fullOuterJoin(). > If required, I can create a pull request on GitHub. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9354) How long does PubSubIO message deduplication last?
[
https://issues.apache.org/jira/browse/BEAM-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-9354:
Labels: gcp pubsubio (was: gcp pubsubio stale-assigned)
> How long does PubSubIO message deduplication last?
> --
>
> Key: BEAM-9354
> URL: https://issues.apache.org/jira/browse/BEAM-9354
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Reporter: Tianzi Cai
>Priority: P2
> Labels: gcp, pubsubio
>
> GCP documentation heavily
> [promotes|https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub]
> Beam's PubSubIO for Pub/Sub message deduplication. Yet nowhere in the
> documentation, including the [source
> code|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java],
> tells users how long this deduplication is supposed to last.
> In
> [`PubsubIO.java`|https://github.com/apache/beam/blob/a24bc3bae54f089b93bd66a118bd4bf09dbc9254/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L842-L853]:
> {code:java}
> /**
> * When reading from Cloud Pub/Sub where unique record identifiers are
> provided as Pub/Sub
> * message attributes, specifies the name of the attribute containing the
> unique identifier. The
> * value of the attribute can be any string that uniquely identifies this
> record.
> *
> * Pub/Sub cannot guarantee that no duplicate data will be delivered
> on the Pub/Sub stream.
> * If {@code idAttribute} is not provided, Beam cannot guarantee that no
> duplicate data will be
> * delivered, and deduplication of the stream will be strictly best
> effort.
> */
> public Read withIdAttribute(String idAttribute) {
> return toBuilder().setIdAttribute(idAttribute).build();
> }
> {code}
> This information here isn't enough for users to know if a second message,
> published with the same custom IdAttribute as that of a first message, which
> was published `x` minutes ago, would be deduplicated by the Dataflow runner.
> Better documentation will help. I imagine a lot of users will wonder about
> this and may even ask how to configure this period, but that will probably
> need a separate ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-3680) Revise code on state & timers blog posts
[ https://issues.apache.org/jira/browse/BEAM-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-3680: Labels: stale-assigned (was: ) > Revise code on state & timers blog posts > > > Key: BEAM-3680 > URL: https://issues.apache.org/jira/browse/BEAM-3680 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: P2 > Labels: stale-assigned > > It looks like the code is either truncated wrong when I moved it to the post, > or implementation has skewed. Perhaps it can be integrated with our snippets > capabilities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1605) Add support for Apex cluster metrics to PerfKit Benchmarker
[ https://issues.apache.org/jira/browse/BEAM-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1605: Labels: stale-assigned (was: ) > Add support for Apex cluster metrics to PerfKit Benchmarker > --- > > Key: BEAM-1605 > URL: https://issues.apache.org/jira/browse/BEAM-1605 > Project: Beam > Issue Type: Bug > Components: runner-apex, testing >Reporter: Jason Kuster >Assignee: Jason Kuster >Priority: P2 > Labels: stale-assigned > > See > https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73#heading=h.exn0s6jsm24q > for more details on what this entails. > Blocked on BEAM-1599, adding support for Apex to PKB -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-1759) add guideline doc 'run on YARN' in page 'Apache Flink Runner'
[ https://issues.apache.org/jira/browse/BEAM-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132440#comment-17132440 ] Beam JIRA Bot commented on BEAM-1759: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > add guideline doc 'run on YARN' in page 'Apache Flink Runner' > - > > Key: BEAM-1759 > URL: https://issues.apache.org/jira/browse/BEAM-1759 > Project: Beam > Issue Type: Task > Components: website >Reporter: Mingmin Xu >Assignee: Mingmin Xu >Priority: P2 > Labels: stale-assigned > > Run on YARN is another option for developers, would like to add doc to show > how to use it, including reference resources in Flink website. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2404) BigQueryIO reading stalls if no data is returned by query
[
https://issues.apache.org/jira/browse/BEAM-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132399#comment-17132399
]
Beam JIRA Bot commented on BEAM-2404:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> BigQueryIO reading stalls if no data is returned by query
> -
>
> Key: BEAM-2404
> URL: https://issues.apache.org/jira/browse/BEAM-2404
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.0.0
>Reporter: Andre
>Assignee: Chamikara Madhusanka Jayalath
>Priority: P2
> Labels: stale-assigned
> Fix For: Not applicable
>
>
> When running a BigQueryIO query that doesn't return any rows (e.g. nothing
> has changed in a delta job) the job seems to stall and nothing happens as no
> temp files are being written which I think might be what it is waiting for.
> Just adding one row to the source table will make the job run through
> successfully.
> Code:
> {code:java}
> PCollection rows = p.apply("ReadFromBQ",
> BigQueryIO.read()
> .fromQuery("SELECT * FROM `myproject.dataset.table`")
> .withoutResultFlattening().usingStandardSql());
> {code}
>
> Log:
> {code:java}
> Jun 02, 2017 9:00:36 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl
> startJob
> INFO: Started BigQuery job: {jobId=beam_job_batch-query,
> projectId=my-project}.
> bq show -j --format=prettyjson --project_id=my-project beam_job_batch-query
> Jun 02, 2017 9:03:11 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
> INFO: Starting BigQuery extract job: beam_job_batch-extract
> Jun 02, 2017 9:03:12 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl
> startJob
> INFO: Started BigQuery job: {jobId=beam_job_batch-extract,
> projectId=my-project}.
> bq show -j --format=prettyjson --project_id=my-project beam_job_batch-extract
> Jun 02, 2017 9:04:06 AM
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
> INFO: BigQuery extract job completed: beam_job_batch-extract
> Jun 02, 2017 9:04:08 AM org.apache.beam.sdk.io.FileBasedSource
> expandFilePattern
> INFO: Matched 1 files for pattern
> gs://my-bucket/tmp/BigQueryExtractTemp/ff594d003c6440a1ad84b9e02858b5c6/.avro
> Jun 02, 2017 9:04:09 AM org.apache.beam.sdk.io.FileBasedSource
> getEstimatedSizeBytes
> INFO: Filepattern
> gs://my-bucket/tmp/BigQueryExtractTemp/ff594d003c6440a1ad84b9e02858b5c6/.avro
> matched 1 files with total size 9750
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-2085) Transforms could give hints to runners on resource requirements
[ https://issues.apache.org/jira/browse/BEAM-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2085: Labels: stale-assigned (was: ) > Transforms could give hints to runners on resource requirements > --- > > Key: BEAM-2085 > URL: https://issues.apache.org/jira/browse/BEAM-2085 > Project: Beam > Issue Type: Improvement > Components: beam-model, runner-core, sdk-java-core >Affects Versions: Not applicable >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: P2 > Labels: stale-assigned > > As discussed in BEAM-673 runners can allocate workers to accomplish their > work in a better way if they can take into account some hints from the > transforms, e.g. a source can hint data locality and with this information > the runner can allocate the workers in an better way, this can also be the > case with a particular transform that can suggest to the runner to be > executed in a worker with a specific resource, e.g. GPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2693) GearpumpRunner Fn API based ParDo operator
[ https://issues.apache.org/jira/browse/BEAM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2693: Labels: stale-assigned (was: ) > GearpumpRunner Fn API based ParDo operator > -- > > Key: BEAM-2693 > URL: https://issues.apache.org/jira/browse/BEAM-2693 > Project: Beam > Issue Type: New Feature > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: P2 > Labels: stale-assigned > > This is copied from Kenn's description in other jiras. > To run non-Java SDK code is to put together an operator that manages a Fn API > client DoFnRunner and an SDK harness Fn API server. > (filing to organize steps, details of this may evolve as it is implemented). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1525) RuntimeWarning: Unable to load plugin beam_test_plugin when using nosetests
[ https://issues.apache.org/jira/browse/BEAM-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1525: Labels: stale-assigned test (was: test) > RuntimeWarning: Unable to load plugin beam_test_plugin when using nosetests > --- > > Key: BEAM-1525 > URL: https://issues.apache.org/jira/browse/BEAM-1525 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: María GH >Assignee: Mark Liu >Priority: P3 > Labels: stale-assigned, test > Fix For: Not applicable > > > Tests pass, but when using nosetests, in this manner: > nosetests --logging-level=INFO apache_beam/io/fileio_test.py:TestFileSink > the RuntimeWarnings shows up. It doesn't when using tox > $ nosetests --logging-level=INFO apache_beam/io/fileio_test.py:TestFileSink > /Users/mariagh/Documents/venvs/no-dataflow/lib/python2.7/site-packages/nose/plugins/manager.py:395: > RuntimeWarning: Unable to load plugin beam_test_plugin = > test_config:BeamTestPlugin: No module named test_config -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2696) Support new State API in GearpumpRunner
[ https://issues.apache.org/jira/browse/BEAM-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2696: Labels: stale-assigned (was: ) > Support new State API in GearpumpRunner > > > Key: BEAM-2696 > URL: https://issues.apache.org/jira/browse/BEAM-2696 > Project: Beam > Issue Type: New Feature > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: P2 > Labels: stale-assigned > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-454) Validate Pubsub Topic exists when reading
[ https://issues.apache.org/jira/browse/BEAM-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132519#comment-17132519 ] Beam JIRA Bot commented on BEAM-454: This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Validate Pubsub Topic exists when reading > - > > Key: BEAM-454 > URL: https://issues.apache.org/jira/browse/BEAM-454 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Frances Perry >Assignee: Borisa Zivkovic >Priority: P3 > Labels: newbie, stale-assigned, starter > > When reading from Pubsub, we should validate the pubsub topic exists at graph > construction time (similar to the way we validate a BQ dataset and table > exist). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3289) Add ReadFromBigQuery and several other cleanups of bigquery.py
[ https://issues.apache.org/jira/browse/BEAM-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-3289: Labels: stale-assigned (was: ) > Add ReadFromBigQuery and several other cleanups of bigquery.py > -- > > Key: BEAM-3289 > URL: https://issues.apache.org/jira/browse/BEAM-3289 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: stale-assigned > > We need to do following cleanups for Python BigQuery module. > * Add ReadFromBigQuery that wraps Read(BigQuerySource). > * Make WriteToBigQuery to use Write(BigQuery) for batch pipelines so that > both both batch and streaming users can use that. > * Update documentation of WriteToBigQuery -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-482) Log RPC requests when they fails to improve error messages
[
https://issues.apache.org/jira/browse/BEAM-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-482:
---
Labels: stale-assigned (was: )
> Log RPC requests when they fails to improve error messages
> --
>
> Key: BEAM-482
> URL: https://issues.apache.org/jira/browse/BEAM-482
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: P3
> Labels: stale-assigned
>
> For example, BigQuery sometime doesn't provide enough information in the
> respond. Users can only know what is invalid in the request, if we log the it.
> Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
> 400 Bad Request
> {
> "code" : 400,
> "errors" : [ {
> "domain" : "global",
> "message" : "Invalid value for: String is not a valid value",
> "reason" : "invalid"
> } ],
> "message" : "Invalid value for: String is not a valid value"
> }
> at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:427)
> at
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:416)
> at
> com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.getOrCreateTable(BigQueryTableInserter.java:373)
> at
> com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.getOrCreateTable(BigQueryIO.java:2127)
> at
> com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.finishBundle(BigQueryIO.java:2099)
> at
> com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.finishBundle(DoFnRunnerBase.java:158)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-2792) Populate All Runner API Components from the Python SDK
[ https://issues.apache.org/jira/browse/BEAM-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132354#comment-17132354 ] Beam JIRA Bot commented on BEAM-2792: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Populate All Runner API Components from the Python SDK > -- > > Key: BEAM-2792 > URL: https://issues.apache.org/jira/browse/BEAM-2792 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: P2 > Labels: stale-assigned > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure
[
https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132345#comment-17132345
]
Beam JIRA Bot commented on BEAM-2837:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Writing To Spanner From Google Cloud DataFlow - Failure
> ---
>
> Key: BEAM-2837
> URL: https://issues.apache.org/jira/browse/BEAM-2837
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0, 2.4.0
> Environment: Google Cloud DataFlow
>Reporter: Al Yaros
>Assignee: Mairbek Khadikov
>Priority: P2
> Labels: stale-assigned
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with
> cryptic error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>
> com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>
> com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>
> com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)*
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9932) Add documentation describing cross-language test pipelines
[ https://issues.apache.org/jira/browse/BEAM-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132194#comment-17132194 ] Beam JIRA Bot commented on BEAM-9932: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Add documentation describing cross-language test pipelines > -- > > Key: BEAM-9932 > URL: https://issues.apache.org/jira/browse/BEAM-9932 > Project: Beam > Issue Type: Improvement > Components: cross-language >Reporter: Chamikara Madhusanka Jayalath >Assignee: Kevin Sijo Puthusseri >Priority: P2 > Labels: stale-assigned > > We designed cross-language test pipelines [1][2] based on the discussion in > [3]. > Adding some pydocs and Java docs regarding rational behind each pipeline will > be helpful. > [1] > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] > [2] > [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] > [3] > [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable
[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132218#comment-17132218 ] Beam JIRA Bot commented on BEAM-4379: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Make ParquetIO Read splittable > -- > > Key: BEAM-4379 > URL: https://issues.apache.org/jira/browse/BEAM-4379 > Project: Beam > Issue Type: Improvement > Components: io-ideas, io-java-parquet >Reporter: Lukasz Gajowy >Assignee: Heejong Lee >Priority: P2 > Labels: stale-assigned > > As the title stands - currently it is not splittable which is not optimal for > runners that support splitting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1754) Will Dataflow ever support Node.js with an SDK similar to Java or Python?
[ https://issues.apache.org/jira/browse/BEAM-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1754: Labels: node.js stale-assigned (was: node.js) > Will Dataflow ever support Node.js with an SDK similar to Java or Python? > - > > Key: BEAM-1754 > URL: https://issues.apache.org/jira/browse/BEAM-1754 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas >Reporter: Diego Zuluaga >Assignee: Ashwin Ramaswami >Priority: P2 > Labels: node.js, stale-assigned > > I like the philosophy behind DataFlow and found the Java and Python samples > highly comprehensible. However, I have to admit that for most Node.js > developers who have little background on typed languages and are used to get > up to speed with frameworks incredibly fast, learning Dataflow might take > some learning curve that they/we're not used to. So, I wonder if at any point > in time Dataflow will provide a Node.js SDK. Maybe this is out of the > question, but I wanted to run it by the team as it would be awesome to have > something along these lines! > Thanks, > Diego > Question originaly posted in SO: > http://stackoverflow.com/questions/42893436/will-dataflow-ever-support-node-js-with-and-sdk-similar-to-java-or-python -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1800) Can't save datastore objects
[
https://issues.apache.org/jira/browse/BEAM-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-1800:
Labels: stale-assigned (was: )
> Can't save datastore objects
>
>
> Key: BEAM-1800
> URL: https://issues.apache.org/jira/browse/BEAM-1800
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Mike Lambert
>Assignee: Vikas Kedigehalli
>Priority: P2
> Labels: stale-assigned
>
> I can't seem to save my database objects using {{WriteToDatastore}}, as it
> errors out on a strange unicode issue when trying to write a batch.
> Stacktrace follows:
> {noformat}
> File "apache_beam/runners/common.py", line 195, in
> apache_beam.runners.common.DoFnRunner.receive
> (apache_beam/runners/common.c:5142)
> self.process(windowed_value)
> File "apache_beam/runners/common.py", line 267, in
> apache_beam.runners.common.DoFnRunner.process
> (apache_beam/runners/common.c:7201)
> self.reraise_augmented(exn)
> File "apache_beam/runners/common.py", line 279, in
> apache_beam.runners.common.DoFnRunner.reraise_augmented
> (apache_beam/runners/common.c:7590)
> raise type(exn), args, sys.exc_info()[2]
> File "apache_beam/runners/common.py", line 263, in
> apache_beam.runners.common.DoFnRunner.process
> (apache_beam/runners/common.c:7090)
> self._dofn_simple_invoker(element)
> File "apache_beam/runners/common.py", line 198, in
> apache_beam.runners.common.DoFnRunner._dofn_simple_invoker
> (apache_beam/runners/common.c:5262)
> self._process_outputs(element, self.dofn_process(element.value))
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/datastoreio.py",
> line 354, in process
> self._flush_batch()
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/datastoreio.py",
> line 363, in _flush_batch
> helper.write_mutations(self._datastore, self._project, self._mutations)
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py",
> line 187, in write_mutations
> commit(commit_request)
> File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py",
> line 174, in wrapper
> return fun(*args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py",
> line 185, in commit
> datastore.commit(req)
> File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py",
> line 140, in commit
> datastore_pb2.CommitResponse)
> File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py",
> line 199, in _call_method
> method='POST', body=payload, headers=headers)
> File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line
> 631, in new_request
> redirections, connection_type)
> File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line
> 1609, in request (response, content)
> = self._request(conn, authority, uri, request_uri, method, body, headers,
> redirections, cachekey)
> File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line
> 1351, in _request (response, content)
> = self._conn_request(conn, request_uri, method, body, headers)
> File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line
> 1273, in _conn_request
> conn.request(method, request_uri, body, headers)
> File "/usr/lib/python2.7/httplib.py", line 1039, in request
> self._send_request(method, url, body, headers)
> File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request
>self.endheaders(body)
> File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders
> self._send_output(message_body)
> File "/usr/lib/python2.7/httplib.py", line 877, in _send_output
> msg += message_body TypeError: must be str, not unicode
> [while running 'write to datastore/Convert to Mutation']
> {noformat}
> My code is basically:
> {noformat}
> | 'convert from entity' >> beam.Map(ConvertFromEntity)
> | 'write to datastore' >> WriteToDatastore(client.project)
> {noformat}
> Where {{ConvertFromEntity}} converts from a google.cloud.datastore object
> (which has a nice API/interface) into the underlying protobuf (which is what
> the beam gcp/datastore library expects):
> {noformat}
> from google.cloud.datastore import helpers
> def ConvertFromEntity(entity):
> return helpers.entity_to_protobuf(entity)
> {noformat}
> I assume entity_to_protobuf works fine/normally, since it's also what is used
> by {{google/cloud/datastore/batch.py}} to write a bunch of
> {{entity_pb2.Entity}} objects into the
> {{datastore_pb2.CommitRequest.mutations[n].upsert}}:
> In batch.py: {{put() -> _assign_entity_to_pb() -> entity_to_protobuf()}}.
> In datastoreio.py:
>
[jira] [Updated] (BEAM-2416) Add AkkaIO
[ https://issues.apache.org/jira/browse/BEAM-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2416: Labels: stale-assigned (was: ) > Add AkkaIO > -- > > Key: BEAM-2416 > URL: https://issues.apache.org/jira/browse/BEAM-2416 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: P2 > Labels: stale-assigned > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-621) Add MapValues and MapKeys functions
[
https://issues.apache.org/jira/browse/BEAM-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132513#comment-17132513
]
Beam JIRA Bot commented on BEAM-621:
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Add MapValues and MapKeys functions
> ---
>
> Key: BEAM-621
> URL: https://issues.apache.org/jira/browse/BEAM-621
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core, sdk-py-core
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: P2
> Labels: newbie, stale-assigned, starter
>
> Currently, we have the {{MapElements}} {{PTransform}} that "convert" a
> {{PCollection}} of {{KV}} to another {{PCollection}} (for instance
> {{String}}).
> A very classic mapping function is to just have the keys or values of {{KV}}.
> To do it currently, we can use {{MapElements}} or a generic {{ParDo}} (with
> {{DoFn}}).
> It would be helpful and reduce the user code to have {{MapValues}} and
> {{MapKeys}}. It would take a {{PCollection}} of {{KV}}: {{MapKeys}} will map
> the input {{PCollection}} to a {{PCollection}} of {{K}} and {{MapValues}} to
> a {{PCollection}} of {{V}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-3937) Track performance of output_counter before feature enabled
[ https://issues.apache.org/jira/browse/BEAM-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-3937: Labels: stale-assigned (was: ) > Track performance of output_counter before feature enabled > -- > > Key: BEAM-3937 > URL: https://issues.apache.org/jira/browse/BEAM-3937 > Project: Beam > Issue Type: Task > Components: sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Labels: stale-assigned > > To track performance of output_counter before feature enabled by default. > Need to make sure there is no adversely reflect before enbale > Related PR: [https://github.com/apache/beam/pull/4741] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-1800) Can't save datastore objects
[
https://issues.apache.org/jira/browse/BEAM-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132437#comment-17132437
]
Beam JIRA Bot commented on BEAM-1800:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Can't save datastore objects
>
>
> Key: BEAM-1800
> URL: https://issues.apache.org/jira/browse/BEAM-1800
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Mike Lambert
>Assignee: Vikas Kedigehalli
>Priority: P2
> Labels: stale-assigned
>
> I can't seem to save my database objects using {{WriteToDatastore}}, as it
> errors out on a strange unicode issue when trying to write a batch.
> Stacktrace follows:
> {noformat}
> File "apache_beam/runners/common.py", line 195, in
> apache_beam.runners.common.DoFnRunner.receive
> (apache_beam/runners/common.c:5142)
> self.process(windowed_value)
> File "apache_beam/runners/common.py", line 267, in
> apache_beam.runners.common.DoFnRunner.process
> (apache_beam/runners/common.c:7201)
> self.reraise_augmented(exn)
> File "apache_beam/runners/common.py", line 279, in
> apache_beam.runners.common.DoFnRunner.reraise_augmented
> (apache_beam/runners/common.c:7590)
> raise type(exn), args, sys.exc_info()[2]
> File "apache_beam/runners/common.py", line 263, in
> apache_beam.runners.common.DoFnRunner.process
> (apache_beam/runners/common.c:7090)
> self._dofn_simple_invoker(element)
> File "apache_beam/runners/common.py", line 198, in
> apache_beam.runners.common.DoFnRunner._dofn_simple_invoker
> (apache_beam/runners/common.c:5262)
> self._process_outputs(element, self.dofn_process(element.value))
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/datastoreio.py",
> line 354, in process
> self._flush_batch()
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/datastoreio.py",
> line 363, in _flush_batch
> helper.write_mutations(self._datastore, self._project, self._mutations)
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py",
> line 187, in write_mutations
> commit(commit_request)
> File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py",
> line 174, in wrapper
> return fun(*args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py",
> line 185, in commit
> datastore.commit(req)
> File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py",
> line 140, in commit
> datastore_pb2.CommitResponse)
> File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py",
> line 199, in _call_method
> method='POST', body=payload, headers=headers)
> File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line
> 631, in new_request
> redirections, connection_type)
> File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line
> 1609, in request (response, content)
> = self._request(conn, authority, uri, request_uri, method, body, headers,
> redirections, cachekey)
> File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line
> 1351, in _request (response, content)
> = self._conn_request(conn, request_uri, method, body, headers)
> File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line
> 1273, in _conn_request
> conn.request(method, request_uri, body, headers)
> File "/usr/lib/python2.7/httplib.py", line 1039, in request
> self._send_request(method, url, body, headers)
> File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request
>self.endheaders(body)
> File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders
> self._send_output(message_body)
> File "/usr/lib/python2.7/httplib.py", line 877, in _send_output
> msg += message_body TypeError: must be str, not unicode
> [while running 'write to datastore/Convert to Mutation']
> {noformat}
> My code is basically:
> {noformat}
> | 'convert from entity' >> beam.Map(ConvertFromEntity)
> | 'write to datastore' >> WriteToDatastore(client.project)
> {noformat}
> Where {{ConvertFromEntity}} converts from a google.cloud.datastore object
> (which has a nice API/interface) into the underlying protobuf (which is what
> the beam gcp/datastore library expects):
> {noformat}
> from google.cloud.datastore import helpers
> def ConvertFromEntity(entity):
> return helpers.entity_to_protobuf(entity)
> {noformat}
> I assume entity_to_protobuf works fine/normally, since it's also
[jira] [Commented] (BEAM-9) Storm Runner
[ https://issues.apache.org/jira/browse/BEAM-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132530#comment-17132530 ] Beam JIRA Bot commented on BEAM-9: -- This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Storm Runner > > > Key: BEAM-9 > URL: https://issues.apache.org/jira/browse/BEAM-9 > Project: Beam > Issue Type: Wish > Components: runner-ideas >Reporter: Frances Perry >Assignee: Harsha >Priority: P2 > Labels: stale-assigned > > Gathering place for interest in a Storm runner for Beam. -- This message was sent by Atlassian Jira (v8.3.4#803005)
