[jira] [Updated] (BEAM-3558) aggregation expression can't apply to math or arithmetic expressions
[ https://issues.apache.org/jira/browse/BEAM-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-3558: -- Parent: (was: BEAM-3517) Issue Type: Improvement (was: Sub-task) > aggregation expression can't apply to math or arithmetic expressions > > > Key: BEAM-3558 > URL: https://issues.apache.org/jira/browse/BEAM-3558 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kai Jiang >Priority: Major > > fails when executing sql > 'select sum(c1)+2 from PCOLLECTION group by c2' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-3517) Support Built-in Aggregation Functions
[ https://issues.apache.org/jira/browse/BEAM-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-3517. --- Fix Version/s: Not applicable Resolution: Fixed No longer using the bucket task but just tracking bugs individually. > Support Built-in Aggregation Functions > -- > > Key: BEAM-3517 > URL: https://issues.apache.org/jira/browse/BEAM-3517 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: Not applicable > > > Support UDAF listed in Calcite. > https://calcite.apache.org/docs/reference.html#aggregate-functions -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9398) Python type hints: AbstractDoFnWrapper does not wrap setup
[ https://issues.apache.org/jira/browse/BEAM-9398?focusedWorklogId=409644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409644 ] ASF GitHub Bot logged work on BEAM-9398: Author: ASF GitHub Bot Created on: 25/Mar/20 16:57 Start Date: 25/Mar/20 16:57 Worklog Time Spent: 10m Work Description: udim commented on pull request #0: [BEAM-9398] runtime_type_check: support setup URL: https://github.com/apache/beam/pull/0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409644) Time Spent: 40m (was: 0.5h) > Python type hints: AbstractDoFnWrapper does not wrap setup > -- > > Key: BEAM-9398 > URL: https://issues.apache.org/jira/browse/BEAM-9398 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > And possibly other methods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-4388) Support optimized logical plan
[ https://issues.apache.org/jira/browse/BEAM-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-4388. --- Fix Version/s: 2.10.0 Resolution: Fixed > Support optimized logical plan > -- > > Key: BEAM-4388 > URL: https://issues.apache.org/jira/browse/BEAM-4388 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: 2.10.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Before converting into Beam Pipeline physical plan, logical plan should be > optimized and it will be super helpful for efficiently executing Beam > PTransforms pipeline. > Calcite has two ways for optimizing logical plan (HepPlanner and > VolcanoPlanner). We can support VolcanoPlanner first and apply calcite > builtin optimize rules (like > FilterJoinRule.FILTER_ON_JOIN) to sql query optimize plans. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4561) Create unit tests for BeamQueryPlanner.convertToBeamRel
[ https://issues.apache.org/jira/browse/BEAM-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-4561: -- Parent: (was: BEAM-4388) Issue Type: Improvement (was: Sub-task) > Create unit tests for BeamQueryPlanner.convertToBeamRel > --- > > Key: BEAM-4561 > URL: https://issues.apache.org/jira/browse/BEAM-4561 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > > As discussion on PR#5481, we should consider a concrete unit test for > [BeamQueryPlanner.convertToBeamRel|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L116] > when apply BeamRuleSets to optimize logical plan. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=409653=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409653 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 25/Mar/20 17:09 Start Date: 25/Mar/20 17:09 Worklog Time Spent: 10m Work Description: udim commented on pull request #11212: [BEAM-8280] Type hints via annotations snippets URL: https://github.com/apache/beam/pull/11212 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409653) Time Spent: 10h 50m (was: 10h 40m) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8307) NPE in Calcite dialect when input PCollection has logical type in schema, from JdbcIO Transform
[ https://issues.apache.org/jira/browse/BEAM-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-8307: -- Summary: NPE in Calcite dialect when input PCollection has logical type in schema, from JdbcIO Transform (was: Support for Logical types introduced by JdbcIO Transform in SqlTransform) > NPE in Calcite dialect when input PCollection has logical type in schema, > from JdbcIO Transform > --- > > Key: BEAM-8307 > URL: https://issues.apache.org/jira/browse/BEAM-8307 > Project: Beam > Issue Type: Bug > Components: dsl-sql, io-java-jdbc >Affects Versions: 2.15.0 >Reporter: Rahul Patwari >Assignee: Kishor Joshi >Priority: Major > Attachments: exception_sqlTransform.txt > > > An exception is thrown when SqlTransform is used with JdbcIO. > Schema of the Source Table: > _CREATE TABLE actor (_ > _actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,_ > _first_name VARCHAR(45) NOT NULL,_ > _last_name VARCHAR(45) NOT NULL,_ > _last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE > CURRENT_TIMESTAMP,_ > _PRIMARY KEY (actor_id),_ > _KEY idx_actor_last_name (last_name)_ > _)ENGINE=InnoDB DEFAULT CHARSET=utf8;_ > > Many of the Jdbc Types are converted to Logical_Types in Beam. > Refer: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java#L102] > The initial observation is that this code: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L207] > is returning null. > > PFA [^exception_sqlTransform.txt] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8307) NPE in Calcite dialect when input PCollection has logical type in schema, from JdbcIO Transform
[ https://issues.apache.org/jira/browse/BEAM-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Patwari reassigned BEAM-8307: --- Assignee: Rahul Patwari (was: Kishor Joshi) > NPE in Calcite dialect when input PCollection has logical type in schema, > from JdbcIO Transform > --- > > Key: BEAM-8307 > URL: https://issues.apache.org/jira/browse/BEAM-8307 > Project: Beam > Issue Type: Bug > Components: dsl-sql, io-java-jdbc >Affects Versions: 2.15.0 >Reporter: Rahul Patwari >Assignee: Rahul Patwari >Priority: Major > Attachments: exception_sqlTransform.txt > > > An exception is thrown when SqlTransform is used with JdbcIO. > Schema of the Source Table: > _CREATE TABLE actor (_ > _actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,_ > _first_name VARCHAR(45) NOT NULL,_ > _last_name VARCHAR(45) NOT NULL,_ > _last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE > CURRENT_TIMESTAMP,_ > _PRIMARY KEY (actor_id),_ > _KEY idx_actor_last_name (last_name)_ > _)ENGINE=InnoDB DEFAULT CHARSET=utf8;_ > > Many of the Jdbc Types are converted to Logical_Types in Beam. > Refer: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java#L102] > The initial observation is that this code: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L207] > is returning null. > > PFA [^exception_sqlTransform.txt] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=409660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409660 ] ASF GitHub Bot logged work on BEAM-9468: Author: ASF GitHub Bot Created on: 25/Mar/20 17:33 Start Date: 25/Mar/20 17:33 Worklog Time Spent: 10m Work Description: jaketf commented on issue #11151: [BEAM-9468] Hl7v2 io URL: https://github.com/apache/beam/pull/11151#issuecomment-603980265 Open Questions: 1. Should we remove adaptive throttling? - Seems that we're using retries in the client request initializer and right now a "bad record" will slow down the Read / Write (even though the error has nothing to do with the HL7v2 store being overwhelmed). Originally we wanted to be safe with overwhelming QPS on the HL7v2 store in batch scenarios. 1. Should we add more to the `HealthcareIOError`? - Add (processing time) Timestamp? - Add a convenience DoFn `HealthcareIOErrrorToTableRowFn` to ease writing deadletter queue to BigQuery. 1. Would it me more useful to expose an error rate metric than an error count? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409660) Time Spent: 3h 50m (was: 3h 40m) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 3h 50m > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9608) Add context managers for FnApiRunner to manage execution of each bundle
Pablo Estrada created BEAM-9608: --- Summary: Add context managers for FnApiRunner to manage execution of each bundle Key: BEAM-9608 URL: https://issues.apache.org/jira/browse/BEAM-9608 Project: Beam Issue Type: Sub-task Components: sdk-py-core Reporter: Pablo Estrada Assignee: Pablo Estrada -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=409683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409683 ] ASF GitHub Bot logged work on BEAM-9468: Author: ASF GitHub Bot Created on: 25/Mar/20 18:12 Start Date: 25/Mar/20 18:12 Worklog Time Spent: 10m Work Description: jaketf commented on issue #11151: [BEAM-9468] Hl7v2 io URL: https://github.com/apache/beam/pull/11151#issuecomment-604001261 Future improvements: - Currently this uses the alpha API because the motivating use case of HL7v2 -> FHIR mapping which requires the `schematizedData` field (not yet available in beta API). - For backwards compatibility the `ListHL7v2MessageIDs` always uses Messages.List to get message IDs and relies on the the `HL7v2IO.Read` to fetch the message contents. This provides a consistent flow for real-time and batch and provides backwards compatibility. However, in the alpha API Messages.List returns the actual message contents (rather than just the message IDs). This leads to us reading the messages contents twice for alpha HL7v2 stores. I suggest we address optimizing this "double fetch" in a future PR as the alpha API stabilizes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409683) Time Spent: 4h 40m (was: 4.5h) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 4h 40m > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9537) Refactor FnApiRunner into its own package
[ https://issues.apache.org/jira/browse/BEAM-9537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Estrada resolved BEAM-9537. - Fix Version/s: 2.21.0 Resolution: Fixed > Refactor FnApiRunner into its own package > - > > Key: BEAM-9537 > URL: https://issues.apache.org/jira/browse/BEAM-9537 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409718 ] ASF GitHub Bot logged work on BEAM-9136: Author: ASF GitHub Bot Created on: 25/Mar/20 18:45 Start Date: 25/Mar/20 18:45 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #11067: [BEAM-9136]Add licenses for dependencies URL: https://github.com/apache/beam/pull/11067#discussion_r39803 ## File path: licenses/go/bufio/LICENSE ## @@ -0,0 +1,27 @@ +Copyright (c) 2009 The Go Authors. All rights reserved. Review comment: > > I'll second the comment that we should not include the list of license files in the beam repository itself, but just add the script to the container build process to install them into the container. > > I agree. It would seem better for all languages to add the license retrieval to the container build script. I will make this change in the next commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409718) Time Spent: 4.5h (was: 4h 20m) > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=409743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409743 ] ASF GitHub Bot logged work on BEAM-3301: Author: ASF GitHub Bot Created on: 25/Mar/20 19:15 Start Date: 25/Mar/20 19:15 Worklog Time Spent: 10m Work Description: youngoli commented on issue #11225: [BEAM-3301] Fix another bug in DoFn validation, in exec. URL: https://github.com/apache/beam/pull/11225#issuecomment-604033853 R: @lostluck As a note for the future, does it seem valuable to add an integration test that catches this case? It just needs to have a GBK followed by a DoFn that reads the values as an Iter or ReIter. I can make a Jira for that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409743) Time Spent: 10h 10m (was: 10h) > Go SplittableDoFn support > - > > Key: BEAM-3301 > URL: https://issues.apache.org/jira/browse/BEAM-3301 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > > SDFs will be the only way to add streaming and liquid sharded IO for Go. > Design doc: https://s.apache.org/splittable-do-fn -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5130) Support ALL (sub query)
[ https://issues.apache.org/jira/browse/BEAM-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066820#comment-17066820 ] Kenneth Knowles commented on BEAM-5130: --- Hi! I am checking on SQL JIRAs. Is this still an issue? Are you working on it? We can unassign and let someone else work on it. Also the parent issue (just a container issue that could be a label instead) is closed so I will detach this sub-issue. > Support ALL (sub query) > --- > > Key: BEAM-5130 > URL: https://issues.apache.org/jira/browse/BEAM-5130 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > Example: > > SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM > special_item_table). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-5829) SQL should probably not support GROUP BY or set operations on floating point numbers
[ https://issues.apache.org/jira/browse/BEAM-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-5829. --- Fix Version/s: 2.10.0 Resolution: Fixed > SQL should probably not support GROUP BY or set operations on floating point > numbers > > > Key: BEAM-5829 > URL: https://issues.apache.org/jira/browse/BEAM-5829 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > These are known to be super unreliable on most SQL engines, and generally > indicate a programming error. Floating points numbers are intended as > stand-ins for real numbers, for which equality (hence grouping and set > operations) are undecidable. And our build is broken because of it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5834) Document lack of support for GROUP BY and set operations on floating points in SQL
[ https://issues.apache.org/jira/browse/BEAM-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5834: -- Parent: (was: BEAM-5829) Issue Type: Improvement (was: Sub-task) > Document lack of support for GROUP BY and set operations on floating points > in SQL > -- > > Key: BEAM-5834 > URL: https://issues.apache.org/jira/browse/BEAM-5834 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8514) ZetaSql should use cost-based optimization to take advantage of Join Reordering Rule and Push-Down Rule
[ https://issues.apache.org/jira/browse/BEAM-8514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066872#comment-17066872 ] Kenneth Knowles commented on BEAM-8514: --- I think there were some incompatibilities so some rules were disabled, but there should be a separate bug tracking anything remaining. [~amaliujia] [~apilloud] is there another Jira or is this fully resolved? > ZetaSql should use cost-based optimization to take advantage of Join > Reordering Rule and Push-Down Rule > --- > > Key: BEAM-8514 > URL: https://issues.apache.org/jira/browse/BEAM-8514 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Default config should use BeamCostModel, as well as tests with custom > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7610) SELECT COALESCE(...) isn't inferred as non-nullable field
[
https://issues.apache.org/jira/browse/BEAM-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066875#comment-17066875
]
Kenneth Knowles commented on BEAM-7610:
---
Still an issue? Still working on this?
> SELECT COALESCE(...) isn't inferred as non-nullable field
> -
>
> Key: BEAM-7610
> URL: https://issues.apache.org/jira/browse/BEAM-7610
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Assignee: Kirill Kozlov
>Priority: Major
>
> In Calcite, Coalesce is described as:
> {code}
> ReturnTypes.cascade(ReturnTypes.LEAST_RESTRICTIVE,
> SqlTypeTransforms.LEAST_NULLABLE)
> {code}
> However, giving non-null constant as an argument doesn't result in a
> non-nullable expression:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true)));
> PCollection input = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = input
> .apply(SqlTransform.query("SELECT COALESCE(name, 'unknown') as name
> FROM PCOLLECTION"))
> .getSchema();
> assertEquals(
> Schema.builder().addStringField("name").build(),
> outputSchema);
> {code}
> Not sure if it's a problem in Calcite or Beam SQL.
> There are no other functions that can be used to produce a non-nullable field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8404) [SQL] Update deprecated method calls
[ https://issues.apache.org/jira/browse/BEAM-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8404. --- Fix Version/s: 2.17.0 Resolution: Fixed > [SQL] Update deprecated method calls > > > Key: BEAM-8404 > URL: https://issues.apache.org/jira/browse/BEAM-8404 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Affects Versions: 2.15.0 >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Improve code health by moving away from using deprecated methods/classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8514) ZetaSql should use cost-based optimization to take advantage of Join Reordering Rule and Push-Down Rule
[ https://issues.apache.org/jira/browse/BEAM-8514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066886#comment-17066886 ] Andrew Pilloud commented on BEAM-8514: -- looks resolved to me. > ZetaSql should use cost-based optimization to take advantage of Join > Reordering Rule and Push-Down Rule > --- > > Key: BEAM-8514 > URL: https://issues.apache.org/jira/browse/BEAM-8514 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Default config should use BeamCostModel, as well as tests with custom > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8751) Beam Dependency Update Request: com.google.apis:google-api-services-cloudresourcemanager
[ https://issues.apache.org/jira/browse/BEAM-8751?focusedWorklogId=409670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409670 ] ASF GitHub Bot logged work on BEAM-8751: Author: ASF GitHub Bot Created on: 25/Mar/20 17:55 Start Date: 25/Mar/20 17:55 Worklog Time Spent: 10m Work Description: suztomo commented on issue #11208: [BEAM-8751] google-api-client 1.30.9 URL: https://github.com/apache/beam/pull/11208#issuecomment-603992166 Java Precommit failed but it seem irrelevant: ``` Error Message org.junit.runners.model.TestTimedOutException: test timed out after 3 milliseconds ``` https://builds.apache.org/job/beam_PreCommit_Java_Commit/10502/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409670) Time Spent: 1h 20m (was: 1h 10m) > Beam Dependency Update Request: > com.google.apis:google-api-services-cloudresourcemanager > > > Key: BEAM-8751 > URL: https://issues.apache.org/jira/browse/BEAM-8751 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > - 2019-11-19 21:04:41.938497 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191018-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:51.401493 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191115-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:09:00.761817 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191115-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:09:01.384571 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:31.850871 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:08:07.241510 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:00.916536 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 >
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=409671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409671 ] ASF GitHub Bot logged work on BEAM-9468: Author: ASF GitHub Bot Created on: 25/Mar/20 17:55 Start Date: 25/Mar/20 17:55 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11151: [BEAM-9468] Hl7v2 io URL: https://github.com/apache/beam/pull/11151#issuecomment-603992173 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409671) Time Spent: 4h (was: 3h 50m) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 4h > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9609) Upgrade to ZetaSQL 2020.03.2
Andrew Pilloud created BEAM-9609: Summary: Upgrade to ZetaSQL 2020.03.2 Key: BEAM-9609 URL: https://issues.apache.org/jira/browse/BEAM-9609 Project: Beam Issue Type: Improvement Components: dsl-sql-zetasql Reporter: Andrew Pilloud It is coming out soon with better OS X support. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amogh Tiwari resolved BEAM-8564. Fix Version/s: 2.20.0 Resolution: Resolved > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 17h 10m > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409733 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 25/Mar/20 19:02 Start Date: 25/Mar/20 19:02 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604027617 R: @robertwb could you review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409733) Time Spent: 2h 50m (was: 2h 40m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9557) Error setting processing time timers near end-of-window
[
https://issues.apache.org/jira/browse/BEAM-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067014#comment-17067014
]
Reuven Lax commented on BEAM-9557:
--
pr/11226 sent for review
> Error setting processing time timers near end-of-window
> ---
>
> Key: BEAM-9557
> URL: https://issues.apache.org/jira/browse/BEAM-9557
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Reporter: Steve Niemitz
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Previously, it was possible to set a processing time timer past the end of a
> window, and it would simply not fire.
> However, now, this results in an error:
> {code:java}
> java.lang.IllegalArgumentException: Attempted to set event time timer that
> outputs for 2020-03-19T18:01:35.000Z but that is after the expiration of
> window 2020-03-19T17:59:59.999Z
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:440)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setAndVerifyOutputTimestamp(SimpleDoFnRunner.java:1011)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setRelative(SimpleDoFnRunner.java:934)
> .processElement(???.scala:187)
> {code}
>
> I think the regression was introduced in commit
> a005fd765a762183ca88df90f261f6d4a20cf3e0. Also notice that the error message
> is wrong, it says that "event time timer" but the timer is in the processing
> time domain.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9601) Interactive test_streaming_wordcount failing
[ https://issues.apache.org/jira/browse/BEAM-9601?focusedWorklogId=409760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409760 ] ASF GitHub Bot logged work on BEAM-9601: Author: ASF GitHub Bot Created on: 25/Mar/20 19:40 Start Date: 25/Mar/20 19:40 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11227: [BEAM-9601] Skip the streaming wordcount test because it uses a Python3.5.3+ feature URL: https://github.com/apache/beam/pull/11227#discussion_r398120887 ## File path: sdks/python/apache_beam/runners/interactive/interactive_runner_test.py ## @@ -150,9 +151,10 @@ def process(self, element): ] self.assertEqual(actual_reified, expected_reified) + @unittest.skipIf( + sys.version_info < (3, 6), + 'InteractiveRunner has a minimum Python version of 3.5.3.') Review comment: should we skip if sys.version_info < 3,5,3? so we'll test on 3.5 if it's the right subversion? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409760) Time Spent: 1h 10m (was: 1h) > Interactive test_streaming_wordcount failing > > > Key: BEAM-9601 > URL: https://issues.apache.org/jira/browse/BEAM-9601 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9601) Interactive test_streaming_wordcount failing
[ https://issues.apache.org/jira/browse/BEAM-9601?focusedWorklogId=409759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409759 ] ASF GitHub Bot logged work on BEAM-9601: Author: ASF GitHub Bot Created on: 25/Mar/20 19:39 Start Date: 25/Mar/20 19:39 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11227: [BEAM-9601] Skip the streaming wordcount test because it uses a Python3.5.3+ feature URL: https://github.com/apache/beam/pull/11227#issuecomment-604045227 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409759) Time Spent: 1h (was: 50m) > Interactive test_streaming_wordcount failing > > > Key: BEAM-9601 > URL: https://issues.apache.org/jira/browse/BEAM-9601 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive, test-failures >Reporter: Pablo Estrada >Assignee: Sam Rohde >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9028) bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
[
https://issues.apache.org/jira/browse/BEAM-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Wasilewski resolved BEAM-9028.
Fix Version/s: Not applicable
Resolution: Not A Problem
> bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
> --
>
> Key: BEAM-9028
> URL: https://issues.apache.org/jira/browse/BEAM-9028
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Kamil Wasilewski
>Priority: Major
> Fix For: Not applicable
>
>
> It's not clear what the root cause is (there's also a "cannot allocate
> memory" error message).
> {code}
> 14:16:06 ERROR: test_iobase_source
> (apache_beam.io.gcp.bigquery_read_it_test.ReadTests)
> 14:16:06
> --
> 14:16:06 Traceback (most recent call last):
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py",
> line 154, in test_iobase_source
> 14:16:06 assert_that(result, equal_to(self.TABLE_DATA))
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 480, in __exit__
> 14:16:06 self.run().wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 460, in run
> 14:16:06 self._options).run(False)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 473, in run
> 14:16:06 return self.runner.run_pipeline(self, self._options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/flink_runner.py",
> line 45, in run_pipeline
> 14:16:06 return super(FlinkRunner, self).run_pipeline(pipeline, options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 330, in run_pipeline
> 14:16:06 result.wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 438, in wait_until_finish
> 14:16:06 for state_response in self._state_stream:
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 416, in __next__
> 14:16:06 return self._next()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 703, in _next
> 14:16:06 raise self
> 14:16:06 grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous
> of RPC that terminated with:
> 14:16:06 status = StatusCode.UNAVAILABLE
> 14:16:06 details = "Socket closed"
> 14:16:06 debug_error_string =
> "{"created":"@1577139361.436477926","description":"Error received from peer
> ipv4:127.0.0.1:51557","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket
> closed","grpc_status":14}"
> 14:16:06 >
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1210/timestamps/?time=HH:mm:ss=GMT-8=en_US
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-2281) call SqlFunctions in operator implementation
[
https://issues.apache.org/jira/browse/BEAM-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-2281.
---
Fix Version/s: 2.10.0
Assignee: Andrew Pilloud (was: Anton Kedin)
Resolution: Fixed
> call SqlFunctions in operator implementation
>
>
> Key: BEAM-2281
> URL: https://issues.apache.org/jira/browse/BEAM-2281
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Mingmin Xu
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Calcite has a collections of functions in
> {{org.apache.calcite.runtime.SqlFunctions}}. It sounds a good source to
> leverage when adding operators as {{BeamSqlExpression}}.
> [~xumingming] [~app-tarush], any comments?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8359) Move ZetaSQL to Calcite translator to its own module
[ https://issues.apache.org/jira/browse/BEAM-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8359. --- Fix Version/s: 2.17.0 Resolution: Fixed > Move ZetaSQL to Calcite translator to its own module > > > Key: BEAM-8359 > URL: https://issues.apache.org/jira/browse/BEAM-8359 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.17.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This module has protobuf and gRPC on its surface (for now anyhow) and is > totally independent of the core Beam SQL. Moving to its own module will > protect Beam SQL users from the protobuf and gRPC dependencies and clean the > build slightly, removing some Gradle hacks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8817) Beam SQL dialects should share table registration and name resolution logic
[ https://issues.apache.org/jira/browse/BEAM-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8817. --- Fix Version/s: Not applicable Resolution: Won't Fix > Beam SQL dialects should share table registration and name resolution logic > --- > > Key: BEAM-8817 > URL: https://issues.apache.org/jira/browse/BEAM-8817 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > > Currently the ZetaSQL and Calcite SQL code paths each have their own version > of "scrape the table names from the query" and "let the table providers know > that we will be asking for these names". There should be a common abstraction > for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8592) DataCatalogTableProvider should not squash table components together into a string
[
https://issues.apache.org/jira/browse/BEAM-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-8592.
---
Fix Version/s: 2.18.0
Resolution: Fixed
> DataCatalogTableProvider should not squash table components together into a
> string
> --
>
> Key: BEAM-8592
> URL: https://issues.apache.org/jira/browse/BEAM-8592
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, dsl-sql-zetasql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently, if a user writes a table name like \{{foo.`baz.bar`.bizzle}}
> representing the components \{{"foo", "baz.bar", "bizzle"}} the
> DataCatalogTableProvider will concatenate the components into a string and
> resolve the identifier as if it represented \{{"foo", "baz", "bar",
> "bizzle"}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8817) Beam SQL dialects should share table registration and name resolution logic
[ https://issues.apache.org/jira/browse/BEAM-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066840#comment-17066840 ] Kenneth Knowles commented on BEAM-8817: --- I spent some time on this. I wanted them to actually share the main planner. But actually each dialect has some of their own custom rules. So it is more of a fancy project to separate the planners. Calcite also really wants there to be just one planner so sharing logic can be hard. Going to close this as won't fix. > Beam SQL dialects should share table registration and name resolution logic > --- > > Key: BEAM-8817 > URL: https://issues.apache.org/jira/browse/BEAM-8817 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Currently the ZetaSQL and Calcite SQL code paths each have their own version > of "scrape the table names from the query" and "let the table providers know > that we will be asking for these names". There should be a common abstraction > for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9416) BIP-1: Convert avro metadata to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9416?focusedWorklogId=409658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409658 ] ASF GitHub Bot logged work on BEAM-9416: Author: ASF GitHub Bot Created on: 25/Mar/20 17:27 Start Date: 25/Mar/20 17:27 Worklog Time Spent: 10m Work Description: alexvanboxel commented on issue #11217: [BEAM-9416] BIP-1: Convert Avro metadata to Schema options URL: https://github.com/apache/beam/pull/11217#issuecomment-603976975 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409658) Time Spent: 20m (was: 10m) > BIP-1: Convert avro metadata to Schema options > -- > > Key: BEAM-9416 > URL: https://issues.apache.org/jira/browse/BEAM-9416 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Fix For: 2.21.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Avro has some metadata that can be added to the normal type information. It > is based on json typing, so the conversion will be best effort (probably we > can bet int, string and float out of it). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9557) Error setting processing time timers near end-of-window
[
https://issues.apache.org/jira/browse/BEAM-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066892#comment-17066892
]
Reuven Lax commented on BEAM-9557:
--
ok, this seems simple. I'll send out a PR.
> Error setting processing time timers near end-of-window
> ---
>
> Key: BEAM-9557
> URL: https://issues.apache.org/jira/browse/BEAM-9557
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Reporter: Steve Niemitz
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.20.0
>
>
> Previously, it was possible to set a processing time timer past the end of a
> window, and it would simply not fire.
> However, now, this results in an error:
> {code:java}
> java.lang.IllegalArgumentException: Attempted to set event time timer that
> outputs for 2020-03-19T18:01:35.000Z but that is after the expiration of
> window 2020-03-19T17:59:59.999Z
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:440)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setAndVerifyOutputTimestamp(SimpleDoFnRunner.java:1011)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setRelative(SimpleDoFnRunner.java:934)
> .processElement(???.scala:187)
> {code}
>
> I think the regression was introduced in commit
> a005fd765a762183ca88df90f261f6d4a20cf3e0. Also notice that the error message
> is wrong, it says that "event time timer" but the timer is in the processing
> time domain.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9607) _SDFBoundedSourceWrapper should expose the same display_data as Read
[ https://issues.apache.org/jira/browse/BEAM-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyuan Zhang updated BEAM-9607: --- Summary: _SDFBoundedSourceWrapper should expose the same display_data as Read (was: _SDFBoundedSourceWrapper should expose underlying source display_data) > _SDFBoundedSourceWrapper should expose the same display_data as Read > > > Key: BEAM-9607 > URL: https://issues.apache.org/jira/browse/BEAM-9607 > Project: Beam > Issue Type: Task > Components: io-py-gcp, sdk-py-core >Reporter: Yichi Zhang >Assignee: Boyuan Zhang >Priority: Minor > > It seems that the _SDFBoundedSourceWrapper will hide the display data added > to the source underneath. We should try to expose those data if it exists. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7610) SELECT COALESCE(...) isn't inferred as non-nullable field
[
https://issues.apache.org/jira/browse/BEAM-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066942#comment-17066942
]
Kirill Kozlov commented on BEAM-7610:
-
The underlying issue was fixed in Calcite. Updating vendored Calite to version
1.22.0 or later should fix this issue.
> SELECT COALESCE(...) isn't inferred as non-nullable field
> -
>
> Key: BEAM-7610
> URL: https://issues.apache.org/jira/browse/BEAM-7610
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Assignee: Kirill Kozlov
>Priority: Major
>
> In Calcite, Coalesce is described as:
> {code}
> ReturnTypes.cascade(ReturnTypes.LEAST_RESTRICTIVE,
> SqlTypeTransforms.LEAST_NULLABLE)
> {code}
> However, giving non-null constant as an argument doesn't result in a
> non-nullable expression:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true)));
> PCollection input = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = input
> .apply(SqlTransform.query("SELECT COALESCE(name, 'unknown') as name
> FROM PCOLLECTION"))
> .getSchema();
> assertEquals(
> Schema.builder().addStringField("name").build(),
> outputSchema);
> {code}
> Not sure if it's a problem in Calcite or Beam SQL.
> There are no other functions that can be used to produce a non-nullable field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409713 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 18:38 Start Date: 25/Mar/20 18:38 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224#issuecomment-604014835 Follow-up to https://github.com/apache/beam/pull/11165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409713) Time Spent: 3h 50m (was: 3h 40m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-1819) Key should be available in @OnTimer methods
[
https://issues.apache.org/jira/browse/BEAM-1819?focusedWorklogId=409724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409724
]
ASF GitHub Bot logged work on BEAM-1819:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:48
Start Date: 25/Mar/20 18:48
Worklog Time Spent: 10m
Work Description: rehmanmuradali commented on pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529536
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
##
@@ -1282,6 +1283,14 @@ private static Parameter analyzeExtraParameter(
rawType.equals(Instant.class),
"@Timestamp argument must have type org.joda.time.Instant.");
return Parameter.timestampParameter();
+} else if (hasAnnotation(DoFn.KeyId.class, param.getAnnotations())) {
+ Type keyType = ((ParameterizedType)
inputT.getType()).getActualTypeArguments()[0];
Review comment:
@reuvenlax , Added with a test case
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409724)
Time Spent: 11h 10m (was: 11h)
> Key should be available in @OnTimer methods
> ---
>
> Key: BEAM-1819
> URL: https://issues.apache.org/jira/browse/BEAM-1819
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Shehzaad Nakhoda
>Priority: Major
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> Every timer firing has an associated key. This key should be available when
> the timer is delivered to a user's {{DoFn}}, so they don't have to store it
> in state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9601) Interactive test_streaming_wordcount failing
[ https://issues.apache.org/jira/browse/BEAM-9601?focusedWorklogId=409758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409758 ] ASF GitHub Bot logged work on BEAM-9601: Author: ASF GitHub Bot Created on: 25/Mar/20 19:37 Start Date: 25/Mar/20 19:37 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #11227: [BEAM-9601] Skip the streaming wordcount test because it uses a Python3.5.3+ feature URL: https://github.com/apache/beam/pull/11227 Change-Id: I9caaf395fd0fc58565e54a8458e8289af761815f The InteractiveRunner requires Python3.5.3+. We have a pattern in which we disable the tests that fail on Python2.7. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8603) Add Python SqlTransform example script
[ https://issues.apache.org/jira/browse/BEAM-8603?focusedWorklogId=409775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409775 ] ASF GitHub Bot logged work on BEAM-8603: Author: ASF GitHub Bot Created on: 25/Mar/20 19:53 Start Date: 25/Mar/20 19:53 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #10055: [BEAM-8603] Add Python SqlTransform URL: https://github.com/apache/beam/pull/10055#issuecomment-604052011 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409775) Time Spent: 3h (was: 2h 50m) > Add Python SqlTransform example script > -- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[ https://issues.apache.org/jira/browse/BEAM-8078?focusedWorklogId=409777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409777 ] ASF GitHub Bot logged work on BEAM-8078: Author: ASF GitHub Bot Created on: 25/Mar/20 19:54 Start Date: 25/Mar/20 19:54 Worklog Time Spent: 10m Work Description: udim commented on pull request #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409777) Time Spent: 6h 20m (was: 6h 10m) > streaming_wordcount_debugging.py is missing a test > -- > > Key: BEAM-8078 > URL: https://issues.apache.org/jira/browse/BEAM-8078 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Aleksey Vysotin >Priority: Minor > Labels: beginner, easy, newbie, starter > Time Spent: 6h 20m > Remaining Estimate: 0h > > It's example code and should have a basic_test (like the other wordcount > variants in [1]) to at least verify that it runs in the latest Beam release. > [1] > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=409788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409788 ] ASF GitHub Bot logged work on BEAM-9468: Author: ASF GitHub Bot Created on: 25/Mar/20 20:10 Start Date: 25/Mar/20 20:10 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11151: [BEAM-9468] Hl7v2 io URL: https://github.com/apache/beam/pull/11151#issuecomment-604060180 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409788) Time Spent: 4h 50m (was: 4h 40m) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 4h 50m > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5130) Support ALL (sub query)
[ https://issues.apache.org/jira/browse/BEAM-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5130: -- Parent: (was: BEAM-5046) Issue Type: New Feature (was: Sub-task) > Support ALL (sub query) > --- > > Key: BEAM-5130 > URL: https://issues.apache.org/jira/browse/BEAM-5130 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > Example: > > SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM > special_item_table). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7832) ZetaSQL Dialect
[ https://issues.apache.org/jira/browse/BEAM-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066845#comment-17066845 ] Kenneth Knowles commented on BEAM-7832: --- Now that we have a JIRA component for the ZetaSQL dialect, I think making the bugs top level is just as good and you can do more with issues when they are top level. > ZetaSQL Dialect > --- > > Key: BEAM-7832 > URL: https://issues.apache.org/jira/browse/BEAM-7832 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Mujuzi Moses >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > We can support ZetaSQL(https://github.com/google/zetasql) dialect in BeamSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7832) ZetaSQL Dialect
[ https://issues.apache.org/jira/browse/BEAM-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7832: -- Component/s: (was: dsl-sql) > ZetaSQL Dialect > --- > > Key: BEAM-7832 > URL: https://issues.apache.org/jira/browse/BEAM-7832 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Mujuzi Moses >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > We can support ZetaSQL(https://github.com/google/zetasql) dialect in BeamSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8514) ZetaSql should use cost-based optimization to take advantage of Join Reordering Rule and Push-Down Rule
[ https://issues.apache.org/jira/browse/BEAM-8514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8514. --- Fix Version/s: 2.18.0 Resolution: Fixed > ZetaSql should use cost-based optimization to take advantage of Join > Reordering Rule and Push-Down Rule > --- > > Key: BEAM-8514 > URL: https://issues.apache.org/jira/browse/BEAM-8514 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.18.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Default config should use BeamCostModel, as well as tests with custom > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9044) BIP-1: Convert protobuf options to Schema options
[ https://issues.apache.org/jira/browse/BEAM-9044?focusedWorklogId=409659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409659 ] ASF GitHub Bot logged work on BEAM-9044: Author: ASF GitHub Bot Created on: 25/Mar/20 17:28 Start Date: 25/Mar/20 17:28 Worklog Time Spent: 10m Work Description: alexvanboxel commented on issue #10529: [BEAM-9044] Protobuf options to Schema options URL: https://github.com/apache/beam/pull/10529#issuecomment-603977599 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409659) Time Spent: 2h (was: 1h 50m) > BIP-1: Convert protobuf options to Schema options > - > > Key: BEAM-9044 > URL: https://issues.apache.org/jira/browse/BEAM-9044 > Project: Beam > Issue Type: Sub-task > Components: extensions-java-protobuf >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Protobuf has a rich metadata system called options. This system is fully > typed and matches Beams Schema Option system. For now we can only convert the > following protobuf options: > * File Options -> _Beam doesn't have this concept_ > * Message Options -> *Beam Schema Options* > * Field Options -> *Beam Schema Options* > * Enum Options -> _This can only be done when logical type options are > available_ > * EnumValue Options -> _This can only be done when logical type options are > available_ > * Service Options -> _Beam doesn't have this concept_ > * Method Options -> _Beam doesn't have this concept_ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-7610) SELECT COALESCE(...) isn't inferred as non-nullable field
[
https://issues.apache.org/jira/browse/BEAM-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066942#comment-17066942
]
Kirill Kozlov edited comment on BEAM-7610 at 3/25/20, 6:10 PM:
---
The underlying issue was fixed in Calcite. Updating vendored Calite to version
1.22.0 or later should fix this issue.
Not actively working on this, will move to Unassigned.
was (Author: kirillkozlov):
The underlying issue was fixed in Calcite. Updating vendored Calite to version
1.22.0 or later should fix this issue.
> SELECT COALESCE(...) isn't inferred as non-nullable field
> -
>
> Key: BEAM-7610
> URL: https://issues.apache.org/jira/browse/BEAM-7610
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Priority: Major
>
> In Calcite, Coalesce is described as:
> {code}
> ReturnTypes.cascade(ReturnTypes.LEAST_RESTRICTIVE,
> SqlTypeTransforms.LEAST_NULLABLE)
> {code}
> However, giving non-null constant as an argument doesn't result in a
> non-nullable expression:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true)));
> PCollection input = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = input
> .apply(SqlTransform.query("SELECT COALESCE(name, 'unknown') as name
> FROM PCOLLECTION"))
> .getSchema();
> assertEquals(
> Schema.builder().addStringField("name").build(),
> outputSchema);
> {code}
> Not sure if it's a problem in Calcite or Beam SQL.
> There are no other functions that can be used to produce a non-nullable field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409682 ] ASF GitHub Bot logged work on BEAM-9136: Author: ASF GitHub Bot Created on: 25/Mar/20 18:10 Start Date: 25/Mar/20 18:10 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #11067: [BEAM-9136]Add licenses for dependencies URL: https://github.com/apache/beam/pull/11067#issuecomment-60356 > I'll second the comment that we should not include the list of license files in the beam repository itself, but just add the script to the container build process to install them into the container. I agree. It would seem better for all languages to add the license retrieval to the container build script. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409682) Time Spent: 3.5h (was: 3h 20m) > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7609) SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
[
https://issues.apache.org/jira/browse/BEAM-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066946#comment-17066946
]
Kirill Kozlov commented on BEAM-7609:
-
Not sure if this issue is still reproducible.
Not actively working on this, will move to Unassigned.
> SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
> ---
>
> Key: BEAM-7609
> URL: https://issues.apache.org/jira/browse/BEAM-7609
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Priority: Major
>
> Works in sqlline shell:
> {code}
> Welcome to Beam SQL 2.14.0-SNAPSHOT (based on sqlline version 1.4.0)
> 0: BeamSQL> CREATE EXTERNAL TABLE s1 (id BIGINT) TYPE 'test';
> No rows affected (0.507 seconds)
> 0: BeamSQL> CREATE EXTERNAL TABLE s2 (id BIGINT) TYPE 'test';
> No rows affected (0.004 seconds)
> 0: BeamSQL> SELECT DISTINCT s1.id as lhs, s2.id as rhs FROM s1 JOIN s2 USING
> (id);
> +-+-+
> | lhs | rhs |
> +-+-+
> +-+-+
> No rows selected (2.568 seconds)
> {code}
> But doesn't work in the test:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("id", Schema.FieldType.INT32));
> PCollection i1 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> PCollection i2 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = PCollectionTuple
> .of("i1", i1)
> .and("i2", i2)
> .apply(SqlTransform.query("SELECT DISTINCT s1.id as lhs, s2.id as rhs
> FROM i1 JOIN i2 USING (id)"))
> .getSchema();
> assertEquals(ImmutableList.of("lhs", "rhs"),
> outputSchema.getFieldNames());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9371) Implement SideInput load test in Java
[ https://issues.apache.org/jira/browse/BEAM-9371?focusedWorklogId=409732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409732 ] ASF GitHub Bot logged work on BEAM-9371: Author: ASF GitHub Bot Created on: 25/Mar/20 19:01 Start Date: 25/Mar/20 19:01 Worklog Time Spent: 10m Work Description: Ardagan commented on issue #10949: [BEAM-9371] Add SideInputLoadTest to Java SDK URL: https://github.com/apache/beam/pull/10949#issuecomment-604026937 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409732) Time Spent: 4h 20m (was: 4h 10m) > Implement SideInput load test in Java > - > > Key: BEAM-9371 > URL: https://issues.apache.org/jira/browse/BEAM-9371 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8751) Beam Dependency Update Request: com.google.apis:google-api-services-cloudresourcemanager
[ https://issues.apache.org/jira/browse/BEAM-8751?focusedWorklogId=409741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409741 ] ASF GitHub Bot logged work on BEAM-8751: Author: ASF GitHub Bot Created on: 25/Mar/20 19:12 Start Date: 25/Mar/20 19:12 Worklog Time Spent: 10m Work Description: aaltay commented on issue #11208: [BEAM-8751] google-api-client 1.30.9 URL: https://github.com/apache/beam/pull/11208#issuecomment-604032702 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409741) Time Spent: 1h 40m (was: 1.5h) > Beam Dependency Update Request: > com.google.apis:google-api-services-cloudresourcemanager > > > Key: BEAM-8751 > URL: https://issues.apache.org/jira/browse/BEAM-8751 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > - 2019-11-19 21:04:41.938497 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191018-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:51.401493 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191115-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:09:00.761817 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191115-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:09:01.384571 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:31.850871 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:08:07.241510 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:00.916536 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20
[jira] [Updated] (BEAM-9608) Add context managers for FnApiRunner to manage execution of each bundle
[ https://issues.apache.org/jira/browse/BEAM-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Estrada updated BEAM-9608: Status: Open (was: Triage Needed) > Add context managers for FnApiRunner to manage execution of each bundle > --- > > Key: BEAM-9608 > URL: https://issues.apache.org/jira/browse/BEAM-9608 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9610) BigQueryWriteIntegrationTests.test_big_query_write flaky
Udi Meiri created BEAM-9610:
---
Summary: BigQueryWriteIntegrationTests.test_big_query_write flaky
Key: BEAM-9610
URL: https://issues.apache.org/jira/browse/BEAM-9610
Project: Beam
Issue Type: Bug
Components: io-py-gcp, test-failures
Reporter: Udi Meiri
{code}
Expected: (Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4, 'привет')])
but: Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4, 'привет')]
Actual data is []
{code}
https://builds.apache.org/job/beam_PostCommit_Python37_PR/118/
https://builds.apache.org/job/beam_PostCommit_Python35/2081/
https://builds.apache.org/job/beam_PostCommit_Python35/2079/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-3647) Default Coder/Reading Coder From File
[ https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3647: - Assignee: Reuven Lax (was: Anton Kedin) > Default Coder/Reading Coder From File > -- > > Key: BEAM-3647 > URL: https://issues.apache.org/jira/browse/BEAM-3647 > Project: Beam > Issue Type: New Feature > Components: beam-model, dsl-sql >Affects Versions: 2.2.0 >Reporter: Kishan Kumar >Assignee: Reuven Lax >Priority: Major > > *Requirement*-: Need to Run Template With Same Logics on Different Tables > Data.(Example is Given Below) > > *Need*: Default Coder is Required So According to Data It Make All Fields as > String and Read Data else Thier must be Dynamic Options to Read Coder From > GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location > Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider. > > > *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, > SUB_PRICE) > And 2 Table is (NAME, ROLL, SUB, TEST_MARKS) > > On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can > Read Coder at Run Time The Same Template Can Be Used For Different Tables at > Run Time. > > Such Situations Make Our Work Easy and Make Our job Easy. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3647) Default Coder/Reading Coder From File
[ https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-3647: -- Component/s: (was: dsl-sql) > Default Coder/Reading Coder From File > -- > > Key: BEAM-3647 > URL: https://issues.apache.org/jira/browse/BEAM-3647 > Project: Beam > Issue Type: New Feature > Components: beam-model >Affects Versions: 2.2.0 >Reporter: Kishan Kumar >Assignee: Reuven Lax >Priority: Major > > *Requirement*-: Need to Run Template With Same Logics on Different Tables > Data.(Example is Given Below) > > *Need*: Default Coder is Required So According to Data It Make All Fields as > String and Read Data else Thier must be Dynamic Options to Read Coder From > GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location > Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider. > > > *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, > SUB_PRICE) > And 2 Table is (NAME, ROLL, SUB, TEST_MARKS) > > On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can > Read Coder at Run Time The Same Template Can Be Used For Different Tables at > Run Time. > > Such Situations Make Our Work Easy and Make Our job Easy. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3647) Default Coder/Reading Coder From File
[ https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066812#comment-17066812 ] Kenneth Knowles commented on BEAM-3647: --- Is all the new schema support helping this? > Default Coder/Reading Coder From File > -- > > Key: BEAM-3647 > URL: https://issues.apache.org/jira/browse/BEAM-3647 > Project: Beam > Issue Type: New Feature > Components: beam-model >Affects Versions: 2.2.0 >Reporter: Kishan Kumar >Assignee: Reuven Lax >Priority: Major > > *Requirement*-: Need to Run Template With Same Logics on Different Tables > Data.(Example is Given Below) > > *Need*: Default Coder is Required So According to Data It Make All Fields as > String and Read Data else Thier must be Dynamic Options to Read Coder From > GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location > Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider. > > > *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, > SUB_PRICE) > And 2 Table is (NAME, ROLL, SUB, TEST_MARKS) > > On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can > Read Coder at Run Time The Same Template Can Be Used For Different Tables at > Run Time. > > Such Situations Make Our Work Easy and Make Our job Easy. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9036) Failure in beam_PostCommit_SQL: testSQLReadAndWriteWithSameFlatTableDefinition
[
https://issues.apache.org/jira/browse/BEAM-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-9036.
---
Fix Version/s: Not applicable
Resolution: Fixed
> Failure in beam_PostCommit_SQL: testSQLReadAndWriteWithSameFlatTableDefinition
> --
>
> Key: BEAM-9036
> URL: https://issues.apache.org/jira/browse/BEAM-9036
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, test-failures
>Reporter: Udi Meiri
>Assignee: Daniel Collins
>Priority: Major
> Fix For: Not applicable
>
>
> Culprit seems to be:
> https://github.com/apache/beam/pull/10331#issuecomment-569309669
> Error 1:
> {code}
> java.lang.AssertionError:
> Expected: iterable with items [hasProperty("payload", is <{name=person1,
> height=80}>), hasProperty("payload", is <{name=person3, height=60}>),
> hasProperty("payload", is <{name=person5, height=40}>)] in any order
> but: no item matches: hasProperty("payload", is <{name=person1,
> height=80}>), hasProperty("payload", is <{name=person3, height=60}>),
> hasProperty("payload", is <{name=person5, height=40}>) in []
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.lambda$assertThatTopicEventuallyReceives$1(TestPubsub.java:262)
> at
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testSQLReadAndWriteWithSameFlatTableDefinition(PubsubJsonIT.java:571)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:101)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:101)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:101)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal$1.evaluate(TestPubsubSignal.java:116)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Resolved] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug
[ https://issues.apache.org/jira/browse/BEAM-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-9203. --- Fix Version/s: 2.20.0 Resolution: Fixed > Programmatically determine if SQL exception is user error, unsupported, or bug > -- > > Key: BEAM-9203 > URL: https://issues.apache.org/jira/browse/BEAM-9203 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Right now there are a lot exceptions thrown by the Calcite SQL dialect and > ZetaSQL dialect of Beam SQL. It is hard to catch just the errors that are > user errors, or just the errors that are unsupported operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9562) Remove timer from PCollection and treat timers as Elements
[ https://issues.apache.org/jira/browse/BEAM-9562?focusedWorklogId=409641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409641 ] ASF GitHub Bot logged work on BEAM-9562: Author: ASF GitHub Bot Created on: 25/Mar/20 16:52 Start Date: 25/Mar/20 16:52 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #11216: [BEAM-9562] Remove TimerSpec from ParDoPayload URL: https://github.com/apache/beam/pull/11216#discussion_r398011965 ## File path: sdks/python/apache_beam/transforms/userstate.py ## @@ -141,6 +143,13 @@ def __init__(self, name, time_domain): def __repr__(self): return '%s(%s)' % (self.__class__.__name__, self.name) + def to_timer_family_runner_api(self, context): Review comment: I'm working on the Java changes meanwhile having some python tests to run to catch any breakages. Changing to `to_runner_api` is the next step. We should also remove `TimerSpec` definition from proto as long as nowhere else uses it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409641) Time Spent: 2h 50m (was: 2h 40m) > Remove timer from PCollection and treat timers as Elements > --- > > Key: BEAM-9562 > URL: https://issues.apache.org/jira/browse/BEAM-9562 > Project: Beam > Issue Type: New Feature > Components: sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8307) NPE in Calcite dialect when input PCollection has logical type in schema, from JdbcIO Transform
[ https://issues.apache.org/jira/browse/BEAM-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066881#comment-17066881 ] Kenneth Knowles commented on BEAM-8307: --- Working on this? If not, we can unassign and let someone else pick up the work. > NPE in Calcite dialect when input PCollection has logical type in schema, > from JdbcIO Transform > --- > > Key: BEAM-8307 > URL: https://issues.apache.org/jira/browse/BEAM-8307 > Project: Beam > Issue Type: Bug > Components: dsl-sql, io-java-jdbc >Affects Versions: 2.15.0 >Reporter: Rahul Patwari >Assignee: Kishor Joshi >Priority: Major > Attachments: exception_sqlTransform.txt > > > An exception is thrown when SqlTransform is used with JdbcIO. > Schema of the Source Table: > _CREATE TABLE actor (_ > _actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,_ > _first_name VARCHAR(45) NOT NULL,_ > _last_name VARCHAR(45) NOT NULL,_ > _last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE > CURRENT_TIMESTAMP,_ > _PRIMARY KEY (actor_id),_ > _KEY idx_actor_last_name (last_name)_ > _)ENGINE=InnoDB DEFAULT CHARSET=utf8;_ > > Many of the Jdbc Types are converted to Logical_Types in Beam. > Refer: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java#L102] > The initial observation is that this code: > [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L207] > is returning null. > > PFA [^exception_sqlTransform.txt] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?focusedWorklogId=409663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409663 ] ASF GitHub Bot logged work on BEAM-9573: Author: ASF GitHub Bot Created on: 25/Mar/20 17:43 Start Date: 25/Mar/20 17:43 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp URL: https://github.com/apache/beam/pull/11220#issuecomment-603985830 LGTM after necessary tests pass This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409663) Time Spent: 5h 40m (was: 5.5h) > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7610) SELECT COALESCE(...) isn't inferred as non-nullable field
[
https://issues.apache.org/jira/browse/BEAM-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Kozlov reassigned BEAM-7610:
---
Assignee: (was: Kirill Kozlov)
> SELECT COALESCE(...) isn't inferred as non-nullable field
> -
>
> Key: BEAM-7610
> URL: https://issues.apache.org/jira/browse/BEAM-7610
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Priority: Major
>
> In Calcite, Coalesce is described as:
> {code}
> ReturnTypes.cascade(ReturnTypes.LEAST_RESTRICTIVE,
> SqlTypeTransforms.LEAST_NULLABLE)
> {code}
> However, giving non-null constant as an argument doesn't result in a
> non-nullable expression:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true)));
> PCollection input = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = input
> .apply(SqlTransform.query("SELECT COALESCE(name, 'unknown') as name
> FROM PCOLLECTION"))
> .getSchema();
> assertEquals(
> Schema.builder().addStringField("name").build(),
> outputSchema);
> {code}
> Not sure if it's a problem in Calcite or Beam SQL.
> There are no other functions that can be used to produce a non-nullable field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9608) Add context managers for FnApiRunner to manage execution of each bundle
[
https://issues.apache.org/jira/browse/BEAM-9608?focusedWorklogId=409712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409712
]
ASF GitHub Bot logged work on BEAM-9608:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:36
Start Date: 25/Mar/20 18:36
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #11202:
[BEAM-9608] Refactoring FnApiRunner to have more context managers
URL: https://github.com/apache/beam/pull/11202#discussion_r398079535
##
File path:
sdks/python/apache_beam/runners/portability/fn_api_runner/execution.py
##
@@ -0,0 +1,337 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Set of utilities for execution of a pipeline by the FnApiRunner."""
+
+from __future__ import absolute_import
+
+import collections
+import itertools
+
+from typing_extensions import Protocol
+
+from apache_beam import coders
+from apache_beam.coders.coder_impl import create_InputStream
+from apache_beam.coders.coder_impl import create_OutputStream
+from apache_beam.portability import common_urns
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.runners.portability.fn_api_runner.translations import
only_element
+from apache_beam.runners.portability.fn_api_runner.translations import
split_buffer_id
+from apache_beam.runners.worker import bundle_processor
+from apache_beam.transforms import trigger
+from apache_beam.transforms.window import GlobalWindow
+from apache_beam.transforms.window import GlobalWindows
+from apache_beam.utils import windowed_value
+
+
+class Buffer(Protocol):
+ def __iter__(self):
+# type: () -> Iterator[bytes]
+pass
+
+ def append(self, item):
+# type: (bytes) -> None
+pass
+
+
+class PartitionableBuffer(Buffer, Protocol):
+ def partition(self, n):
+# type: (int) -> List[List[bytes]]
+pass
+
+
+class ListBuffer(object):
+ """Used to support parititioning of a list."""
+ def __init__(self, coder_impl):
+self._coder_impl = coder_impl
+self._inputs = [] # type: List[bytes]
+self._grouped_output = None
+self.cleared = False
+
+ def append(self, element):
+# type: (bytes) -> None
+if self.cleared:
+ raise RuntimeError('Trying to append to a cleared ListBuffer.')
+if self._grouped_output:
+ raise RuntimeError('ListBuffer append after read.')
+self._inputs.append(element)
+
+ def partition(self, n):
+# type: (int) -> List[List[bytes]]
+if self.cleared:
+ raise RuntimeError('Trying to partition a cleared ListBuffer.')
+if len(self._inputs) >= n or len(self._inputs) == 0:
+ return [self._inputs[k::n] for k in range(n)]
+else:
+ if not self._grouped_output:
+output_stream_list = [create_OutputStream() for _ in range(n)]
+idx = 0
+for input in self._inputs:
+ input_stream = create_InputStream(input)
+ while input_stream.size() > 0:
+decoded_value = self._coder_impl.decode_from_stream(
+input_stream, True)
+self._coder_impl.encode_to_stream(
+decoded_value, output_stream_list[idx], True)
+idx = (idx + 1) % n
+self._grouped_output = [[output_stream.get()]
+for output_stream in output_stream_list]
+ return self._grouped_output
+
+ def __iter__(self):
+# type: () -> Iterator[bytes]
+if self.cleared:
+ raise RuntimeError('Trying to iterate through a cleared ListBuffer.')
+return iter(self._inputs)
+
+ def clear(self):
+# type: () -> None
+self.cleared = True
+self._inputs = []
+self._grouped_output = None
+
+
+class GroupingBuffer(object):
+ """Used to accumulate groupded (shuffled) results."""
+ def __init__(self,
+ pre_grouped_coder, # type: coders.Coder
+ post_grouped_coder, # type: coders.Coder
+ windowing
+ ):
+# type: (...) -> None
+self._key_coder = pre_grouped_coder.key_coder()
+self._pre_grouped_coder = pre_grouped_coder
+self._post_grouped_coder = post_grouped_coder
+self._table =
[jira] [Work logged] (BEAM-9578) Enumerating artifacts is too expensive in Java
[ https://issues.apache.org/jira/browse/BEAM-9578?focusedWorklogId=409734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409734 ] ASF GitHub Bot logged work on BEAM-9578: Author: ASF GitHub Bot Created on: 25/Mar/20 19:04 Start Date: 25/Mar/20 19:04 Worklog Time Spent: 10m Work Description: ihji commented on issue #11205: [BEAM-9578] Enumerating artifacts is too expensive in Java URL: https://github.com/apache/beam/pull/11205#issuecomment-604028675 directory type and unzip_to role added. didn't create a interface for resolving artifacts since we only have one (resolving directory) now. maybe refactor later for supporting more resolvers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409734) Time Spent: 1h 10m (was: 1h) > Enumerating artifacts is too expensive in Java > -- > > Key: BEAM-9578 > URL: https://issues.apache.org/jira/browse/BEAM-9578 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Heejong Lee >Priority: Critical > Labels: portability > Fix For: 2.21.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > There are a lot of places (e.g. *ParDoTranslation#getParDoPayload*) which > effectively call *Environments#createOrGetDefaultEnvironment* which causes > [artifacts to be > computed|https://github.com/apache/beam/blob/fc6cef9972780ca6b7525d4aadd65a8344221f1b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L114]. > This leads to zipping directories for non-jar dependencies. > Similar problems may exist for Python/Go. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=409742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409742 ] ASF GitHub Bot logged work on BEAM-3301: Author: ASF GitHub Bot Created on: 25/Mar/20 19:13 Start Date: 25/Mar/20 19:13 Worklog Time Spent: 10m Work Description: youngoli commented on pull request #11225: [BEAM-3301] Fix another bug in DoFn validation, in exec. URL: https://github.com/apache/beam/pull/11225 This was the same error from the previous bugfix, but I missed it in the exec package since I only checked locally. Instead of fixing it the same way as before, I'm loosening the strictness of the validation since we probably don't need fully strict validation in the exec package. At this point, every DoFn should already have been validated by the graph package. Previous PR is #11179 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[ https://issues.apache.org/jira/browse/BEAM-8078?focusedWorklogId=409773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409773 ] ASF GitHub Bot logged work on BEAM-8078: Author: ASF GitHub Bot Created on: 25/Mar/20 19:52 Start Date: 25/Mar/20 19:52 Worklog Time Spent: 10m Work Description: udim commented on issue #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#issuecomment-604051436 Opened https://issues.apache.org/jira/browse/BEAM-9610 for the flaky postcommit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409773) Time Spent: 6h 10m (was: 6h) > streaming_wordcount_debugging.py is missing a test > -- > > Key: BEAM-8078 > URL: https://issues.apache.org/jira/browse/BEAM-8078 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Aleksey Vysotin >Priority: Minor > Labels: beginner, easy, newbie, starter > Time Spent: 6h 10m > Remaining Estimate: 0h > > It's example code and should have a basic_test (like the other wordcount > variants in [1]) to at least verify that it runs in the latest Beam release. > [1] > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9610) BigQueryWriteIntegrationTests.test_big_query_write flaky
[
https://issues.apache.org/jira/browse/BEAM-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9610:
---
Assignee: Juta Staes
> BigQueryWriteIntegrationTests.test_big_query_write flaky
>
>
> Key: BEAM-9610
> URL: https://issues.apache.org/jira/browse/BEAM-9610
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, test-failures
>Reporter: Udi Meiri
>Assignee: Juta Staes
>Priority: Major
>
> {code}
> Expected: (Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4,
> 'привет')])
> but: Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4, 'привет')]
> Actual data is []
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37_PR/118/
> https://builds.apache.org/job/beam_PostCommit_Python35/2081/
> https://builds.apache.org/job/beam_PostCommit_Python35/2079/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8040) NPE in table name resolver when selecting from a table that doesn't exist
[
https://issues.apache.org/jira/browse/BEAM-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-8040.
---
Fix Version/s: 2.17.0
Resolution: Fixed
> NPE in table name resolver when selecting from a table that doesn't exist
> -
>
> Key: BEAM-8040
> URL: https://issues.apache.org/jira/browse/BEAM-8040
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.17.0
>
>
> NullPointerException when selecting from a table that doesn't exist.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolverImpl.assumeLeafIsTable(TableResolverImpl.java:42)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.resolveCalciteTable(TableResolution.java:48)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.addTableToLeafCatalog(SqlAnalyzer.java:174)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.lambda$createPopulatedCatalog$0(SqlAnalyzer.java:132)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8040) NPE in table name resolver when selecting from a table that doesn't exist
[
https://issues.apache.org/jira/browse/BEAM-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066843#comment-17066843
]
Kenneth Knowles commented on BEAM-8040:
---
I think the PR I just added, plus my work on nulls.
> NPE in table name resolver when selecting from a table that doesn't exist
> -
>
> Key: BEAM-8040
> URL: https://issues.apache.org/jira/browse/BEAM-8040
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kenneth Knowles
>Priority: Major
>
> NullPointerException when selecting from a table that doesn't exist.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolverImpl.assumeLeafIsTable(TableResolverImpl.java:42)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.resolveCalciteTable(TableResolution.java:48)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.addTableToLeafCatalog(SqlAnalyzer.java:174)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.SqlAnalyzer.lambda$createPopulatedCatalog$0(SqlAnalyzer.java:132)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8445) ZetaSQL translator returns null from unsupported methods
[ https://issues.apache.org/jira/browse/BEAM-8445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-8445. --- Fix Version/s: 2.17.0 Resolution: Fixed > ZetaSQL translator returns null from unsupported methods > > > Key: BEAM-8445 > URL: https://issues.apache.org/jira/browse/BEAM-8445 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.17.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This is an error-prone style that was missed because our null analysis is > disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409647 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 17:02 Start Date: 25/Mar/20 17:02 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165#issuecomment-603962869 Now org.apache.beam.sdk.io.cassandra.CassandraIOTest.classMethod is failing. These are unrelated and didn't fail before. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409647) Time Spent: 3h 20m (was: 3h 10m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409648 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 17:02 Start Date: 25/Mar/20 17:02 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409648) Time Spent: 3.5h (was: 3h 20m) > Properly populate pipeline proto requirements. > -- > > Key: BEAM-9340 > URL: https://issues.apache.org/jira/browse/BEAM-9340 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?focusedWorklogId=409656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409656 ] ASF GitHub Bot logged work on BEAM-9573: Author: ASF GitHub Bot Created on: 25/Mar/20 17:23 Start Date: 25/Mar/20 17:23 Worklog Time Spent: 10m Work Description: mxm commented on pull request #11220: [BEAM-9573][release-2.0] Correct computing of watermark hold for timer output timestamp URL: https://github.com/apache/beam/pull/11220 ### Backport of #11200. With the introduction of timer output timestamps, a new watermark hold had been added to the Flink Runner. The watermark computation works on the keyed state backend which computes a key-scoped watermark hold and not the desired operator-wide watermark hold. Computation: https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 Key-scoped state: https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 The solution is to iterate over all available state backend keys. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute
[ https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=409667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409667 ] ASF GitHub Bot logged work on BEAM-4150: Author: ASF GitHub Bot Created on: 25/Mar/20 17:51 Start Date: 25/Mar/20 17:51 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11222: [BEAM-4150] Don't window PCollection coders. URL: https://github.com/apache/beam/pull/11222 Now that no SDKs require it. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8751) Beam Dependency Update Request: com.google.apis:google-api-services-cloudresourcemanager
[ https://issues.apache.org/jira/browse/BEAM-8751?focusedWorklogId=409681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409681 ] ASF GitHub Bot logged work on BEAM-8751: Author: ASF GitHub Bot Created on: 25/Mar/20 18:08 Start Date: 25/Mar/20 18:08 Worklog Time Spent: 10m Work Description: suztomo commented on issue #11208: [BEAM-8751] google-api-client 1.30.9 URL: https://github.com/apache/beam/pull/11208#issuecomment-603992166 Java Precommit failed but it seem irrelevant: ``` Error Message org.junit.runners.model.TestTimedOutException: test timed out after 3 milliseconds ``` https://builds.apache.org/job/beam_PreCommit_Java_Commit/10502/ . There's an email thread "FlinkRequiresStableInput test is very flaky" and [BEAM-9578](https://issues.apache.org/jira/browse/BEAM-9578) to fix the timeout. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409681) Time Spent: 1.5h (was: 1h 20m) > Beam Dependency Update Request: > com.google.apis:google-api-services-cloudresourcemanager > > > Key: BEAM-8751 > URL: https://issues.apache.org/jira/browse/BEAM-8751 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > - 2019-11-19 21:04:41.938497 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191018-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:51.401493 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191115-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:09:00.761817 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191115-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:09:01.384571 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:04:31.850871 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:08:07.241510 > - > Please consider upgrading the dependency > com.google.apis:google-api-services-cloudresourcemanager. > The current version is v1-rev20181015-1.28.0. The latest version is > v2-rev20191206-1.30.3 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:00.916536 > - > Please consider upgrading the dependency >
[jira] [Assigned] (BEAM-7609) SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
[
https://issues.apache.org/jira/browse/BEAM-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Kozlov reassigned BEAM-7609:
---
Assignee: (was: Kirill Kozlov)
> SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
> ---
>
> Key: BEAM-7609
> URL: https://issues.apache.org/jira/browse/BEAM-7609
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Priority: Major
>
> Works in sqlline shell:
> {code}
> Welcome to Beam SQL 2.14.0-SNAPSHOT (based on sqlline version 1.4.0)
> 0: BeamSQL> CREATE EXTERNAL TABLE s1 (id BIGINT) TYPE 'test';
> No rows affected (0.507 seconds)
> 0: BeamSQL> CREATE EXTERNAL TABLE s2 (id BIGINT) TYPE 'test';
> No rows affected (0.004 seconds)
> 0: BeamSQL> SELECT DISTINCT s1.id as lhs, s2.id as rhs FROM s1 JOIN s2 USING
> (id);
> +-+-+
> | lhs | rhs |
> +-+-+
> +-+-+
> No rows selected (2.568 seconds)
> {code}
> But doesn't work in the test:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("id", Schema.FieldType.INT32));
> PCollection i1 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> PCollection i2 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = PCollectionTuple
> .of("i1", i1)
> .and("i2", i2)
> .apply(SqlTransform.query("SELECT DISTINCT s1.id as lhs, s2.id as rhs
> FROM i1 JOIN i2 USING (id)"))
> .getSchema();
> assertEquals(ImmutableList.of("lhs", "rhs"),
> outputSchema.getFieldNames());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409706
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398071329
##
File path: sdks/go/container/license_scripts/pull_licenses_go.py
##
@@ -0,0 +1,132 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""A script to pull licenses for Go.
+"""
+
+import os
+import re
+import shutil
+import subprocess
+
+from tenacity import retry
+from tenacity import stop_after_attempt
+
+def run_bash_command(command):
+ process = subprocess.Popen(command.split(),
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ result, error = process.communicate()
+ if error:
+raise RuntimeError('Error occurred when running a bash command.',
+ 'command: ', command, 'error message: ',
+ error.decode('utf-8'))
+ return result.decode('utf-8')
+
+
+def get_final_dependencies(deps):
+ final_deps = set()
+ for dep in deps:
+# remove Beam internal dependencies.
+if dep.startswith('github.com/apache/beam'):
+ continue
+if dep.startswith('vendor/'):
+ dep = dep.replace('vendor/', '')
+# the list contains all nested dependencies, following if statements
+# dedup nested dependencies and includes root dependencies only.
+# if dep is from github.com, ex: github.com/golang/protobuf/proto
+if dep.startswith('github.com'):
+ final_deps.add('/'.join(dep.split('/')[:3]))
+# if dep is from google.golang.org, ex:google.golang.org/grpc
+elif dep.startswith('google.golang.org'):
+ final_deps.add('/'.join(dep.split('/')[:2]))
+# if dep is from golang.org, ex: golang.org/x/net/http2
+elif dep.startswith('golang.org'):
+ final_deps.add('/'.join(dep.split('/')[:3]))
+else: # embedded dependencies, ex: debug, crypto/tls
+ final_deps.add(dep.split('/')[0])
+ return final_deps
+
+
+def get_dependency_list():
+ command = "go list -f '{{.Deps}}' github.com/apache/beam/sdks/go/pkg/beam"
+ dependencies = run_bash_command(command)
+ # dependencies returned from the command is '[dep0 dpe1 ...]'.
+ # "'", "[", "]" should be removed from the bytes.
+ str_dependencies = re.sub(r"([\'\[\]])", r"", dependencies)
+ final_dependencies = get_final_dependencies(str_dependencies.split())
+ return final_dependencies
+
+
+@retry(stop=stop_after_attempt(3))
+def pull_license(dep):
Review comment:
I see many duplicates in go licenses. Can only unique licenses be stored?
docker run -it --entrypoint bash apache/beam_go_sdk:latest
find /opt/apache/beam/licenses/ -type f -exec sha1sum {} \; | awk '{print
$1}' | sort | uniq -c
Shows only 3 unique licenses:
2 2b8b815229aa8a61e483fb4ba0588b8b6c491890
1 aa9b240f558caed367795f667629ccbca28f20b2
36 d6a5f1ecaedd723c325a2063375b3517e808a2b5
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409706)
Time Spent: 4h 10m (was: 4h)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Scan dependencies and add
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409707
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398077003
##
File path: sdks/python/container/py37/build.gradle
##
@@ -51,6 +51,11 @@ task copyLauncherDependencies(type: Copy) {
}
}
+task copyDependencyLicenses(type: Copy) {
Review comment:
Can this task be added to
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy in
applyDockerNature?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409707)
Time Spent: 4h 20m (was: 4h 10m)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409708
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398076817
##
File path: sdks/python/container/py35/build.gradle
##
@@ -51,6 +51,11 @@ task copyLauncherDependencies(type: Copy) {
}
}
+task copyDependencyLicenses(type: Copy) {
Review comment:
Can this task be added to
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy in
applyDockerNature?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409708)
Time Spent: 4h 20m (was: 4h 10m)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=409700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409700 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 25/Mar/20 18:35 Start Date: 25/Mar/20 18:35 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11224: [BEAM-9340] Validate pipeline requirements in PipelineValidator. URL: https://github.com/apache/beam/pull/11224 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409703 ] ASF GitHub Bot logged work on BEAM-9136: Author: ASF GitHub Bot Created on: 25/Mar/20 18:35 Start Date: 25/Mar/20 18:35 Worklog Time Spent: 10m Work Description: alanmyrvold commented on pull request #11067: [BEAM-9136]Add licenses for dependencies URL: https://github.com/apache/beam/pull/11067#discussion_r398076198 ## File path: licenses/java/index.html ## @@ -0,0 +1,6290 @@ + Review comment: Does this file need to be checked in? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409703) Time Spent: 3h 50m (was: 3h 40m) > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409701
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398067882
##
File path: sdks/go/container/build.gradle
##
@@ -45,6 +45,11 @@ golang {
}
}
+task copyDependencyLicenses(type: Copy) {
Review comment:
I don't see a task for running pull_licenses_go.py ... can that be added?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409701)
Time Spent: 3h 40m (was: 3.5h)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409704
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398073526
##
File path: sdks/java/container/build.gradle
##
@@ -82,3 +87,4 @@ docker {
// Ensure that we build the required resources and copy and file dependencies
from related projects
dockerPrepare.dependsOn goBuild
dockerPrepare.dependsOn copyDockerfileDependencies
+dockerPrepare.dependsOn copyDependencyLicenses
Review comment:
Can this dependency be added to
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy in
applyDockerNature?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409704)
Time Spent: 4h (was: 3h 50m)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409702 ] ASF GitHub Bot logged work on BEAM-9136: Author: ASF GitHub Bot Created on: 25/Mar/20 18:35 Start Date: 25/Mar/20 18:35 Worklog Time Spent: 10m Work Description: alanmyrvold commented on pull request #11067: [BEAM-9136]Add licenses for dependencies URL: https://github.com/apache/beam/pull/11067#discussion_r398065394 ## File path: licenses/go/bufio/LICENSE ## @@ -0,0 +1,27 @@ +Copyright (c) 2009 The Go Authors. All rights reserved. Review comment: Instead of checking the LICENSE files in, can these be gathered during container creation? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409702) Time Spent: 3h 40m (was: 3.5h) > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409710
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398076913
##
File path: sdks/python/container/py36/build.gradle
##
@@ -51,6 +51,11 @@ task copyLauncherDependencies(type: Copy) {
}
}
+task copyDependencyLicenses(type: Copy) {
Review comment:
Can this task be added to
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy in
applyDockerNature?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409710)
Time Spent: 4h 20m (was: 4h 10m)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409705
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398073171
##
File path: sdks/java/container/build.gradle
##
@@ -68,6 +68,11 @@ golang {
}
}
+task copyDependencyLicenses(type: Copy) {
Review comment:
Can this task be added to
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy in
applyDockerNature?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409705)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[
https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=409709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409709
]
ASF GitHub Bot logged work on BEAM-9136:
Author: ASF GitHub Bot
Created on: 25/Mar/20 18:35
Start Date: 25/Mar/20 18:35
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on pull request #11067:
[BEAM-9136]Add licenses for dependencies
URL: https://github.com/apache/beam/pull/11067#discussion_r398076695
##
File path: sdks/python/container/py2/build.gradle
##
@@ -51,6 +51,11 @@ task copyLauncherDependencies(type: Copy) {
}
}
+task copyDependencyLicenses(type: Copy) {
Review comment:
Can this task be added to
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy in
applyDockerNature?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409709)
Time Spent: 4h 20m (was: 4h 10m)
> Add LICENSES and NOTICES to docker images
> -
>
> Key: BEAM-9136
> URL: https://issues.apache.org/jira/browse/BEAM-9136
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Scan dependencies and add licenses and notices of the dependencies to SDK
> docker images.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9557) Error setting processing time timers near end-of-window
[
https://issues.apache.org/jira/browse/BEAM-9557?focusedWorklogId=409749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409749
]
ASF GitHub Bot logged work on BEAM-9557:
Author: ASF GitHub Bot
Created on: 25/Mar/20 19:20
Start Date: 25/Mar/20 19:20
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #11226: [BEAM-9557]
Fix timer window boundary checking
URL: https://github.com/apache/beam/pull/11226
R: @steveniemitz
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 409749)
Remaining Estimate: 0h
Time Spent: 10m
> Error setting processing time timers near end-of-window
> ---
>
> Key: BEAM-9557
> URL: https://issues.apache.org/jira/browse/BEAM-9557
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Reporter: Steve Niemitz
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.20.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Previously, it was possible to set a processing time timer past the end of a
> window, and it would simply not fire.
> However, now, this results in an error:
> {code:java}
> java.lang.IllegalArgumentException: Attempted to set event time timer that
> outputs for 2020-03-19T18:01:35.000Z but that is after the expiration of
> window 2020-03-19T17:59:59.999Z
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:440)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setAndVerifyOutputTimestamp(SimpleDoFnRunner.java:1011)
>
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$TimerInternalsTimer.setRelative(SimpleDoFnRunner.java:934)
> .processElement(???.scala:187)
> {code}
>
> I think the regression was introduced in commit
> a005fd765a762183ca88df90f261f6d4a20cf3e0. Also notice that the error message
> is wrong, it says that "event time timer" but the timer is in the processing
> time domain.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5111) SUM0/SUM
[ https://issues.apache.org/jira/browse/BEAM-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066825#comment-17066825 ] Kenneth Knowles commented on BEAM-5111: --- Hi! I am checking on SQL JIRAs. Is this still an issue? Are you working on it? We can unassign and let someone else work on it. Also the parent issue (just a container issue that could be a label instead) is closed so I will detach this sub-issue. > SUM0/SUM > > > Key: BEAM-5111 > URL: https://issues.apache.org/jira/browse/BEAM-5111 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > SUM and SUM0 share the same code, either one could be wrong. Should fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4561) Create unit tests for BeamQueryPlanner.convertToBeamRel
[ https://issues.apache.org/jira/browse/BEAM-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066827#comment-17066827 ] Kenneth Knowles commented on BEAM-4561: --- Hi! I am checking on SQL JIRAs. Is this still an issue? Are you working on it? For this one maybe we can close it since it is just an idea that we might do something. > Create unit tests for BeamQueryPlanner.convertToBeamRel > --- > > Key: BEAM-4561 > URL: https://issues.apache.org/jira/browse/BEAM-4561 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > > As discussion on PR#5481, we should consider a concrete unit test for > [BeamQueryPlanner.convertToBeamRel|https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamQueryPlanner.java#L116] > when apply BeamRuleSets to optimize logical plan. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9606) Example in gradle's combine test lacks of parameters
[ https://issues.apache.org/jira/browse/BEAM-9606?focusedWorklogId=409639=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409639 ] ASF GitHub Bot logged work on BEAM-9606: Author: ASF GitHub Bot Created on: 25/Mar/20 16:50 Start Date: 25/Mar/20 16:50 Worklog Time Spent: 10m Work Description: kamilwu commented on pull request #11219: [BEAM-9606] Add missing parameters in combine_test.py gradle example URL: https://github.com/apache/beam/pull/11219 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409639) Time Spent: 40m (was: 0.5h) > Example in gradle's combine test lacks of parameters > > > Key: BEAM-9606 > URL: https://issues.apache.org/jira/browse/BEAM-9606 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Affects Versions: Not applicable >Reporter: Piotr Szuberski >Assignee: Piotr Szuberski >Priority: Minor > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > The combine test example run with gradle lacks of fanout and top_count params -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9606) Example in gradle's combine test lacks of parameters
[ https://issues.apache.org/jira/browse/BEAM-9606?focusedWorklogId=409637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409637 ] ASF GitHub Bot logged work on BEAM-9606: Author: ASF GitHub Bot Created on: 25/Mar/20 16:49 Start Date: 25/Mar/20 16:49 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11219: [BEAM-9606] Add missing parameters in combine_test.py gradle example URL: https://github.com/apache/beam/pull/11219#issuecomment-603955446 Looks good, thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409637) Time Spent: 0.5h (was: 20m) > Example in gradle's combine test lacks of parameters > > > Key: BEAM-9606 > URL: https://issues.apache.org/jira/browse/BEAM-9606 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Affects Versions: Not applicable >Reporter: Piotr Szuberski >Assignee: Piotr Szuberski >Priority: Minor > Fix For: Not applicable > > Time Spent: 0.5h > Remaining Estimate: 0h > > The combine test example run with gradle lacks of fanout and top_count params -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7609) SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
[
https://issues.apache.org/jira/browse/BEAM-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066876#comment-17066876
]
Kenneth Knowles commented on BEAM-7609:
---
Still an issue? Working on this?
> SqlTransform#getSchema for "SELECT DISTINCT + JOIN" has invalid field names
> ---
>
> Key: BEAM-7609
> URL: https://issues.apache.org/jira/browse/BEAM-7609
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Assignee: Kirill Kozlov
>Priority: Major
>
> Works in sqlline shell:
> {code}
> Welcome to Beam SQL 2.14.0-SNAPSHOT (based on sqlline version 1.4.0)
> 0: BeamSQL> CREATE EXTERNAL TABLE s1 (id BIGINT) TYPE 'test';
> No rows affected (0.507 seconds)
> 0: BeamSQL> CREATE EXTERNAL TABLE s2 (id BIGINT) TYPE 'test';
> No rows affected (0.004 seconds)
> 0: BeamSQL> SELECT DISTINCT s1.id as lhs, s2.id as rhs FROM s1 JOIN s2 USING
> (id);
> +-+-+
> | lhs | rhs |
> +-+-+
> +-+-+
> No rows selected (2.568 seconds)
> {code}
> But doesn't work in the test:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("id", Schema.FieldType.INT32));
> PCollection i1 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> PCollection i2 = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = PCollectionTuple
> .of("i1", i1)
> .and("i2", i2)
> .apply(SqlTransform.query("SELECT DISTINCT s1.id as lhs, s2.id as rhs
> FROM i1 JOIN i2 USING (id)"))
> .getSchema();
> assertEquals(ImmutableList.of("lhs", "rhs"),
> outputSchema.getFieldNames());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-3188) [Calcite SQL] Query Parametrization
[
https://issues.apache.org/jira/browse/BEAM-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066877#comment-17066877
]
Kenneth Knowles commented on BEAM-3188:
---
This is now supported in ZetaSQL dialect but not Calcite dialect, correct?
> [Calcite SQL] Query Parametrization
> ---
>
> Key: BEAM-3188
> URL: https://issues.apache.org/jira/browse/BEAM-3188
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kyle Weaver
>Priority: Major
>
> Look into SQL query parametrization/templating.
> Calcite supports parameters:
> https://issues.apache.org/jira/browse/CALCITE-2054
> Beam does not:
> {code:java}
> Caused by: java.lang.UnsupportedOperationException: class
> org.apache.calcite.rex.RexDynamicParam is not supported yet!
> at
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:430)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9607) _SDFBoundedSourceWrapper should expose underlying source display_data
Yichi Zhang created BEAM-9607: - Summary: _SDFBoundedSourceWrapper should expose underlying source display_data Key: BEAM-9607 URL: https://issues.apache.org/jira/browse/BEAM-9607 Project: Beam Issue Type: Task Components: io-py-gcp, sdk-py-core Reporter: Yichi Zhang Assignee: Boyuan Zhang It seems that the _SDFBoundedSourceWrapper will hide the display data added to the source underneath. We should try to expose those data if it exists. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=409679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409679 ] ASF GitHub Bot logged work on BEAM-9468: Author: ASF GitHub Bot Created on: 25/Mar/20 18:04 Start Date: 25/Mar/20 18:04 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11151: [BEAM-9468] Hl7v2 io URL: https://github.com/apache/beam/pull/11151#issuecomment-603996758 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409679) Time Spent: 4.5h (was: 4h 20m) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 4.5h > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
