[
https://issues.apache.org/jira/browse/BEAM-8307?focusedWorklogId=429409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-429409
]
ASF GitHub Bot logged work on BEAM-8307:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/May/20 08:57
Start Date: 01/May/20 08:57
Worklog Time Spent: 10m
Work Description: rahul8383 commented on pull request #11581:
URL: https://github.com/apache/beam/pull/11581#issuecomment-622308296
I agree that this is not a scalable solution. Providing a Calcite
RelDataType Mapping for every Logical Type defined(which is the solution
presented in this PR) by every IO is not scalable.
Another approach to solving the problem is:
Provide Calcite RelDataType Mapping depending on the Base Type defined in
the logical type.
But, as CHAR and VARCHAR logical types have Base Type as STRING, we have to
choose a default Calcite RelDataType Mapping. The same thing applies to BINARY
and VARBINARY logical types.
We also have to choose a default RelDataType Mapping if the Base Type is
DATETIME.
By using this approach, I think we might be missing some built-in
Aggregation functions provided by Calcite for specific types.
I was also thinking if we can use the IDENTIFIER of the logical type to
determine the corresponding Calcite RelDataType. But, as the IDENTIFIER type is
String and not an enum, it cannot be used. For example, all the logical types
defined by JdbcIO use java.sql.JDBCType name as the IDENTIFIER.
Please correct me if my understanding is incorrect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 429409)
Time Spent: 1h (was: 50m)
> NPE in Calcite dialect when input PCollection has logical type in schema,
> from JdbcIO Transform
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-8307
> URL: https://issues.apache.org/jira/browse/BEAM-8307
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, io-java-jdbc
> Affects Versions: 2.15.0
> Reporter: Rahul Patwari
> Assignee: Rahul Patwari
> Priority: Major
> Attachments: exception_sqlTransform.txt
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> An exception is thrown when SqlTransform is used with JdbcIO.
> Schema of the Source Table:
> _CREATE TABLE actor (_
> _actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,_
> _first_name VARCHAR(45) NOT NULL,_
> _last_name VARCHAR(45) NOT NULL,_
> _last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE
> CURRENT_TIMESTAMP,_
> _PRIMARY KEY (actor_id),_
> _KEY idx_actor_last_name (last_name)_
> _)ENGINE=InnoDB DEFAULT CHARSET=utf8;_
>
> Many of the Jdbc Types are converted to Logical_Types in Beam.
> Refer:
> [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java#L102]
> The initial observation is that this code:
> [https://github.com/apache/beam/blob/113461a4275eb28a101e416897ccd607aa51f83f/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L207]
> is returning null.
>
> PFA [^exception_sqlTransform.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
