[
https://issues.apache.org/jira/browse/BEAM-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun updated BEAM-14431:
-------------------------
Attachment: csv_mapper_data.png
snowflake_data.png
> Handle nulls using SnowflakeIO
> ------------------------------
>
> Key: BEAM-14431
> URL: https://issues.apache.org/jira/browse/BEAM-14431
> Project: Beam
> Issue Type: Bug
> Components: io-java-snowflake
> Affects Versions: 2.38.0
> Reporter: Varun
> Priority: P2
> Labels: snowflake
> Attachments: csv_mapper_data.png, snowflake_data.png
>
>
> When SnowflakeIO is used within python beam code using ReadFromSnowflake [1]
> ; the returned data isn't able to list the nulls from snowflake side.
> The nulls from snowflake becomes b'N' ; which is a wrong value and gives
> incorrect value for the column. In case of strings the actual value can also
> be 'N'. So it becomes tough to handle it in csv_mapper.
> To mitigate this we are using query instead of table and using ifnull /
> coleasce within the select query for String columns. For non-string columns
> we are converting back b'N' to null or empty as per requirement.
>
> [1] :
> [https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]|https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
