Varun created BEAM-14431:
----------------------------
Summary: Handle nulls using SnowflakeIO
Key: BEAM-14431
URL: https://issues.apache.org/jira/browse/BEAM-14431
Project: Beam
Issue Type: Bug
Components: io-java-snowflake
Affects Versions: 2.38.0
Reporter: Varun
When SnowflakeIO is used within python beam code using ReadFromSnowflake [1] ;
the returned data isn't able to list the nulls from snowflake side.
The nulls from snowflake becomes b'N' ; which is a wrong value and gives
incorrect value for the column. In case of strings the actual value can also be
'N'. So it becomes tough to handle it in csv_mapper.
To mitigate this we are using query instead of table and using ifnull /
coleasce within the select query for String columns. For non-string columns we
are converting back b'N' to null or empty as per requirement.
[1] :
[https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]|https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
