damccorm opened a new issue, #21659: URL: https://github.com/apache/beam/issues/21659
When SnowflakeIO is used within python beam code using ReadFromSnowflake [1] ; the returned data isn't able to list the nulls from snowflake side. The nulls from snowflake becomes b'N' ; which is a wrong value and gives incorrect value for the column. In case of strings the actual value can also be 'N'. So it becomes tough to handle it in csv_mapper. To mitigate this we are using query instead of table and using ifnull / coleasce within the select query for String columns. For non-string columns we are converting back b'N' to null or empty as per requirement. [1] : [https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake](https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake) Imported from Jira [BEAM-14431](https://issues.apache.org/jira/browse/BEAM-14431). Original Jira may contain additional context. Reported by: Varun Rauthan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
