cBrayton commented on PR #27548:
URL: https://github.com/apache/airflow/pull/27548#issuecomment-1307397119
The code that calls the snowflake version is below:
`def validate_temp_data():
snowflake_sql_command = f"select count(*) from {REMOTE_TEMP_LOG_TABLE}"
remote_row_count = snowflake_hook.run(sql=snowflake_sql_command)
postgres_sql_command = f"select count(*) from {LOCAL_TEMP_LOG_TABLE}"
local_row_count = postgres_hook.get_records(sql=postgres_sql_command)
logging.info(f"Remote_row_count: {remote_row_count}")
logging.info(f"Local_row_count: {local_row_count}")
if remote_row_count[0]["COUNT(*)"] == local_row_count[0][0]:
logging.info("Same number of records/rows are in the local and
remote tables.")
else:
raise ValueError(
"Number of rows are not the same in the local and remote tables."
)`
The relevant log file output of this is:
`[2022-10-31, 16:53:41 UTC] {connection.py:257} INFO - Snowflake Connector
for Python Version: 2.7.9, Python Version: 3.9.12, Platform: Linux
[2022-10-31, 16:53:44 UTC] {logs_sync.py:120} INFO - Remote_row_count: None
[2022-10-31, 16:53:44 UTC] {logs_sync.py:121} INFO - Local_row_count: [(0,)]
[2022-10-31, 16:53:44 UTC] {taskinstance.py:1774} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py",
line 174, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py",
line 188, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/logs_sync.py", line 122, in
validate_temp_data
if remote_row_count[0]["COUNT(*)"] == local_row_count[0][0]:
TypeError: 'NoneType' object is not subscriptable`
The expect log output is this (run with an older version of the Snowflake
Connector for Python):
`[2022-10-31, 20:46:07 UTC] {connection.py:257} INFO - Snowflake Connector
for Python Version: 2.7.8, Python Version: 3.9.12, Platform: Linux
[2022-10-31, 20:46:08 UTC] {logs_sync.py:120} INFO - Remote_row_count:
[{'COUNT(*)': 0}]
[2022-10-31, 20:46:08 UTC] {logs_sync.py:121} INFO - Local_row_count: [(0,)]
[2022-10-31, 20:46:08 UTC] {logs_sync.py:123} INFO - Same number of
records/rows are in the local and remote tables.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
