bolkedebruin commented on code in PR #36205:
URL: https://github.com/apache/airflow/pull/36205#discussion_r1433958522
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -241,13 +267,23 @@ def run(
else:
return results
- @staticmethod
- def _make_serializable(result):
- """Transform the databricks Row objects into JSON-serializable
lists."""
- if isinstance(result, list):
- return [list(row) for row in result]
- elif isinstance(result, Row):
- return list(result)
+ def _make_common_data_structure(
+ self, result: list[T] | T | None
+ ) -> list[T] | T | list[tuple] | tuple | None:
+ """Transform the databricks Row objects into namedtuple."""
+ # Below ignored lines respect namedtuple docstring, but mypy do not
support dynamically
+ # instantiated namedtuple, and will never do:
https://github.com/python/mypy/issues/848
+ if self.return_tuple:
+ if isinstance(result, list) and all(isinstance(item, Row) for item
in result):
Review Comment:
Potentially you are checking thousands of rows here and `isinstance` is
quite slow as it needs to walk the inheritance tree. Is there a way to check a
attribute(s) to verify you are dealing with a Row?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
