stegololz commented on code in PR #68022:
URL: https://github.com/apache/airflow/pull/68022#discussion_r3408056503
##########
providers/apache/hdfs/src/airflow/providers/apache/hdfs/assets/hdfs.py:
##########
@@ -26,10 +27,12 @@
from airflow.providers.common.compat.openlineage.facet import Dataset as
OpenLineageDataset
+# Preserve the empty-authority "hdfs:///path" (fs.defaultFS) form through
urlunsplit, like "file".
+if "hdfs" not in urllib.parse.uses_netloc:
+ urllib.parse.uses_netloc.append("hdfs")
Review Comment:
I can drop the line or implement a similar behaviour somewhere else, it is
not mandatory.
The SDK normalizes asset URIs through urlunsplit here:
https://github.com/apache/airflow/blob/e5bf1e30c1bf7b5ccf14f2f4cd611458f5bb6891/task-sdk/src/airflow/sdk/definitions/asset/__init__.py#L183-L185
That makes `hdfs:///apps/x` normalize to `hdfs:/apps/x`. I'd like to keep
the canonical `hdfs:///` form, which is why the if is there.
Caveat: as long as I stay on the provider, I won't be able to make a
distinction between `hdfs:/path` and `hdfs:///path` (both end up as
`hdfs:///path`), because the normalizer only sees the parsed result where both
already have an empty netloc. To keep them distinct I'd have to also modify the
Task SDK.
What would be the preference here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
