ninsbl commented on issue #55200: URL: https://github.com/apache/airflow/issues/55200#issuecomment-4862780267
> Do you read the asset extra in Airflow? If so, how? As background information: I use Apache Airflow for processing satellite data. I built a (pydantic) model for my assets, that I use to store a bunch of constants that not only contain information for human consumption, but also are used in standardized DAG logic. To access it I wrote an asset-module from which DAGs import a _get_asset_ function that is used to fetch _outlet_ and _inlet_ assets using their names. Constants like: _platform_, _spacecraft_, _processing_level_, _processing_baseline_, _product_type_, _sensor_, _sensor mode_, _bands_ are constants that become valuable for processing the data, in addition to URLs and other references in the user facing storage and provisioning systems. For example, we inject the satellite images into both a server for image provisioning and another platform for processing. Both systems use overlaping metadata for dataset, and since everything is orchestrated with Airflow anyways, we found it more natural to make Assets in Airflow the main source of all that metadata. Of course I can store all that information other places, like e.g. a python module associated with the DAG code. However, I figured the Assets to be a central element of both data processing and and human thinking that. And from the Asset.extra, Airflow can push the metadata to all our end-user facing systems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
