eric-czech commented on issue #1370: URL: https://github.com/apache/hamilton/issues/1370#issuecomment-3331115324
Thanks @zilto, great explanation! That diagram does help a lot. All use cases I have in mind would involve passing nothing other than URIs to paths on cloud storage systems between Hamilton steps, and they would all be known prior to DAG execution. I.e. my outputs would all be individual strings that rarely change or change in predictable ways (e.g. based on dates or versions). That would be far more amenable to a dry-run, but I understand what you're saying that hamilton is supporting content-based, runtime dependencies. It would be great if something like `visualize_execution` could run prior to execution for DAGs that make stronger assumptions somehow. I am essentially asking for `--dry-run` as it works in `make` since that's more or less how I would like to use hamilton. Or snakemake might be a better comparison given that it's a bit more cloud native. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
