dstandish commented on PR #55070: URL: https://github.com/apache/airflow/pull/55070#issuecomment-3237191496
mentioned it on slack but just adding here should check with @eladkal re backcompat issues. changing `get_file_metadata` to iterator likely means major release. and given that maybe we should update the whole check key funciton logic so that it _never_ has to hold everything in memory, i.e. by making it so that check_fn takes a key and not a list but thinking about that..... the problem is it's somewhat ambiguous what the behavior should be for best performance, you would want check_fn to return on first "pass" but, the behavior that would be most similar to current, would be to return all the files for which check_fn evaluates to true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
