o-nikolas commented on pull request #17145: URL: https://github.com/apache/airflow/pull/17145#issuecomment-898595938
> * The user can distinguish keys from prefixes from the list returned by the operator. I would assume that keys have a file extension and prefixes would include the delimiter... but it seems possible that keys may not _always_ have a file extension and I'm basing my understanding of the prefixes containing the delimiter on the unit tests. This doesn't sound overly confident 😆 > > So I have this question: > > If we want to refactor the operator to return both prefixes and keys but a user might want to use different optional params between keys and prefixes, I don't see an alternative other than requiring the user to use the operator twice with different params. With this in mind, is there a valid argument to have a dedicated `S3ListPrefixes` operator after all? It does sound like you're finding significant friction here. Two paths forward I see are: 1. Change the type of the new parameter to something more advanced that will allow the user to specify exactly what they want out of the command (something like a string or an enum, that can be `keys`, `prefixes`, or `all`). This doesn't get around the issue of having two calls for different optional params though. But, at that point, the user really should have two separate calls if they are querying for a completely distinct set of results IMHO. 1. Or, just bail on a single operator and go back to having two as you suggest. What do others think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
