cloud-fan commented on pull request #35734: URL: https://github.com/apache/spark/pull/35734#issuecomment-1060758497
I think the problem today is we don't have a good abstraction for this feature at the framework level. This is a special hive file format that changes the behavior of file listing, while in Spark the `FileIndex` API assumes the file listing behavior is unrelated to the file format. This PR simply adds special handling of `SymlinkTextInputFormat` in several places, and I'm OK with it if it's super hard to come up with a good abstraction for it, but we should give it a try first. cc @viirya @dongjoon-hyun @AngersZhuuuu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
