peter-toth commented on PR #57040: URL: https://github.com/apache/spark/pull/57040#issuecomment-4913294892
I've reverted the `filter='data'` change back to the previous general version and added a comment. Agreed `filter='data'` is the nicest fix - it handles all of this natively. The only problem is that technically it might not be available in every Python version we support - it's missing in older patch releases like 3.11.3 and earlier. As @dongjoon-hyun said, these are unlikely to still be in use, but it might happen. So how about: land this PR (the plain `realpath` check, works everywhere) on the active branches, and I open a follow-up under a new JIRA to switch to `filter='data'` on master/5.0.0 only, once we drop the Pythons that lack it? How does that sound, @dongjoon-hyun @gaogaotiantian? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
