rednaxelafx commented on PR #44977: URL: https://github.com/apache/spark/pull/44977#issuecomment-1920739267
I'm neutral on lifting the `isClosureCandidate` restriction (as I already mentioned in the comment, it was expected that someone would life it in the future ;-) What I had in mind at the time was like "what if we need to also clean Java lambdas?", which only matters if we add JShell support into Spark. All I'd ask is to implement it right instead of doing it half-baked. My original intent with the `isClosureCandidate` check was to use a "quick" check to reduce the number of candidates flowing into the later parts. It had nothing to do with Scala 2.11 compatibility/support (so please correct that from your PR description). I had only been developing and testing with Scala 2.12's simple lambda cases, and I know that the ones I were testing with all implemented the `scala.FunctionN` traits, so just being conservative and limiting the support to the cases I've tested with, I added the `isClosureCandidate` and a few other restrictions (like the `$anonfunc$` check that immediately followed, which again isn't a perfect filter, but one that likely filters out regular named functions instead of lambdas). The special thing about language-level lambdas is that you can have higher confidence in that the class is generated by the compiler (or at least the recipe of the class comes from the compiler), so they'll follow certain patterns and won't have surprises with fields that are hand-added. It certainly is possible to abuse the imperfect checks/filters I had put in place, to trick the cleaner to trust a non-compiler-generated class as being a Scala lambda... there's quite some room for improvement here. @vsevolodstep-db Seva had been working on ClosureCleaner improvements like this one (https://github.com/apache/spark/pull/42995) which found quite some cases that I didn't handle right... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
