Re: Scala type checking thread-safety issue, and global locks to resolve it
Forgot to link the ticket that removed the global ScalaReflectionLock: https://issues.apache.org/jira/browse/SPARK-19810 Best Regards, Ryan On Fri, Mar 15, 2019 at 10:40 AM Shixiong(Ryan) Zhu wrote: > Hey Sean, > > Sounds good to me. At least, it's not worse than any versions prior to > 2.3.0 which has a global ScalaReflectionLock. In addition, if someone hits > a performance regression caused by this, they probably are creating too > many Encoders. Reusing Encoders is a better solution for this case. > > Best regards, > Shixiong > > On Thu, Mar 14, 2019 at 2:25 PM Sean Owen wrote: > >> This is worth a look: https://github.com/apache/spark/pull/24085 >> >> Scala has a surprising thread-safety bug in the "<:<" operator that's >> used to check subtypes, which can lead to incorrect results in >> non-trivial situations. >> >> The fix on the table is to introduce a global lock to protect a lot of >> the Scala-related reflection code to resolve it. This may be the best >> we can do, but 'global lock' sounds ominous. >> >> Any thoughts on other ways to resolve it? I'm not sure there are. >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: Scala type checking thread-safety issue, and global locks to resolve it
Hey Sean, Sounds good to me. At least, it's not worse than any versions prior to 2.3.0 which has a global ScalaReflectionLock. In addition, if someone hits a performance regression caused by this, they probably are creating too many Encoders. Reusing Encoders is a better solution for this case. Best regards, Shixiong On Thu, Mar 14, 2019 at 2:25 PM Sean Owen wrote: > This is worth a look: https://github.com/apache/spark/pull/24085 > > Scala has a surprising thread-safety bug in the "<:<" operator that's > used to check subtypes, which can lead to incorrect results in > non-trivial situations. > > The fix on the table is to introduce a global lock to protect a lot of > the Scala-related reflection code to resolve it. This may be the best > we can do, but 'global lock' sounds ominous. > > Any thoughts on other ways to resolve it? I'm not sure there are. > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Scala type checking thread-safety issue, and global locks to resolve it
This is worth a look: https://github.com/apache/spark/pull/24085 Scala has a surprising thread-safety bug in the "<:<" operator that's used to check subtypes, which can lead to incorrect results in non-trivial situations. The fix on the table is to introduce a global lock to protect a lot of the Scala-related reflection code to resolve it. This may be the best we can do, but 'global lock' sounds ominous. Any thoughts on other ways to resolve it? I'm not sure there are. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
