Hello, I attended the Tungsten-related presentations at Spark Summit (by Josh Rosen) and at Big Data Scala (by Matei Zaharia). Needless to say, this project holds great promise for major performance improvements.
At Josh's talk, I heard about the use of sun.misc.Unsafe as a way of achieving some of these optimizations (e.g. slides 11-17 of Josh's presentation: http://www.slideshare.net/SparkSummit/deep-dive-into-project-tungsten-josh-rosen). I have no problems with the use of Unsafe in the code itself (I've done it before myself, too), however I think there is a considerable risk associated with beginning the use of Unsafe now, because Oracle is determined to limit access to APIs such as Unsafe starting in Java 9. JEP 260 <http://openjdk.java.net/jeps/260> was filed specifically to limit access to internal JDK APIs that were "never intended for external use, including "sun.misc.*" The JEP does say that the functionality of sun.misc.Unsafe is to remain available even as other internal APIs are blocked for non-JDK use, however, it also says that "the functionality of many methods of this class is now available via *variable handles (JEP 193 <http://openjdk.java.net/jeps/193>).*" If the direct access to sun.misc.Unsafe is blocked and only the variable handles access remains, this may mean more than just a need for code refactoring - functionality such as doing "malloc" from Spark core may be restricted. JEP 260 has evolved quite a bit over time and the wording available now (after the Aug. 4, 2015) seems more reasonable than before. Nevertheless, Hazelcast and other companies whose technologies depend on the availability of Unsafe started a Google doc here <https://docs.google.com/document/d/1GDm_cAxYInmoHMor-AkStzWvwE9pw6tnz_CebJQxuUE/edit#heading=h.brct71tr6e13> . I doubt that Oracle would want to make life difficult for everyone. In addition to Spark's code base, projects such as Akka, Cassandra, Hibernate, Netty, Neo4j and Spring (among many others) depend on Unsafe. Still, there are tons of posts about this issue in the Java community (e.g. here <https://jaxenter.com/hazelcast-on-java-unsafe-class-119286.html>'s a Hazelcast interview, also from Aug. 3, the day before the latest update to JEP 260). There are tons of concerned posts on the blogosphere, too (e.g. here <http://blog.dripstat.com/removal-of-sun-misc-unsafe-a-disaster-in-the-making/> ). Have the leaders of the Spark community been following these Unsafe-related developments and if so, what's Spark's plan of handling whatever Oracle throws our way? Marek
