I'm not sure what the best way to handle this is. Ideally we would use an alternative that doesn't require setting a property, but I don't know Netty well enough to make a recommendation. I also want to be careful not to introduce anything that would hurt performance or cause any other side effects. I made https://issues.apache.org/jira/browse/ARROW-7223 to track this, we can continue the discussion there and I will try to do some research into possible solutions.
On Wed, Nov 20, 2019 at 2:51 AM Fan Liya <liya.fa...@gmail.com> wrote: > Hi Bryan, > > Thanks for bringing this up. > +1 for the change. > > I am not clear what is the right place to override the jvm property. > It is possible that when we try to override it (possibly in a static > block), the old property value has already been read by netty library. > To avoid this problem, do we need to control the order of class loading? > > Best, > Liya Fan > > On Mon, Nov 18, 2019 at 3:17 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > This sounds reasonable to me. At this point I think having our consumers > > have a better experience is more important then library purity concerns > > I've had in the past. > > > > Do we need to handle jdk8 as a special case? Do you think it pays to try > > to find an alternate library that doesn't require special flags for > > whatever we are using this functionality for? > > > > Thanks, > > Micah > > > > On Sunday, November 17, 2019, Bryan Cutler <cutl...@gmail.com> wrote: > > > > > After ARROW-3191 [1], consumers of Arrow Java with a JDK 9 and above > are > > > required to set the JVM property "io.netty.tryReflectionSetAccessible= > > > true" > > > at startup, each time Arrow code is run, as documented at [2]. Not > doing > > > this will result in the error "java.lang.UnsupportedOperationException: > > > sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not > available". > > > This is due to a part of the Netty codebase, and I'm not sure if there > > any > > > way around it, but I don't think it's the correct behavior for Arrow > > > out-of-the-box to fail with a 3rd party error by default. This has come > > up > > > before in our own unit testing [3], and most recently when upgrading > > Arrow > > > in Spark [4]. > > > > > > I'd like to propose that Arrow Java change to the following behavior: > > > > > > 1) check to see if the property io.netty.tryReflectionSetAccessible has > > > been set > > > 2) if not set, automatically set to "true" > > > 3) else if set to "false", catch the Netty error and prepend the error > > > message with the suggested setting of "true" > > > > > > What do other devs think? > > > > > > [1] https://issues.apache.org/jira/browse/ARROW-3191 > > > [2] https://github.com/apache/arrow/tree/master/java#java-properties > > > [3] https://issues.apache.org/jira/browse/ARROW-5412 > > > [4] https://github.com/apache/spark/pull/26552 > > > > > >
