[
https://issues.apache.org/jira/browse/ARROW-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911465#comment-16911465
]
Jim Northrup commented on ARROW-6206:
-------------------------------------
> Could you provide a link to the text you quoted I'd be interested in reading
> it.
this is the benefit of having written what amounts to a netty analog over the
course of 4 years, including an SSL/TLS sockets layer for http at one point.
ultimately there is danger in long-lived services using NIO, end-of-story.
the process cleanup of the underlying OS will be the best protection against
java NIO/JNI memory handles -- if you have a daemon or long-running thing, or
you must use directbuffers, assume that the reference counting is imperfect,
and it will bite you one day (it may take days) if you trust it. so thing that
use nio should be short lived, and wherever possible process encapsulated.
netty is the jboss-endorsed c10k java representative with the popular
marketshare. iiuc arrow is a team that picked up netty derived off-heap tools
naively and demonstrated that in 2019 it's still prone to some gotchas that are
a little bit stronger than edge cases when the unit tests pass. indeed, my
initial testing with writing jdbc to arrow on kilobytes of records succeeded
well, and gave me the confidence to assume this will do the job faster than
python. and so began this thread on 800+ megabytes of data.
considering the age and size of the netty ecosystem, there is no lack of
scrutiny or open source virtue here. it's a VM-level weakness that java NIO is
still something like peanuts in the kitchen, you should really put a consumer
facing notice on where NIO is and is not present.
> [Java][Docs] Document environment variables/java properties
> -----------------------------------------------------------
>
> Key: ARROW-6206
> URL: https://issues.apache.org/jira/browse/ARROW-6206
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Documentation, Java
> Reporter: Micah Kornfield
> Assignee: Ji Liu
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.15.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Specifically, "-Dio.netty.tryReflectionSetAccessible=true" for JVMs >= 9 and
> BoundsChecking/NullChecking for get.
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
