[
https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436942#comment-16436942
]
Sercan Karaoglu commented on SPARK-23891:
-----------------------------------------
Debian and centos based linux distros are the most popular ones even if
nowadays we see people tend to use alpine based jdk images because of their
size, we still have most libraries like netty has their native bindings built
in both for centos and debian. As far as I know spark also have netty
dependency, if you want to get less gc pressure more performance from the TCP
layer you can add netty linux native bindings to the classpath through the jars
that you can find most repos like maven central, then Betty automatically binds
to those .so's
my problem was specific to SSL communication with google big table, in this
case google depends on netty tcnative library in their SDK. I had two ways to
solve this problem, one is rebuilt the tcnative for alpine and exclude .so from
classpath coming through existing jars and add my custom built .so to the
classpath, the other way was to change base image which is way easier. I solved
this problem by customizing that dockerfile you provide as a reference and
would like to report the issue here
> Debian based Dockerfile
> -----------------------
>
> Key: SPARK-23891
> URL: https://issues.apache.org/jira/browse/SPARK-23891
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 2.3.0
> Reporter: Sercan Karaoglu
> Priority: Minor
>
> Current dockerfile inherits from alpine linux which causes netty tcnative ssl
> bindings to fail while loading which is the case when we use Google Cloud
> Platforms Bigtable Client on top of spark cluster. would be better to have
> another debian based dockerfile
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
