Sercan Karaoglu commented on SPARK-23891:

Debian and centos based linux distros are the most popular ones even if 
nowadays we see people tend to use alpine based jdk images because of their 
size, we still have most libraries like netty has their native bindings built 
in both for centos and debian. As far as I know spark also have netty 
dependency, if you want to get less gc pressure more performance from the TCP 
layer you can add netty linux native bindings to the classpath through the jars 
that you can find most repos like maven central, then Betty automatically binds 
to those .so's

my problem was specific to SSL communication with google big table, in this 
case google depends on netty tcnative library in their SDK. I had two ways to 
solve this problem, one is rebuilt the tcnative for alpine and exclude .so from 
classpath coming through existing jars and add my custom built .so to the 
classpath, the other way was to change base image which is way easier. I solved 
this problem by customizing that dockerfile you provide as a reference and 
would like to report the issue here

> Debian based Dockerfile
> -----------------------
>                 Key: SPARK-23891
>                 URL: https://issues.apache.org/jira/browse/SPARK-23891
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Sercan Karaoglu
>            Priority: Minor
> Current dockerfile inherits from alpine linux which causes netty tcnative ssl 
> bindings to fail while loading which is the case when we use Google Cloud 
> Platforms Bigtable Client on top of spark cluster. would be better to have 
> another debian based dockerfile

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to