[jira] [Comment Edited] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436942#comment-16436942 ] Sercan Karaoglu edited comment on SPARK-23891 at 4/13/18 7:24 AM: -- Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find from most repos like maven central, then netty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here was (Author: sercankaraoglu): Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then netty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436942#comment-16436942 ] Sercan Karaoglu edited comment on SPARK-23891 at 4/13/18 7:23 AM: -- Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then netty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here was (Author: sercankaraoglu): Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then Betty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
