On 19 Jan 2017, at 10:59, Sean Owen <so...@cloudera.com<mailto:so...@cloudera.com>> wrote:
It's a message from Hadoop libs, not Spark. It can be safely ignored. It's just saying you haven't installed the additional (non-Apache-licensed) native libs that can accelerate some operations. This is something you can easily have read more about online. I'd worry about in production as it means that higher performance code for compression and HDFS encryption at rest (if used) aren' there, and it will fall back to the java libraries. Which aren't as bad as you'd think, at least for decompression, but they do tend underperform for dev/test/local work, you can turn off the warning entirely log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR On Thu, Jan 19, 2017 at 10:57 AM Md. Rezaul Karim <rezaul.ka...@insight-centre.org<mailto:rezaul.ka...@insight-centre.org>> wrote: Hi All, I'm the getting the following WARNING while running Spark jobs in standalone mode: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Please note that I have configured the native path and the other ENV variables as follows: export JAVA_HOME=/usr/lib/jvm/java-8-oracle export HADOOP_HOME=/usr/local/hadoop export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/hadoop/lib/native export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native/:$LD_LIBRARY_PATH export JAVA_LIBRARY_PATH=/usr/local/hadoop/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native" Although my Spark job executes successfully and writes the results to a file at the end. However, I am not getting any logs to track the progress. Could someone help me to solve this problem? Regards, _________________________________ Md. Rezaul Karim, BSc, MSc PhD Researcher, INSIGHT Centre for Data Analytics National University of Ireland, Galway IDA Business Park, Dangan, Galway, Ireland Web: http://www.reza-analytics.eu/index.html<http://139.59.184.114/index.html>
