Thanks for the response.
I will try with log4j.
That said, I am running in windows using winutil.exe and still getting the
warning.
Thanks,
Assaf.
From: Steve Loughran [mailto:ste...@hortonworks.com]
Sent: Tuesday, May 16, 2017 6:55 PM
To: Mendelson, Assaf
Cc: user@spark.apache.org
Subject: Re: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
On 10 May 2017, at 13:40, Mendelson, Assaf
<assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote:
Hi all,
When running spark I get the following warning: [WARN]
org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Now I know that in general it is possible to ignore this warning, however, it
means that utilities that catch “WARN” in the log keep flagging this.
I saw many answers to handling this (e.g.
http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker,
http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning,http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform),
however, I am unable to solve this on my local machine.
Specifically, I can’t find any such solution for windows (i.e. when running
developer local builds) or on a centos 7 machine with no HDFS (basically it is
a single node machine which uses spark standalone for testing).
Log4J is your friend. I usually have (at least)
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN
if you are working on Windows though, you do actually need the native libraries
an winutils.exe on your path, or things won't work
Any help would be appreciated.
Thanks,
Assaf.
