Hi I want to make sure we use short-circuit local reads for performance. I have set the LD_LIBRARY_PATH correctly, confirmed that the native libraries match our platform (i.e. are 64 bit and are loaded successfully). When I start spark, i get the following message after increasing the logging level for the relevant classes.
*15/11/18 17:47:23 DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: Native Library /usr/hdp/2.3.2.0-2950/hadoop/lib/native/libhadoop.so.1.0.0 already loaded in another classloader* Any idea what might be causing it and how to resolve this. Regards Deenar [spark@edgenode1 spark-1.5.2-bin-hadoop2.6]$ bin/spark-shell 15/11/18 17:46:56 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library... 15/11/18 17:46:56 DEBUG NativeCodeLoader: Loaded the native-hadoop library Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65) Type in expressions to have them evaluated. Type :help for more information. 15/11/18 17:47:00 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 15/11/18 17:47:00 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 15/11/18 17:47:00 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. spark.yarn.driver.memoryOverhead is set but does not apply in client mode. Spark context available as sc. 15/11/18 17:47:23 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library... *15/11/18 17:47:23 DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: Native Library /usr/hdp/2.3.2.0-2950/hadoop/lib/native/libhadoop.so.1.0.0 already loaded in another classloader* 15/11/18 17:47:23 DEBUG NativeCodeLoader: java.library.path=/usr/hdp/2.3.2.0-2950/hadoop/lib/native/:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 15/11/18 17:47:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/11/18 17:47:24 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. SQL context available as sqlContext. *Think Reactive Ltd* deenar.toras...@thinkreactive.co.uk 07714140812