If you link against the pre-built binary, that's for Hadoop 1.0.4. Can you show your deps to clarify what you are depending on? Building custom Spark and depending on it is a different thing from depending on plain Spark and changing its deps. I think you want the latter.
On Fri, Jul 25, 2014 at 5:46 PM, Bharath Ravi Kumar <reachb...@gmail.com> wrote: > Thanks for responding. I used the pre built spark binaries meant for > hadoop1,cdh3u5. I do not intend to build spark against a specific > distribution. Irrespective of whether I build my app with the explicit cdh > hadoop client dependency, I get the same error message. I also verified > that my app's uber jar had pulled in the cdh hadoop client dependencies. > > On 25-Jul-2014 9:26 pm, "Sean Owen" <so...@cloudera.com> wrote: >> >> This indicates your app is not actually using the version of the HDFS >> client you think. You built Spark from source with the right deps it >> seems, but are you sure you linked to your build in your app? >> >> On Fri, Jul 25, 2014 at 4:32 PM, Bharath Ravi Kumar <reachb...@gmail.com> >> wrote: >> > Any suggestions to work around this issue ? The pre built spark >> > binaries >> > don't appear to work against cdh as documented, unless there's a build >> > issue, which seems unlikely. >> > >> > On 25-Jul-2014 3:42 pm, "Bharath Ravi Kumar" <reachb...@gmail.com> >> > wrote: >> >> >> >> >> >> I'm encountering a hadoop client protocol mismatch trying to read from >> >> HDFS (cdh3u5) using the pre-build spark from the downloads page (linked >> >> under "For Hadoop 1 (HDP1, CDH3)"). I've also followed the >> >> instructions at >> >> >> >> http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html >> >> (i.e. building the app against hadoop-client 0.20.2-cdh3u5), but >> >> continue to >> >> see the following error regardless of whether I link the app with the >> >> cdh >> >> client: >> >> >> >> 14/07/25 09:53:43 INFO client.AppClient$ClientActor: Executor updated: >> >> app-20140725095343-0016/1 is now RUNNING >> >> 14/07/25 09:53:43 WARN util.NativeCodeLoader: Unable to load >> >> native-hadoop >> >> library for your platform... using builtin-java classes where >> >> applicable >> >> 14/07/25 09:53:43 WARN snappy.LoadSnappy: Snappy native library not >> >> loaded >> >> Exception in thread "main" org.apache.hadoop.ipc.RPC$VersionMismatch: >> >> Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version >> >> mismatch. >> >> (client = 61, server = 63) >> >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:401) >> >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) >> >> >> >> >> >> While I can build spark against the exact hadoop distro version, I'd >> >> rather work with the standard prebuilt binaries, making additional >> >> changes >> >> while building the app if necessary. Any workarounds/recommendations? >> >> >> >> Thanks, >> >> Bharath