Re: thrift-server does not load jars files (Azure HDInsight)

2015-07-08 Thread Daniel Haviv 
Hi,
Just updating back that  setting spark.driver.extraClassPath worked.

Thanks,
Daniel

On Fri, Jul 3, 2015 at 5:35 PM, Ted Yu  wrote:

> Alternatively, setting spark.driver.extraClassPath should work.
>
> Cheers
>
> On Fri, Jul 3, 2015 at 2:59 AM, Steve Loughran 
> wrote:
>
>>
>>> On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
>>> daniel.ha...@veracity-group.com> wrote:
>>>
 Hi,
 I'm trying to start the thrift-server and passing it azure's blob
 storage jars but I'm failing on :
  Caused by: java.io.IOException: No FileSystem for scheme: wasb
 at
 org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
 at
 org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
 at
 org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
 at
 org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
 at
 org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
 at
 org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
 ... 16 more

  If I start the spark-shell the same way, everything works fine.

  spark-shell command:
   ./bin/spark-shell --master yarn --jars
 /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
 --num-executors 4

  thrift-server command:
  ./sbin/start-thriftserver.sh --master yarn--jars
 /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
 --num-executors 4

  How can I pass dependency jars to the thrift server?

  Thanks,
 Daniel

>>>
>>>
>>
>>  you should be able to add the JARs to the environment variable
>> SPARK_SUBMIT_CLASSPATH or SPARK_CLASSPATH and have them picked up when
>> bin/compute-classpath.{cmd.sh} builds up the classpath
>>
>>
>>
>

Re: thrift-server does not load jars files (Azure HDInsight)

2015-07-03 Thread Ted Yu 
Alternatively, setting spark.driver.extraClassPath should work.

Cheers

On Fri, Jul 3, 2015 at 2:59 AM, Steve Loughran 
wrote:

>
>> On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
>> daniel.ha...@veracity-group.com> wrote:
>>
>>> Hi,
>>> I'm trying to start the thrift-server and passing it azure's blob
>>> storage jars but I'm failing on :
>>>  Caused by: java.io.IOException: No FileSystem for scheme: wasb
>>> at
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>> at
>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
>>> ... 16 more
>>>
>>>  If I start the spark-shell the same way, everything works fine.
>>>
>>>  spark-shell command:
>>>   ./bin/spark-shell --master yarn --jars
>>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>>> --num-executors 4
>>>
>>>  thrift-server command:
>>>  ./sbin/start-thriftserver.sh --master yarn--jars
>>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>>> --num-executors 4
>>>
>>>  How can I pass dependency jars to the thrift server?
>>>
>>>  Thanks,
>>> Daniel
>>>
>>
>>
>
>  you should be able to add the JARs to the environment variable
> SPARK_SUBMIT_CLASSPATH or SPARK_CLASSPATH and have them picked up when
> bin/compute-classpath.{cmd.sh} builds up the classpath
>
>
>

Re: thrift-server does not load jars files (Azure HDInsight)

2015-07-03 Thread Steve Loughran 

On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv 
mailto:daniel.ha...@veracity-group.com>> wrote:
Hi,
I'm trying to start the thrift-server and passing it azure's blob storage jars 
but I'm failing on :
Caused by: java.io.IOException: No FileSystem for scheme: wasb
at 
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
... 16 more

If I start the spark-shell the same way, everything works fine.

spark-shell command:
 ./bin/spark-shell --master yarn --jars 
/home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
 --num-executors 4

thrift-server command:
 ./sbin/start-thriftserver.sh --master yarn--jars 
/home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
 --num-executors 4

How can I pass dependency jars to the thrift server?

Thanks,
Daniel



you should be able to add the JARs to the environment variable 
SPARK_SUBMIT_CLASSPATH or SPARK_CLASSPATH and have them picked up when 
bin/compute-classpath.{cmd.sh} builds up the classpath

Re: thrift-server does not load jars files (Azure HDInsight)

2015-07-02 Thread Daniel Haviv 
Hi,
I'm using 1.4.
It's indeed a typo in the email itself.

Thanks,
Daniel

On Thu, Jul 2, 2015 at 6:06 PM, Ted Yu  wrote:

> Which Spark release are you using ?
>
> bq. yarn--jars
>
> I guess the above was just a typo in your email (missing space).
>
> Cheers
>
> On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
> daniel.ha...@veracity-group.com> wrote:
>
>> Hi,
>> I'm trying to start the thrift-server and passing it azure's blob storage
>> jars but I'm failing on :
>> Caused by: java.io.IOException: No FileSystem for scheme: wasb
>> at
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>> at
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
>> ... 16 more
>>
>> If I start the spark-shell the same way, everything works fine.
>>
>> spark-shell command:
>>  ./bin/spark-shell --master yarn --jars
>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>> --num-executors 4
>>
>> thrift-server command:
>>  ./sbin/start-thriftserver.sh --master yarn--jars
>> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
>> --num-executors 4
>>
>> How can I pass dependency jars to the thrift server?
>>
>> Thanks,
>> Daniel
>>
>
>

Re: thrift-server does not load jars files (Azure HDInsight)

2015-07-02 Thread Ted Yu 
Which Spark release are you using ?

bq. yarn--jars

I guess the above was just a typo in your email (missing space).

Cheers

On Thu, Jul 2, 2015 at 7:38 AM, Daniel Haviv <
daniel.ha...@veracity-group.com> wrote:

> Hi,
> I'm trying to start the thrift-server and passing it azure's blob storage
> jars but I'm failing on :
> Caused by: java.io.IOException: No FileSystem for scheme: wasb
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:342)
> ... 16 more
>
> If I start the spark-shell the same way, everything works fine.
>
> spark-shell command:
>  ./bin/spark-shell --master yarn --jars
> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
> --num-executors 4
>
> thrift-server command:
>  ./sbin/start-thriftserver.sh --master yarn--jars
> /home/hdiuser/azureclass/azure-storage-1.2.0.jar,/home/hdiuser/azureclass/hadoop-azure-2.7.0.jar
> --num-executors 4
>
> How can I pass dependency jars to the thrift server?
>
> Thanks,
> Daniel
>