Re: error while submitting job

2018-09-29 Thread Ted Yu 
Can you tell us the version of Spark and the connector you used ?
Thanks 
 Original message From: yuvraj singh 
<19yuvrajsing...@gmail.com> Date: 9/29/18  10:42 PM  (GMT-08:00) To: 
user@spark.apache.org Subject: error while submitting job 


Hi , i am getting this error please help me .

18/09/30 05:14:44 INFO Client:   client token: N/A   diagnostics: User 
class threw exception: java.lang.NoClassDefFoundError: 
org/apache/commons/configuration/ConfigurationException   at 
org.apache.spark.sql.cassandra.DefaultSource$.(DefaultSource.scala:135)   
 at 
org.apache.spark.sql.cassandra.DefaultSource$.(DefaultSource.scala)  at 
org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:55)
  at 
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
  at 
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239) at 
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at 
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164) at 
com.ola.ss.lhf.LoginHourFetcher.main(LoginHourFetcher.java:39)   at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)Caused
 by: java.lang.ClassNotFoundException: 
org.apache.commons.configuration.ConfigurationException  at 
java.net.URLClassLoader.findClass(URLClassLoader.java:381)   at 
java.lang.ClassLoader.loadClass(ClassLoader.java:424)at 
java.lang.ClassLoader.loadClass(ClassLoader.java:357)... 13 more
 ApplicationMaster host: 10.14.58.163ApplicationMaster RPC port: 0  
 queue: signals  start time: 1538284434296   final status: FAILED
tracking URL: 
http://as-data3.prod-ambari16.olacabs.net:8088/proxy/application_1537943933870_0088/
  user: yubrajsinghException in thread "main" 
org.apache.spark.SparkException: Application application_1537943933870_0088 
finished with failed statusat 
org.apache.spark.deploy.yarn.Client.run(Client.scala:1269)   at 
org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1627) at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) 
 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)   at 
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at 
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


Thanks Yubraj Singh 

error while submitting job

2018-09-29 Thread yuvraj singh 

> 
> Hi , i am getting this error please help me .
> 
> 
> 18/09/30 05:14:44 INFO Client: 
>client token: N/A
>diagnostics: User class threw exception: 
> java.lang.NoClassDefFoundError: 
> org/apache/commons/configuration/ConfigurationException
>   at 
> org.apache.spark.sql.cassandra.DefaultSource$.(DefaultSource.scala:135)
>   at 
> org.apache.spark.sql.cassandra.DefaultSource$.(DefaultSource.scala)
>   at 
> org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:55)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
>   at 
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
>   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
>   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
>   at com.ola.ss.lhf.LoginHourFetcher.main(LoginHourFetcher.java:39)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.commons.configuration.ConfigurationException
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 13 more
> 
>ApplicationMaster host: 10.14.58.163
>ApplicationMaster RPC port: 0
>queue: signals
>start time: 1538284434296
>final status: FAILED
>tracking URL: 
> http://as-data3.prod-ambari16.olacabs.net:8088/proxy/application_1537943933870_0088/
>  
> 
>user: yubrajsingh
> Exception in thread "main" org.apache.spark.SparkException: Application 
> application_1537943933870_0088 finished with failed status
>   at org.apache.spark.deploy.yarn.Client.run(Client.scala:1269)
>   at 
> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1627)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


Thanks 
Yubraj Singh

Re: Time-Series Forecasting

2018-09-29 Thread Peyman Mohajerian 
Here's a blog on Flint:
https://databricks.com/blog/2018/09/11/introducing-flint-a-time-series-library-for-apache-spark.html
I don't have an opinion about it, just that Flint was mentioned earlier.

On Thu, Sep 20, 2018 at 2:12 AM, Gourav Sengupta 
wrote:

> Hi,
>
> If you are following the time series forecasting with the mathematical
> rigour and tractability then I think that using R is the best option. I do
> think that people tend to claim quite a lot these days that SPARK ML and
> other Python libraries are better, but just pick up a classical text book
> on time series forecasting and start asking fundamental mathematical
> questions and compare for yourself.
>
>
> Regards,
> Gourav Sengupta
>
> On Wed, Sep 19, 2018 at 5:02 PM Mina Aslani  wrote:
>
>> Hi,
>> I have a question for you. Do we have any Time-Series Forecasting library
>> in Spark?
>>
>> Best regards,
>> Mina
>>
>

Re: can Spark 2.4 work on JDK 11?

2018-09-29 Thread Felix Cheung 
Not officially. We have seen problem with JDK 10 as well. It will be great if 
you or someone would like to contribute to get it to work..



From: kant kodali 
Sent: Tuesday, September 25, 2018 2:31 PM
To: user @spark
Subject: can Spark 2.4 work on JDK 11?

Hi All,

can Spark 2.4 work on JDK 11? I feel like there are lot of features that are 
added in JDK 9, 10, 11 that can make deployment process a whole lot better and 
of course some more syntax sugar similar to Scala.

Thanks!