Re: Hive permanent functions are not available in Spark SQL

+user list On Tue, Sep 29, 2015 at 3:43 PM, Pala M Muthaia wrote: > Hi, > > I am trying to use internal UDFs that we have added as permanent functions > to Hive, from within Spark SQL query (using HiveContext), but i encounter > NoSuchObjectException, i.e. the

Re: using JavaRDD in spark-redis connector

You can create a JavaRDD as normal and then call the .rdd() to get the RDD. Thanks Best Regards On Mon, Sep 28, 2015 at 9:01 PM, Rohith P wrote: > Hi all, > I am trying to work with spark-redis connector (redislabs) which > requires all transactions between

CQs on WindowedStream created on running StreamingContext

Hi, We intend to run adhoc windowed continuous queries on spark streaming data. The queries could be registered/deregistered dynamically or can be submitted through command line. Currently Spark streaming doesn’t allow adding any new inputs, transformations, and output operations after starting a

Task Execution

Concerning task execution, a worker executes its assigned tasks in parallel or sequentially? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Task-Execution-tp14411.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

GraphX PageRank keeps 3 copies of graph in memory

Dear Spark developers, I would like to understand GraphX caching behavior with regards to PageRank in Spark, in particular, the following implementation of PageRank: https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala On each iteration

Re: Speculatively using spare capacity

Why change the number of partitions of RDDs? especially since you can't generally do that without a shuffle. If you just mean to ramp up and down resource usage, dynamic allocation (of executors) already does that. On Wed, Sep 30, 2015 at 10:49 PM, Muhammed Uluyol wrote: >

Speculatively using spare capacity

Hello, How feasible would it be to have spark speculatively increase the number of partitions when there is spare capacity in the system? We want to do this to increase to decrease application runtime. Initially, we will assume that function calls of the same type will have the same runtime (e.g.

Re: failed to run spark sample on windows

thanks a lot, it works now after I set %HADOOP_HOME% On Tue, Sep 29, 2015 at 1:22 PM, saurfang wrote: > See > > http://stackoverflow.com/questions/26516865/is-it-possible-to-run-hadoop-jobs-like-the-wordcount-sample-in-the-local-mode > , >

Re: unsubscribe

Hi Sukesh, To unsubscribe from the dev list, please send a message to dev-unsubscr...@spark.apache.org. To unsubscribe from the user list, please send a message user-unsubscr...@spark.apache.org. Please see: http://spark.apache.org/community.html#mailing-lists. Thanks, -Rick sukesh kumar