Experts need your help,
I'm using spark 1.4.1 and when set this hive.metastore.metadb.dir
programmatically for a hivecontext i.e for local metastore i.e the default
metastore_db for derby, the metastore_db is still getting creating in the
same path as user.dir.
Can you guys provide some insights re
I was about say whatever the previous post said,so +1 to the previous
post,from my understanding (gut feeling) of your requirement it very easy to
do this with spark-job-server.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-RDDs-in-memory-betwe
Yep,I saw that in your previous post and I thought it was a typing mistake
that you did while posting,I never imagined that it was done on R
studio.Glad it worked.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-sqlContext-or-sc-not-found-in-RStudio-t
I'm sorry, I have no idea why it is failing on your side.I have been using
this for a while now and it works fine.All I can say is use version 1.4.0
but I don't think so it is going to make a big difference.This is the one
which I use,a/b are my directories.
Sys.setenv(SPARK_HOME="/a/b/spark-1.4.0
I'm assuming you are building sparkR from github for apache/spark and not
github of amplab.
If that is correct,I don't see you intializing sqlcontext like this
sqlContext <- sparkRSQL.init(sc)..
If you have done both, then I don't have an idea as it is working fine for
me.
--
View this message i
A small correction when I typed it is not RDDBackend it is RBackend,sorry.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Share-RDD-from-SparkR-and-another-application-tp23795p23828.html
Sent from the Apache Spark User List mailing list archive at Nabble.co
I appreciate your reply.
Yes,you are right by putting in a parquet etc and reading from another app,I
would rather use spark-jobserver or IBM kernel to achieve the same if it is
not SparkR as it gives more flexibility/scalabilty.
Anyway,I have found a way to run R for my poc from my existing app us
Hello,
I would like to share RDD between an application and sparkR.
I understand we have job-server and IBM kernel for sharing the context for
different applications but not sure how we can use it with sparkR as it is
some sort of front end (R shell) with spark.
Any insights appreciated.
Hari
Guys,
I'm trying to join 2-3 schemaRDD's for approx 30,000 rows and it is terribly
slow.No doubt I get the results but it takes 8s to do the join and get the
results.
I'm running on a standalone spark in my m/c having 8 cores and 12gb RAM with
4 workers.
Not sure why it is consuming time,any input