?????? can not use udf in hivethriftserver2

hi,lalit sharma. I add jar to SPARK_CLASSPATH env variable. But spark thriftserver can not start. errors: ### 16/05/31 02:29:12 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Found both spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former.

Fwd: User finding issue in Spark Thrift server

Warm Regards, -Radhika -- Forwarded message -- From: Radhika Kothari Date: Tue, May 31, 2016 at 11:44 AM Subject: Fwd: User finding issue in Spark Thrift server To: user@spark.apache.org Warm Regards, -Radhika -- Forwarded message -- From: Radhika Kothari Date

Fwd: User finding issue in Spark Thrift server

Warm Regards, -Radhika -- Forwarded message -- From: Radhika Kothari Date: Tue, May 31, 2016 at 11:42 AM Subject: Fwd: User finding issue in Spark Thrift server To: user@spark.apache.org Warm Regards, -Radhika -- Forwarded message -- From: Radhika Kothari Date

Fwd: User finding issue in Spark Thrift server

Warm Regards, -Radhika -- Forwarded message -- From: Radhika Kothari Date: Tue, May 31, 2016 at 11:41 AM Subject: Fwd: User finding issue in Spark Thrift server To: user-i...@spark.apache.org Warm Regards, -Radhika -- Forwarded message -- From: Radhika Kothari

RE: GraphX Java API

Hi , Scala has similar package structure as java and finally it runs on JVM so probably you get an impression that its in Java. As far as I know there are no Java API for GraphX. I had used GraphX last year and at that time I had to code in Scala to use the GraphX APIs. Regards, Santosh Akhilesh

Re: Spark SQL Errors

Unfortunately, I do not have it, as it is 3rd party code :( But essentially I am trying to overwrite data to a hive table from a source On Tue, May 31, 2016 at 4:01 PM, Mich Talebzadeh wrote: > ok what is the exact spark code that is causing the issue. > > can you show it in its entirety? > > H

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

I'm not sure what do you mean by saying "does not return any value". How do you use this method? I will use this method as following : def extract_feature(rf_model, x): text = getFeatures(x).split(',') fea = [float(i) for i in text] prediction = rf_model.predict(fea) return (prediction, x) def pro

Re: Spark SQL Errors

ok what is the exact spark code that is causing the issue. can you show it in its entirety? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

Sorry for lots of typos (writing from mobile) On Tuesday, 31 May 2016, obaidul karim wrote: > foreachRDD does not return any value. I can be used just to send result to > another place/context, like db,file etc. > I could use that but seems like over head of having another hop. > I wanted to mak

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

foreachRDD does not return any value. I can be used just to send result to another place/context, like db,file etc. I could use that but seems like over head of having another hop. I wanted to make it simple and light. On Tuesday, 31 May 2016, nguyen duc tuan wrote: > How about using foreachRDD

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

How about using foreachRDD ? I think this is much better than your trick. 2016-05-31 12:32 GMT+07:00 obaidul karim : > Hi Guys, > > In the end, I am using below. > The trick is using "native python map" along with "spark spreaming > transform". > May not an elegent way, however it works :). > >

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

Hi Guys, In the end, I am using below. The trick is using "native python map" along with "spark spreaming transform". May not an elegent way, however it works :). def predictScore(texts, modelRF): predictions = texts.map( lambda txt : (txt , getFeatures(txt)) ).\ map(lambda (txt, featur

Re: Spark SQL Errors

No there is no semicolon. This is the query: 16/05/31 14:34:29 INFO SparkExecuteStatementOperation: Running query 'DESCRIBE EXTENDED `sds.unhealthy_om_delta`' with e24282a8-43d1-4c3a-a3f3-2645761ed40f On Tue, May 31, 2016 at 3:10 PM, Raju Bairishetti wrote: > > > On Tue, May 31, 2016 at 1:02

Re: Spark SQL Errors

On Tue, May 31, 2016 at 1:02 PM, ayan guha wrote: > Hi > > While running spark thrift, we are getting 2 issues. > > 1.. 16/05/31 14:36:18 WARN ThriftCLIService: Error executing statement: > org.apache.hive.service.cli.HiveSQLException: > org.apache.spark.sql.AnalysisException: Table not found: >

Spark SQL Errors

Hi While running spark thrift, we are getting 2 issues. 1.. 16/05/31 14:36:18 WARN ThriftCLIService: Error executing statement: org.apache.hive.service.cli.HiveSQLException: org.apache.spark.sql.AnalysisException: Table not found: sds.unhealthy_om_delta; at org.apache.spark.sql.hive.thrif

Re: Spark + Kafka processing trouble

how are you getting your data from the database. Are you using JDBC. Can you actually call the database first (assuming the same data, put it in temp table in Spark and cache it for the duration of windows length and use the data from the cached table? Dr Mich Talebzadeh LinkedIn * https://ww

Re: equvalent beewn join sql and data frame

one is sql and the other one is its equivalent in functional programming val s = HiveContext.table("sales").select("AMOUNT_SOLD","TIME_ID","CHANNEL_ID") val c = HiveContext.table("channels").select("CHANNEL_ID","CHANNEL_DESC") val t = HiveContext.table("times").select("TIME_ID","CALENDAR_MONTH_DES

Re: Spark Streaming heap space out of memory

Hi Christian can you please try if 30seconds works for your case .I think your batches are getting queued up .Regards Shahbaz On Tuesday 31 May 2016, Dancuart, Christian wrote: > While it has heap space, batches run well below 15 seconds. > > > > Once it starts to run out of space, processing ti

Re: equvalent beewn join sql and data frame

Hi, The same they are. If you check the equality, you can use DataFrame#explain. // maropu On Tue, May 31, 2016 at 12:26 PM, pseudo oduesp wrote: > hi guys , > it s similare thing to do : > > sqlcontext.join("select * from t1 join t2 on condition) and > > df1.join(df2,condition,'inner")?

equvalent beewn join sql and data frame

hi guys , it s similare thing to do : sqlcontext.join("select * from t1 join t2 on condition) and df1.join(df2,condition,'inner")?? ps:df1.registertable('t1') ps:df2.registertable('t2') thanks

Re: Spark + Kafka processing trouble

On Tue, May 31, 2016 at 3:14 PM, Darren Govoni wrote: > Well that could be the problem. A SQL database is essential a big > synchronizer. If you have a lot of spark tasks all bottlenecking on a single > database socket (is the database clustered or colocated with spark workers?) > then you will ha

Re: Spark + Kafka processing trouble

Well that could be the problem. A SQL database is essential a big synchronizer. If you have a lot of spark tasks all bottlenecking on a single database socket (is the database clustered or colocated with spark workers?) then you will have blocked threads on the database server. Sent from

Re: Spark + Kafka processing trouble

On Tue, May 31, 2016 at 1:56 PM, Darren Govoni wrote: > So you are calling a SQL query (to a single database) within a spark > operation distributed across your workers? Yes, but currently with very small sets of data (1-10,000) and on a single (dev) machine right now. (sorry didn't reply to

RE: Spark + Kafka processing trouble

So you are calling a SQL query (to a single database) within a spark operation distributed across your workers?  Sent from my Verizon Wireless 4G LTE smartphone Original message From: Malcolm Lockyer Date: 05/30/2016 9:45 PM (GMT-05:00) To: user@spark.apache.org Su

Spark + Kafka processing trouble

Hopefully this is not off topic for this list, but I am hoping to reach some people who have used Kafka + Spark before. We are new to Spark and are setting up our first production environment and hitting a speed issue that maybe configuration related - and we have little experience in configuring

Re: Can we use existing R model in Spark

I mean train random forest model (not using R) and use it for prediction together using Spark ML. > On May 30, 2016, at 20:15, Neha Mehta wrote: > > Thanks Sujeet.. will try it out. > > Hi Sun, > > Can you please tell me what did you mean by "Maybe you can try using the > existing random fore

Re: GraphX Java API

btw, GraphX in Action is one of the better books out on Spark. Michael did a great job with this one. He even breaks down snippets of Scala for newbies to understand the seemingly-arbitrary syntax. I learned quite a bit about not only Spark, but also Scala. And of course, we shouldn't forget ab

Re: Using Spark on Hive with Hive also using Spark as its execution engine

I think we are going to move to a model that the computation stack will be separate from storage stack and moreover something like Hive that provides the means for persistent storage (well HDFS is the one that stores all the data) will have an in-memory type capability much like what Oracle TimesTe

Re: Using Spark on Hive with Hive also using Spark as its execution engine

And you have MapR supporting Apache Drill. So these are all alternatives to Spark, and its not necessarily an either or scenario. You can have both. > On May 30, 2016, at 12:49 PM, Mich Talebzadeh > wrote: > > yep Hortonworks supports Tez for one reason or other which I am going > hopefull

Re: Using Spark on Hive with Hive also using Spark as its execution engine

I do not think that in-memory itself will make things faster in all cases. Especially if you use Tez with Orc or parquet. Especially for ad hoc queries on large dataset (indecently if they fit in-memory or not) this will have a significant impact. This is an experience I have also with the in-m

RE: Spark Streaming heap space out of memory

While it has heap space, batches run well below 15 seconds. Once it starts to run out of space, processing time takes about 1.5 minutes. Scheduling delay is around 4 minutes and total delay around 5.5 minutes. I usually shut it down at that point. The number of stages (and pending stages) does

Re: Using Spark on Hive with Hive also using Spark as its execution engine

Spark in relation to Tez can be like a Flink runner for Apache Beam? The use case of Tez however may be interesting (but current implementation only YARN-based?) Spark is efficient (or faster) for a number of reasons, including its ‘in-memory’ execution (from my understanding and experiments).

Re: Does Spark support updates or deletes on underlying Hive tables

Hi, Remember that acidity and transactional support was added to Hive 0.14 onward because of advent of ORC tables. Now Spark does not support transactions because quote "there is a piece in the execution side that needs to send heartbeats to Hive metastore saying a transaction is still alive". Th

Re: Using Spark on Hive with Hive also using Spark as its execution engine

yep Hortonworks supports Tez for one reason or other which I am going hopefully to test it as the query engine for hive. Tthough I think Spark will be faster because of its in-memory support. Also if you are independent then you better off dealing with Spark and Hive without the need to support an

Re: Secondary Indexing?

your point on "At the same time… if you are dealing with a large enough set of data… you will have I/O. Both in terms of networking and Physical. This is true of both Spark and in-memory RDBMs. .." Well an IMDB will not start flushing to disk when it gets full, thus doing PIO. It won't be able to

Re: Spark Streaming heap space out of memory

Hi Christian, - What is the processing time of each of your Batch,is it exceeding 15 seconds. - How many jobs are queued. - Can you take a heap dump and see which objects are occupying the heap. Regards, Shahbaz On Tue, May 31, 2016 at 12:21 AM, christian.dancu...@rbc.com < christ

Re: Using Spark on Hive with Hive also using Spark as its execution engine

Mich, Most people use vendor releases because they need to have the support. Hortonworks is the vendor who has the most skin in the game when it comes to Tez. If memory serves, Tez isn’t going to be M/R but a local execution engine? Then LLAP is the in-memory piece to speed up Tez? HTH -M

Re: HiveContext standalone => without a Hive metastore

Going from memory… Derby is/was Cloudscape which IBM acquired from Informix who bought the company way back when. (Since IBM released it under Apache licensing, Sun Microsystems took it and created JavaDB…) I believe that there is a networking function so that you can either bring it up in st

Re: Secondary Indexing?

I have to clarify something… In SparkSQL, we can query against both immutable existing RDDs, and Hive/HBase/MapRDB/ which are mutable. So we have to keep this in mind while we are talking about secondary indexing. (Its not just RDDs) I think the only advantage to being immutable is that once

Re: Bulk loading Serialized RDD into Hbase throws KryoException - IndexOutOfBoundsException

Put is a type of Mutation so not sure what you mean by if I use mutation. Anyway I registered all 3 classes to kryo. kryo.register(classOf[org.apache.hadoop.hbase.client.Put]) kryo.register(classOf[ImmutableBytesWritable]) kryo.register(classOf[Mutable]) It still fails with the same exception

Spark Streaming heap space out of memory

Hi All, We have a spark streaming v1.4/java 8 application that slows down and eventually runs out of heap space. The less driver memory, the faster it happens. Appended is our spark configuration and a snapshot of the of heap taken using jmap on the driver process. The RDDInfo, $colon$colon and [

Does Spark support updates or deletes on underlying Hive tables

Hi, I can do inserts from Spark on Hive tables. How about updates or deletes. They are failing when I tried? Thanking

Re: Secondary Indexing?

Hi, have you tried using partitioning and parquet format. It works super fast in SPARK. Regards, Gourav On Mon, May 30, 2016 at 5:08 PM, Michael Segel wrote: > I’m not sure where to post this since its a bit of a philosophical > question in terms of design and vision for spark. > > If we look

Re: Secondary Indexing?

Just a thought Well in Spark RDDs are immutable which is an advantage compared to a conventional IMDB like Oracle TimesTen meaning concurrency is not an issue for certain indexes. The overriding optimisation (as there is no Physical IO) has to be reducing memory footprint and CPU demands and usin

Re: can not use udf in hivethriftserver2

Can you try adding jar to SPARK_CLASSPATH env variable ? On Mon, May 30, 2016 at 9:55 PM, 喜之郎 <251922...@qq.com> wrote: > HI all, I have a problem when using hiveserver2 and beeline. > when I use CLI mode, the udf works well. > But when I begin to use hiveserver2 and beeline, the udf can not work

Window Operation on Dstream Fails

Hello, I am using 1.6.0 version of Spark and trying to run window operation on DStreams. Window_TwoMin = 4*60*1000 Slide_OneMin = 2*60*1000 census = ssc.textFileStream("./census_stream/").filter(lambda a: a.startswith('-') == False).map(lambda b: b.split("\t")) .map(lambda c: (c[0],c[2],c[3],e

can not use udf in hivethriftserver2

HI all, I have a problem when using hiveserver2 and beeline. when I use CLI mode, the udf works well. But when I begin to use hiveserver2 and beeline, the udf can not work. My Spark version is 1.5.1. I tried 2 methods, first: ## add jar /home/hadoop/dmp-udf-0.0.1-SNAPSHOT.jar; create temporary

Secondary Indexing?

I’m not sure where to post this since its a bit of a philosophical question in terms of design and vision for spark. If we look at SparkSQL and performance… where does Secondary indexing fit in? The reason this is a bit awkward is that if you view Spark as querying RDDs which are temporary, i

Re: GraphX Java API

Yes, it is possible to use GraphX from Java but it requires 10x the amount of code and involves using obscure typing and pre-defined lambda prototype facilities. I give an example of it in my book, the source code for which can be downloaded for free from  https://www.manning.com/books/spark-gra

Re: Bug of PolynomialExpansion ?

The 2-degree expansion of (x,y,z) is, in this implementation: (x, x^2, y, xy, y^2, z, xz, yz, z^2) Given your input is (1,0,1), the output (1,1,0,0,0,1,1,0,1) is right. On Mon, May 30, 2016 at 12:37 AM, Jeff Zhang wrote: > I use PolynomialExpansion to convert one vector to 2-degree vector. I a

Re: Running glm in sparkR (data pre-processing step)

Yes, you are right. 2016-05-30 2:34 GMT-07:00 Abhishek Anand : > > Thanks Yanbo. > > So, you mean that if I have a variable which is of type double but I want > to treat it like String in my model I just have to cast those columns into > string and simply run the glm model. String columns will be

Re: GraphX Java API

No, you can call any Scala API in Java. It is somewhat less convenient if the method was not written with Java in mind but does work. On Mon, May 30, 2016, 00:32 Takeshi Yamamuro wrote: > These package are used only for Scala. > > On Mon, May 30, 2016 at 2:23 PM, Kumar, Abhishek (US - Bengaluru)

Re: FAILED_TO_UNCOMPRESS Error - Spark 1.3.1

Hi, This is a known issue. You need to check a related JIRA ticket: https://issues.apache.org/jira/browse/SPARK-4105 // maropu On Mon, May 30, 2016 at 7:51 PM, Prashant Singh Thakur < prashant.tha...@impetus.co.in> wrote: > Hi, > > > > We are trying to use Spark Data Frames for our use case whe

Re: Can we use existing R model in Spark

Thanks Sujeet.. will try it out. Hi Sun, Can you please tell me what did you mean by "Maybe you can try using the existing random forest model" ? did you mean creating the model again using Spark MLLIB? Thanks, Neha > From: sujeet jog > Date: Mon, May 30, 2016 at 4:52 PM > Subject: Re: Can

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

How about this ? def extract_feature(rf_model, x): text = getFeatures(x).split(',') fea = [float(i) for i in text] prediction = rf_model.predict(fea) return (prediction, x) output = texts.map(lambda x: extract_feature(rf_model, x)) 2016-05-30 14:17 GMT+07:00 obaidul karim : > Hi, > > Anybody has

Re: Can we use existing R model in Spark

Try to invoke a R script from Spark using rdd pipe method , get the work done & and receive the model back in RDD. for ex :- . rdd.pipe("") On Mon, May 30, 2016 at 3:57 PM, Sun Rui wrote: > Unfortunately no. Spark does not support loading external modes (for > examples, PMML) for now. > May

Re: JDBC Cluster

when you start master it stats applicationmaster. it does not slaves/workers! you need to start slaves with start-slaves.sh slaves will look at the file $SPARK_HOME/conf/slaves to get a list of nodes to start slaves. then it will start slaves/workers in each node. you can see all this in spark GU

FAILED_TO_UNCOMPRESS Error - Spark 1.3.1

Hi, We are trying to use Spark Data Frames for our use case where we are getting this exception. The parameters used are listed below. Kindly suggest if we are missing something. Version used is Spark 1.3.1 Jira is still showing this issue as Open https://issues.apache.org/jira/browse/SPARK-410

DAG of Spark Sort application spanning two jobs

I've written a very simple Sort scala program with Spark. /object Sort { def main(args: Array[String]): Unit = { if (args.length < 2) { System.err.println("Usage: Sort " + " []") System.exit(1) } val conf = new SparkConf().setAppN

Re: Can we use existing R model in Spark

Unfortunately no. Spark does not support loading external modes (for examples, PMML) for now. Maybe you can try using the existing random forest model in Spark. > On May 30, 2016, at 18:21, Neha Mehta wrote: > > Hi, > > I have an existing random forest model created using R. I want to use that

Re: HiveContext standalone => without a Hive metastore

Michael, Mitch, Silvio, Thanks! The own directoy is the issue. We are running the Spark Notebook, which uses the same dir per server (i.e. for all notebooks). So this issue prevents us from running 2 notebooks using HiveContext. I'll look in a proper Hive installation and I'm glad to know that t

Can we use existing R model in Spark

Hi, I have an existing random forest model created using R. I want to use that to predict values on Spark. Is it possible to do the same? If yes, then how? Thanks & Regards, Neha

Re: Running glm in sparkR (data pre-processing step)

Thanks Yanbo. So, you mean that if I have a variable which is of type double but I want to treat it like String in my model I just have to cast those columns into string and simply run the glm model. String columns will be directly one-hot encoded by the glm provided by sparkR ? Just wanted to cl

Re: Running glm in sparkR (data pre-processing step)

Hi Abhi, In SparkR glm, category features (columns of type string) will be one-hot encoded automatically. So pre-processing like `as.factor` is not necessary, you can directly feed your data to the model training. Thanks Yanbo 2016-05-30 2:06 GMT-07:00 Abhishek Anand : > Hi , > > I want to run

Re: Launch Spark shell using differnt python version

Hi Stuti 2016-03-15 10:08 GMT+01:00 Stuti Awasthi : > Thanks Prabhu, > > I tried starting in local mode but still picking Python 2.6 only. I have > exported “DEFAULT_PYTHON” in my session variable and also included in PATH. > > > > Export: > > export DEFAULT_PYTHON="/home/stuti/Python/bin/python2

Re: JDBC Cluster

Normally, when you start the master, the slaves should also be started automatically. This, however, presupposes that you've configured the slaves. In the $SPARK_HOME/conf directory there should be a slaves or slaves.template file. If it only contains localhost, then you have not set up any worker

Re: List of questios about spark

No, the limit is given by your setup. If you use Spark on a YARN cluster, then the number of concurrent jobs is really limited to the resources allocated to each job and how the YARN queues are set up. For instance, if you use the FIFO scheduler (default), then it can be the case that the first job

Running glm in sparkR (data pre-processing step)

Hi , I want to run glm variant of sparkR for my data that is there in a csv file. I see that the glm function in sparkR takes a spark dataframe as input. Now, when I read a file from csv and create a spark dataframe, how could I take care of the factor variables/columns in my data ? Do I need t

Re: Spark Streaming: Combine MLlib Prediction and Features on Dstreams

Hi, Anybody has any idea on below? -Obaid On Friday, 27 May 2016, obaidul karim wrote: > Hi Guys, > > This is my first mail to spark users mailing list. > > I need help on Dstream operation. > > In fact, I am using a MLlib randomforest model to predict using spark > streaming. In the end, I wa

RE: Query related to spark cluster

Hi All, @Deepak: Thanks for your suggestion, we are using Mesos to handle spark cluster. @Jorn : the reason we chose postgresXL was of its geo-spational support as we store location data. We were seeing how to quickly put things better and what is the right approach Our original thinking was t