Re: installation of spark

/Home. 5. echo $*JAVA_HOME* on *Terminal* to confirm the path. 6. You should now be able to run your application. *How to set JAVA_HOME on Mac OS X permanently* $ vim .bash_profile $ export JAVA_HOME=$(/usr/libexec/java_home) $ source .bash_profile $ echo $JAVA_HOME Have fun! Alonso

Re: write files of a specific size

t; Hi All, >> My spark sql job produces output as per default partition and creates N >> number of files. >> I want to create each file as 100Mb sized in the final result. >> >> how can I do it ? >> >> thanks >> rajat >> >>

Re: [Spark2.1] SparkStreaming to Cassandra performance problem

rk2-submit --packages >>>> org.apache.spark:spark-streaming-kafka_2.11:1.6.3,anguenot:pyspark-cassandra:0.7.0,org.apache.spark:spark-core_2.11:1.5.2 >>>> --conf spark.cassandra.connection.host='localhost' --num-executors 2 >>>> --executor-cores 2 SensorDa

Re: is it possible to create one KafkaDirectStream (Dstream) per topic?

Dstream) per topic within the same JVM i.e using only > one sparkcontext? > > Thanks! > -- Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>

Re: Testing Spark-Cassandra

nit test > easily or which one would be the best way to do it? library? Cassandra with > docker? > -- Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>

Re: Multiple Kafka topics processing in Spark 2.2

pics through console producer for > example. But when Spark receive the message, how it will know which topic > is this piece of message coming from? Thanks a lot for any of your helps! > > Cheers, > Dan > -- Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman

Re: Need some help around a Spark Error

I hope that helps https://stackoverflow.com/questions/40623957/slave-lost-and-very-slow-join-in-spark Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign

Re: how to identify the alive master spark via Zookeeper ?

e-command-line> ... Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2017-07-17 14:21 GMT+02:00 : > Hello, > > > >

Re: Need Spark(Scala) Performance Tuning tips

, Jacek`s <https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-tuning.html> Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=extern

Re: Java SPI jar reload in Spark

Hi, a quick search on google. https://github.com/spark-jobserver/spark-jobserver/issues/130 Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_

Re: Spark 2.1 - Infering schema of dataframe after reading json files not during

where schema_parquet can be something like this: {"type" : "struct","fields" : [ {"name" : "column0","type" : "string","nullable" : false},{"name":"column1", "type":"string", "

Re: Worker node log not showed

Are you running the code with yarn? if so, figure out the applicationID through the web ui, then run the next command: yarn logs your_application_id Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=em

Re: Spark Streaming: Custom Receiver OOM consistently

could you share the code? Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2017-05-20 7:54 GMT+02:00 Manish Malhotra : > Hello,

Re: How to read large size files from a directory ?

please create a github repo and upload the code there... Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2017-05-09 8:47 GMT+0

Re: Has anyone used CoreNLP from stanford for sentiment analysis in Spark? It does not work as desired for me.

I forked some time ago a twitter analyzer, but i think the best is to provide the original link <https://github.com/vspiewak/twitter-sentiment-analysis>. If you want you can take a look to my fork <https://github.com/alonsoir/twitter-sentiment-analysis>. regards Alonso Isidoro

Re: 答复: 答复: How to store 10M records in HDFS to speed up further filtering?

forgive my ignorance, but, what does it mean HAR? a acronym to High available record? Thanks Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_

Re: Any NLP library for sentiment analysis in Spark?

I forked some time ago a project, maybe you can use it. https://github.com/alonsoir/SparkTwitterAnalyzer Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&a

Re: Any NLP library for sentiment analysis in Spark?

i did not use it yet, but this library looks promising: https://github.com/databricks/spark-corenlp Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign

Re: Benchmarking streaming frameworks

I remember that yahoo did something similar... https://github.com/yahoo/streaming-benchmarks Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=extern

Re: How best we can store streaming data on dashboards for real time user experience?

you can check if you want this link <https://github.com/alonsoir/twitter-sentiment-analysis> elastic, kibana and spark working together. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_s

Re: How best we can store streaming data on dashboards for real time user experience?

-analytics/ https://speakerdeck.com/elasticsearch/using-elasticsearch-logstash-and-kibana-to-create-realtime-dashboards https://www.youtube.com/watch?v=PuvHINcU9DI then take a look to https://kudu.apache.org/ Tell us later what you think. Alonso Isidoro Roman [image: https://]about.me

Re: Spark Streaming-- for each new file in HDFS

Why flume isn't an option here? Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-10-05 11:14 GMT+02:00 K

Re: Architecture recommendations for a tricky use case

"Using Spark to query the data in the backend of the web UI?" Dont do that. I would recommend that spark streaming process stores data into some nosql or sql database and the web ui to query data from that database. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.ro

Re: What is the difference between mini-batch vs real time streaming in practice (not theory)?

mini batch or near real time: processing frames within 500 ms or more real time: processing frames in 5 ms-10ms. The main difference is processing velocity, i think. Apache Spark Streaming is mini batch, not true real time. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman

Re: Extract timestamp from Kafka message

hum, i think you have to embed the timestamp within the message... Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-09-26 0:

Re: Small files

, but you still will have the small files problem anyway. You have to create an uber file before you upload it to the HDFS. Regards Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_mediu

Re: Small files

, tell us something about this issue Alonso Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-09-12 12:39 GMT+02:00 ayan guha :

Re: Real Time Recommendation Engines with Spark and Scala

By the way, i would love to work in your project, looks promising! Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-09-05 16:5

Re: Real Time Recommendation Engines with Spark and Scala

in the scala shell to create this unix command. Regards! Alonso Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-09-05 16:39 G

Re: Real Time Recommendation Engines with Spark and Scala

Hi Mitch, i wrote few months ago a tiny project with this issue in mind. The idea is to apply ALS algorithm in order to get some valid recommendations from another users. The url of the project <https://github.com/alonsoir/awesome-recommendation-engine> Alonso Isidoro Roman [image:

Re: Design patterns involving Spark

Thanks Mitch, i will check it. Cheers Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-08-30 9:52 GMT+02:00 Mich Talebzade

Re: Design patterns involving Spark

HBase for real time queries? HBase was designed with the batch in mind. Impala should be a best choice, but i do not know what Druid can do Cheers Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=em

Re: Can I control the execution of Spark jobs?

Hi Wang, maybe you can consider to use an integration framework like Apache Camel in order to run differents jobs... Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&a

Re: Should I avoid "state" in an Spark application?

Hi Haopu, please check these threads: http://stackoverflow.com/questions/24331815/spark-streaming-historical-state https://databricks.gitbooks.io/databricks-spark-reference-applications/content/logs_analyzer/chapter1/total.html Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman

Re: Spark Getting data from MongoDB in JAVA

why *spark-mongodb_2.11 dependency is written twice in pom.xml?* Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-06-10 11:39 G

Re: About a problem running a spark job in a cdh-5.7.0 vmware image.

-count]$ free total used free sharedbuffers cached Mem: 806110466870441374060 3464 5796 484416 -/+ buffers/cache:61968321864272 Swap: 8388604 6875007701104 so, i can only use at least 1GB... Alonso Isidoro Roman

Re: About a problem running a spark job in a cdh-5.7.0 vmware image.

uickstart ~]$ I know that sbt-launch is the sbt command running in another terminal, but, ¿Are NameNode processes and DataNode should not appear? Thank you very much for reading until here. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro

Re: About a problem running a spark job in a cdh-5.7.0 vmware image.

example.spark.AmazonKafkaConnector.main(AmazonKafkaConnectorWithMongo.scala) Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-

Re: About a problem running a spark job in a cdh-5.7.0 vmware image.

Thank you David, so, i would have to change the way that i am creating SparkConf object, isn't? I can see in this link that the way to run a spark job using YARN is using this kin

About a problem running a spark job in a cdh-5.7.0 vmware image.

nRecommendationsPerUser <= r._2.size && r._2.size < MaxRecommendationsPerUser) .flatMap(_._2) .repartition(NumPartitions) .cache() println(s"Parsed $

Re: About a problem when mapping a file located within a HDFS vmware cdh-5.7 image

Thank you David, i will try to follow your advise. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-05-31 21:28 GMT+02:

Re: About a problem when mapping a file located within a HDFS vmware cdh-5.7 image

Hi David, the one of the develop branch. I think It should be the same, but actually not sure... Regards Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&a

About a problem when mapping a file located within a HDFS vmware cdh-5.7 image

57534 errors:0 dropped:0 overruns:0 frame:0 TX packets:57534 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0656 (42.3 MiB) TX bytes:44440656 (42.3 MiB) I think that this must be a misconfiguration in a cloudera configuration file, b

Re: Spark + Kafka processing trouble

Mich`s idea is quite fine, if i was you, i will follow his idea... Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-05-31 6

Re: Logistic Regression in Spark Streaming

I do not have any experience using LR in spark, but you can see that LR is already implemented in mllib. http://spark.apache.org/docs/latest/mllib-linear-methods.html Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_

Re: about an exception when receiving data from kafka topic using Direct mode of Spark Streaming

Thank you Cody, i will try to follow your advice. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-05-26 17:00 GMT+02

Re: about an exception when receiving data from kafka topic using Direct mode of Spark Streaming

ng data from kafka topic...") } //println("jsonParsed is " + jsonParsed) //The idea is to save results from Recommender.predict within mongodb, so i will have to deal with this issue //after resolving the issue of .foreachRDD(_.foreachPartition(recommender.predic

Re: about an exception when receiving data from kafka topic using Direct mode of Spark Streaming

story ordered by rating (higher first) and only keep the first NumRecommendations val myUserId = userDict.getIndex(MyUsername) val recommendations = model.predict(candidates.map((myUserId, _))).collect val endAls = DateTime.now val result = recommendations.sortBy(-_.rating).take(NumRecom

about an exception when receiving data from kafka topic using Direct mode of Spark Streaming

e? possibly not, what about using avro or parquet? producer.send(Json.toJson(amazonRating).toString) //producer.send(amazonRating) println("amazon product with rating sent to kafka cluster..." + amazonRating.toString) System.exit(0) } } } I have written

about an exception when receiving data from kafka topic using Direct mode of Spark Streaming

e? possibly not, what about using avro or parquet? producer.send(Json.toJson(amazonRating).toString) //producer.send(amazonRating) println("amazon product with rating sent to kafka cluster..." + amazonRating.toString) System.exit(0) } } } I have

Re: Spark JOIN Not working

Could you share a bit of the dataset? difficult to test without data... Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-05

Re: My notes on Spark Performance & Tuning Guide

Me too, send me the guide. Enviado desde mi iPhone > El 12 may 2016, a las 12:11, Ashok Kumar > escribió: > > Hi Dr Mich, > > I will be very keen to have a look at it and review if possible. > > Please forward me a copy > > Thanking you warmly > > > On Thursday, 12 May 2016, 11:08, Mich T

Re: [Spark 1.5.2]Check Foreign Key constraint

I think that Impala and Hive have this feature. Impala is faster than hive, hive is probably better to use in batch mode. Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_mediu

Re: Multiple Spark Applications that use Cassandra, how to share resources/nodes

Andy, i think there are some ideas to implement a pool of spark context, but, for now, it is only an idea. https://github.com/spark-jobserver/spark-jobserver/issues/365 It is possible to share a spark context between apps, i did not have to use this feature, sorry about that. Regards, Alonso

Re: migration from Teradata to Spark SQL

I agree with Deepak and i would try to save data in parquet and avro format, if you can, try to measure the performance and choose the best, it will probably be parquet, but you have to know for yourself. Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de q

Re: Spark support for Complex Event Processing (CEP)

Stratio decision could do the job https://github.com/Stratio/Decision Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar debe ser el proceso de introducirlos..." - Edsger Dijkstra My favorite quo

Re: Code optimization

Hi Angel, how about to use this : k.filter(k("WT_ID") as a val variable? i think you can avoid that and do not forget to use System.nanoTime to know the profit... Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de sof

Re: Run a self-contained Spark app on a Spark standalone cluster

I don't know how to do it with python, but scala has a plugin named sbt-pack that creates an auto contained unix command with your code, no need to use spark-submit. It should be out there something similar to this tool. Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si dep

Re: Problem with jackson lib running on spark

As you can see jackson-core is provided by several libraries, try to exclude it from spark-core, i think the minor version is included within it. Use this guide to see how to do it: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html Alonso

Re: Problem with jackson lib running on spark

Run mvn dependency:tree and print the output here, i suspect that jackson library is included within more than one dependency. Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar debe ser el proce

What version of twitter4j should I use with Spark Streaming?UPDATING thread

.deploy.SparkSubmit$.main(SparkSubmit.scala:121) 42. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 43. MacBook-Pro-Retina-de-Alonso:spark-1.6 aironman$ Do i have to update the jars version? because i see that actual version of twitter4j is 4.0.4... thanks Alonso Isidoro Roman. Mis

Re: Spark Job Hanging on Join

thanks for sharing the know how guys Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar debe ser el proceso de introducirlos..." - Edsger Dijkstra My favorite quotes (today): "If debugging is

Re: Stored proc with spark

relational databases? what about sqoop? https://en.wikipedia.org/wiki/Sqoop Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar debe ser el proceso de introducirlos..." - Edsger Dijkstra My favor

Re: Spark job does not perform well when some RDD in memory and some on Disk

"But learned that it is better not to reduce it to 0." could you explain a bit more this sentence? thanks Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar debe ser el proceso de introducirlos

Re: How to integrate Spark with OpenCV?

thanks Jorn, sorry for the special character your name needs, i dont know how to use it. I was thinking the same. Do you know somebody that tries to use this approach? Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, ent

Re: How to integrate Spark with OpenCV?

, please provide. Thanks again and forgive the possible off topic. Alonso Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar debe ser el proceso de introducirlos..." - Edsger Dijkstra My favorite quotes (t

Re: how to run spark function in a tomcat servlet

Hi, i think this post <http://stackoverflow.com/questions/2681759/is-there-anyway-to-exclude-artifacts-inherited-from-a-parent-pom> shall help you. Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quitar los errores de software, entonces programar

about a JavaWordCount example with spark-core_2.10-1.0.0.jar

closures, is it possible to use this fw without using it? Another question, are this objects, JavaSparkContext sc, JavaPairRDD mongoRDD ThreadSafe? Can i use them as singleton? Thank you very much and apologizes if the questions are not trending topic :) Alonso Isidoro Roman. Mis citas preferid

Re: Beginners Hadoop question

copy MYFILE into hadoop distributed file system. Can i recommend you what i have done? go to BigDataUniversity.com and take the Hadoop Fundamentals I course. It is free and very well documented. Regards Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depurar es el proceso de quita