Re: Spark performance in cluster mode using yarn

Sachin On Fri, May 15, 2015 at 6:57 AM, ayan guha guha.a...@gmail.com wrote: With this information it is hard to predict. What's the performance you are getting? What's your desired performance? Maybe you can post your code and experts can suggests improvement? On 14 May 2015 15:02, sachin Singh

Spark performance in cluster mode using yarn

Hi Friends, please someone can give the idea, Ideally what should be time(complete job execution) for spark job, I have data in a hive table, amount of data would be 1GB , 2 lacs rows for whole month, I want to do monthly aggregation, using SQL queries,groupby I have only one node,1

spark yarn-cluster job failing in batch processing

Hi All, I am trying to execute batch processing in yarn-cluster mode i.e. I have many sql insert queries,based on argument provided it will it will fetch the queries ,create context , schema RDD and insert in hive tables, Please Note- in standalone mode its working and in cluster mode working is

Re: Spark sql failed in yarn-cluster mode when connecting to non-default hive database

Hi Linlin, have you got the solution for this issue, if yes then what are the thing need to make correct,because I am also getting same error,when submitting spark job in cluster mode getting error as under - 2015-04-14 18:16:43 DEBUG Transaction - Transaction rolled back in 0 ms 2015-04-14

ExceptionDriver-Memory while running Spark job on Yarn-cluster

Hi , When I am submitting spark job as --master yarn-cluster with below command/options getting driver memory error- spark-submit --jars ./libs/mysql-connector-java-5.1.17.jar,./libs/log4j-1.2.17.jar --files datasource.properties,log4j.properties --master yarn-cluster --num-executors 1

need info on Spark submit on yarn-cluster mode

Hi , I observed that we have installed only one cluster, and submiting job as yarn-cluster then getting below error, so is this cause that installation is only one cluster? Please correct me, if this is not cause then why I am not able to run in cluster mode, spark submit command is - spark-submit

issue while submitting Spark Job as --master yarn-cluster

Hi , when I am submitting spark job in cluster mode getting error as under in hadoop-yarn log, someone has any idea,please suggest, 2015-03-25 23:35:22,467 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1427124496008_0028 State change from FINAL_SAVING to FAILED

Re: issue while submitting Spark Job as --master yarn-cluster

. On Wed, Mar 25, 2015 at 9:07 PM sachin Singh sachin.sha...@gmail.com wrote: Hi , when I am submitting spark job in cluster mode getting error as under in hadoop-yarn log, someone has any idea,please suggest, 2015-03-25 23:35:22,467 INFO

Re: issue while creating spark context

but the configuration specifies a local path. See the exception message. On Tue, Mar 24, 2015 at 1:08 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Its in your local file system, not in hdfs. Thanks Best Regards On Tue, Mar 24, 2015 at 6:25 PM, Sachin Singh sachin.sha...@gmail.com wrote

Re: issue while creating spark context

permission over */user/spark* directory. Thanks Best Regards On Tue, Mar 24, 2015 at 5:21 PM, sachin Singh sachin.sha...@gmail.com wrote: hi all, all of sudden I getting below error when I am submitting spark job using master as yarn its not able to create spark context,previously

Re: issue while creating spark context

Regards On Tue, Mar 24, 2015 at 6:08 PM, Sachin Singh sachin.sha...@gmail.com wrote: Hi Akhil, thanks for your quick reply, I would like to request please elaborate i.e. what kind of permission required .. thanks in advance, Regards Sachin On Tue, Mar 24, 2015 at 5:29 PM, Akhil Das ak

issue while creating spark context

hi all, all of sudden I getting below error when I am submitting spark job using master as yarn its not able to create spark context,previously working fine, I am using CDH5.3.1 and creating javaHiveContext spark-submit --jars

Re: issue while creating spark context

thanks Sean and Akhil, I changed the the permission of */user/spark/applicationHistory, *now it works, On Tue, Mar 24, 2015 at 7:35 PM, Sachin Singh sachin.sha...@gmail.com wrote: thanks Sean, please can you suggest in which file or configuration I need to modify proper path, please

issue creating spark context with CDH 5.3.1

Hi, I am using CDH5.3.1 I am getting bellow error while, even spark context not getting created, I am submitting my job like this - submitting command- spark-submit --jars

Re: issue creating spark context with CDH 5.3.1

I have copied hive-site.xml to spark conf folder cp /etc/hive/conf/hive-site.xml /usr/lib/spark/conf -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/issue-creating-spark-context-with-CDH-5-3-1-tp21968p21969.html Sent from the Apache Spark User List mailing

Re: issue Running Spark Job on Yarn Cluster

Not yet, Please let. Me know if you found solution, Regards Sachin On 4 Mar 2015 21:45, mael2210 [via Apache Spark User List] ml-node+s1001560n21909...@n3.nabble.com wrote: Hello, I am facing the exact same issue. Could you solve the problem ? Kind regards --

Re: issue Running Spark Job on Yarn Cluster

Yes. On 19 Feb 2015 23:40, Harshvardhan Chauhan ha...@gumgum.com wrote: Is this the full stack trace ? On Wed, Feb 18, 2015 at 2:39 AM, sachin Singh sachin.sha...@gmail.com wrote: Hi, I want to run my spark Job in Hadoop yarn Cluster mode, I am using below command - spark-submit --master

issue Running Spark Job on Yarn Cluster

Hi, I want to run my spark Job in Hadoop yarn Cluster mode, I am using below command - spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --class com.dc.analysis.jobs.AggregationJob sparkanalitic.jar param1 param2 param3 I am getting error as under,

how to get SchemaRDD SQL exceptions i.e. table not found exception

Hi, can some one guide how to get SQL Exception trapped for query executed using SchemaRDD, i mean suppose table not found thanks in advance, -- View this message in context:

how to avoid Spark and Hive log from Application log

Hi, Please can somebody help ,how to avoid Spark and Hive log from Application log, I mean both spark and hive are using log4j property file , I have configured log4j.property file as per my application as under but its printing Spark and hive console logging also,please suggest its urgent for me,

getting error when submit spark with master as yarn

Hi, when I am trying to execute my program as spark-submit --master yarn --class com.mytestpack.analysis.SparkTest sparktest-1.jar I am getting error bellow error- java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!

how to send JavaDStream RDD using foreachRDD using Java

Hi I want to send streaming data to kafka topic, I am having RDD data which I converted in JavaDStream ,now I want to send it to kafka topic, I don't want kafka sending code, just I need foreachRDD implementation, my code is look like as public void publishtoKafka(ITblStream t) {

Spark SQL implementation error

I have a table(csv file) loaded data on that by creating POJO as per table structure,and created SchemaRDD as under JavaRDDTest1 testSchema = sc.textFile(D:/testTable.csv).map(GetTableData);/* GetTableData will transform the all table data in testTable object*/ JavaSchemaRDD schemaTest =

JavaRDD (Data Aggregation) based on key

Hi, I have a csv file having fields as a,b,c . I want to do aggregation(sum,average..) based on any field(a,b or c) as per user input, using Apache Spark Java API,Please Help Urgent! Thanks in advance, Regards Sachin -- View this message in context: