Re: 'ExecutorTaskSummary' alternative in Spark 2.3 onwards
Any advise/help here is much appreciated. On Mon, Dec 30, 2019 at 4:16 PM Ninja Coder wrote: > I have a spark streaming application (currently spark 2.2) which is using > `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details > like > duration, tasks failed, tasks completed, GC time, etc after each batch is > completed. These metrics are then loaded to prometheus and pagerduty alerts > are set on it. > > We are planning to upgrade spark and looks like`ExecutorTaskSummary` is no > longer available from spark 2.3 > > I would like to know what are the other alternatives which I can use. > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
'ExecutorTaskSummary' alternative in Spark 2.3 onwards
I have a spark streaming application (currently spark 2.2) which is using `org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details like duration, tasks failed, tasks completed, GC time, etc after each batch is completed. These metrics are then loaded to prometheus and pagerduty alerts are set on it. We are planning to upgrade spark and looks like`ExecutorTaskSummary` is no longer available from spark 2.3 I would like to know what are the other alternatives which I can use. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Using Aggregate and group by on spark Dataset api
JavaRDD prdd = sc.textFile("c:\\fls\\people.txt").map( new Function() { public Person call(String line) throws Exception { String[] parts = line.split(","); Person person = new Person(); person.setName(parts[0]); person.setAge(Integer.parseInt(parts[1].trim())); person.setSal(Integer.parseInt(parts[2].trim())); return person; } }); RDDpersonRDD = prdd.toRDD(prdd); Dataset dss= sqlContext.createDataset(personRDD , Encoders.bean(Person.class)); GroupedDataset dq=dss.groupBy(new Column("name")); I have to calculate sum of age and salary group by name on the dataset. Please help how to query dataset ? I tried using GroupedDataset but don't know how to proceed with it. I acn not find much help for using dataset api. Please help -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-Aggregate-and-group-by-on-spark-Dataset-api-tp26824.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org