the issue about the + in column,can we support the string please?

[1427357...@qq.com]

Hi  all,

I have a table like below:

+---+-+---+
| id| name|sharding_id|
+---+-+---+
|  1|leader us|  1|
|  3|mycat|  1|
+---+-+---+

My schema is :
root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- sharding_id: integer (nullable = false)

I want add a new column named newName. The new column is based on "name" and 
append "abc" after it. My code looks like:

stud_scoreDF.withColumn("newName", stud_scoreDF.col("name") +  "abc"  ).show()
When I run the code, I got the reslult:
+---+-+---+---+
| id| name|sharding_id|newName|
+---+-+---+---+
|  1|leader us|  1|   null|
|  3|mycat|  1|   null|
+---+-+---+---+


I checked the code, the key code is  in arithmetic.scala. line 165.
It looks like:

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = dataType 
match {
  case dt: DecimalType =>
defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.$$plus($eval2)")
  case ByteType | ShortType =>
defineCodeGen(ctx, ev,
  (eval1, eval2) => s"(${ctx.javaType(dataType)})($eval1 $symbol $eval2)")
  case CalendarIntervalType =>
defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.add($eval2)")
  case _ =>
defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1 $symbol $eval2")
}

My issue is:
Can we add case StringType in this class to support string append please?





1427357...@qq.com

Re:Re: how to use lit() in spark-java

[崔苗]

It works,thanks

在 2018-03-23 21:33:41，Anil Langote  写道：
You have import functions 


dataset.withColumn(columnName,functions.lit("constant"))


Thank you
Anil Langote


Sent from my iPhone

_
 From: 崔苗 
 Sent: Friday, March 23, 2018 8:33 AM
 Subject: how to use lit() in spark-java
 To: 


Hi Guys, 

I want to add a constant column to dataset by lit function in java, like that:
  dataset.withColumn(columnName,lit("constant"))
 but it's seems that idea coundn't found the lit() function,so how to use lit() 
function in java?

 thanks for any reply

Re:Re: how to use lit() in spark-java

[崔苗]

It works,thanks

在 2018-03-23 21:47:52，"Anthony, Olufemi"  写道：
You can us import static to import it directly:
 
import static org.apache.spark.sql.functions.lit;
 
Femi
From: 崔苗 
 Date: Friday, March 23, 2018 at 8:34 AM
 To: "user@spark.apache.org" 
 Subject: how to use lit() in spark-java

 

Hi Guys, 
 

I want to add a constant column to dataset by lit function in java, like that:
  dataset.withColumn(columnName,lit("constant"))
 but it's seems that idea coundn't found the lit() function,so how to use lit() 
function in java?

 thanks for any reply











The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

[Shmuel Blitz]

Hi Rohit,

Thanks for the analysis.

I can use repartition on the slow task. But how can I tell what part of the
code is in charge of the slow tasks?

It would be great if you could further explain the rest of the output.

Thanks in advance,
Shmuel

On Sun, Mar 25, 2018 at 12:46 PM, Rohit Karlupia  wrote:

> Thanks Shamuel for trying out sparklens!
>
> Couple of things that I noticed:
> 1) 250 executors is probably overkill for this job. It would run in same
> time with around 100.
> 2) Many of stages that take long time have only 200 tasks where as we have
> 750 cores available for the job. 200 is the default value for
> spark.sql.shuffle.partitions.  Alternatively you could try increasing the
> value of spark.sql.shuffle.partitions to latest 750.
>
> thanks,
> rohitk
>
> On Sun, Mar 25, 2018 at 1:25 PM, Shmuel Blitz  > wrote:
>
>> I ran it on a single job.
>> SparkLens has an overhead on the job duration. I'm not ready to enable it
>> by default on all our jobs.
>>
>> Attached is the output.
>>
>> Still trying to understand what exactly it means.
>>
>> On Sun, Mar 25, 2018 at 10:40 AM, Fawze Abujaber 
>> wrote:
>>
>>> Nice!
>>>
>>> Shmuel, Were you able to run on a cluster level or for a specific job?
>>>
>>> Did you configure it on the spark-default.conf?
>>>
>>> On Sun, 25 Mar 2018 at 10:34 Shmuel Blitz 
>>> wrote:
>>>
 Just to let you know, I have managed to run SparkLens on our cluster.

 I switched to the spark_1.6 branch, and also compiled against the
 specific image of Spark we are using (cdh5.7.6).

 Now I need to figure out what the output means... :P

 Shmuel

 On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber 
 wrote:

> Quick question:
>
> how to add the  --jars /path/to/sparklens_2.11-0.1.0.jar to the
> spark-default conf, should it be using:
>
> spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i
> should use spark.jars option? anyone who could give an example how it
> should be, and if i the path for the jar should be an hdfs path as i'm
> using it in cluster mode.
>
>
>
>
> On Fri, Mar 23, 2018 at 6:33 AM, Fawze Abujaber 
> wrote:
>
>> Hi Shmuel,
>>
>> Did you compile the code against the right branch for Spark 1.6.
>>
>> I tested it and it looks working and now i'm testing the branch for a
>> wide tests, Please use the branch for Spark 1.6
>>
>> On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz <
>> shmuel.bl...@similarweb.com> wrote:
>>
>>> Hi Rohit,
>>>
>>> Thanks for sharing this great tool.
>>> I tried running a spark job with the tool, but it failed with an 
>>> *IncompatibleClassChangeError
>>> *Exception.
>>>
>>> I have opened an issue on Github.(https://github.com/qub
>>> ole/sparklens/issues/1)
>>>
>>> Shmuel
>>>
>>> On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz <
>>> shmuel.bl...@similarweb.com> wrote:
>>>
 Thanks.

 We will give this a try and report back.

 Shmuel

 On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia 
 wrote:

> Thanks everyone!
> Please share how it works and how it doesn't. Both help.
>
> Fawaze, just made few changes to make this work with spark 1.6.
> Can you please try building from branch *spark_1.6*
>
> thanks,
> rohitk
>
>
>
> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber <
> fawz...@gmail.com> wrote:
>
>> It's super amazing  i see it was tested on spark 2.0.0 and
>> above, what about Spark 1.6 which is still part of Cloudera's main 
>> versions?
>>
>> We have a vast Spark applications with version 1.6.0
>>
>> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau <
>> hol...@pigscanfly.ca> wrote:
>>
>>> Super exciting! I look forward to digging through it this
>>> weekend.
>>>
>>> On Wed, Mar 21, 2018 at 9:33 PM ☼ R Nair (रविशंकर नायर) <
>>> ravishankar.n...@gmail.com> wrote:
>>>
 Excellent. You filled a missing link.

 Best,
 Passion

 On Wed, Mar 21, 2018 at 11:36 PM, Rohit Karlupia <
 roh...@qubole.com> wrote:

> Hi,
>
> Happy to announce the availability of Sparklens as open source
> project. It helps in understanding the  scalability limits of 
> spark
> applications and can be a useful guide on the path towards tuning
> applications for lower runtime or cost.
>
> Please clone from here: https://github.com/qubole/sparklens
> Old blogpost: https://www.qubole.c
> om/blog/introducing-quboles-spark-tuning-tool/

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

[Rohit Karlupia]

Thanks Shamuel for trying out sparklens!

Couple of things that I noticed:
1) 250 executors is probably overkill for this job. It would run in same
time with around 100.
2) Many of stages that take long time have only 200 tasks where as we have
750 cores available for the job. 200 is the default value for
spark.sql.shuffle.partitions.  Alternatively you could try increasing the
value of spark.sql.shuffle.partitions to latest 750.

thanks,
rohitk

On Sun, Mar 25, 2018 at 1:25 PM, Shmuel Blitz 
wrote:

> I ran it on a single job.
> SparkLens has an overhead on the job duration. I'm not ready to enable it
> by default on all our jobs.
>
> Attached is the output.
>
> Still trying to understand what exactly it means.
>
> On Sun, Mar 25, 2018 at 10:40 AM, Fawze Abujaber 
> wrote:
>
>> Nice!
>>
>> Shmuel, Were you able to run on a cluster level or for a specific job?
>>
>> Did you configure it on the spark-default.conf?
>>
>> On Sun, 25 Mar 2018 at 10:34 Shmuel Blitz 
>> wrote:
>>
>>> Just to let you know, I have managed to run SparkLens on our cluster.
>>>
>>> I switched to the spark_1.6 branch, and also compiled against the
>>> specific image of Spark we are using (cdh5.7.6).
>>>
>>> Now I need to figure out what the output means... :P
>>>
>>> Shmuel
>>>
>>> On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber 
>>> wrote:
>>>
 Quick question:

 how to add the  --jars /path/to/sparklens_2.11-0.1.0.jar to the
 spark-default conf, should it be using:

 spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i
 should use spark.jars option? anyone who could give an example how it
 should be, and if i the path for the jar should be an hdfs path as i'm
 using it in cluster mode.




 On Fri, Mar 23, 2018 at 6:33 AM, Fawze Abujaber 
 wrote:

> Hi Shmuel,
>
> Did you compile the code against the right branch for Spark 1.6.
>
> I tested it and it looks working and now i'm testing the branch for a
> wide tests, Please use the branch for Spark 1.6
>
> On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz <
> shmuel.bl...@similarweb.com> wrote:
>
>> Hi Rohit,
>>
>> Thanks for sharing this great tool.
>> I tried running a spark job with the tool, but it failed with an 
>> *IncompatibleClassChangeError
>> *Exception.
>>
>> I have opened an issue on Github.(https://github.com/qub
>> ole/sparklens/issues/1)
>>
>> Shmuel
>>
>> On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz <
>> shmuel.bl...@similarweb.com> wrote:
>>
>>> Thanks.
>>>
>>> We will give this a try and report back.
>>>
>>> Shmuel
>>>
>>> On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia 
>>> wrote:
>>>
 Thanks everyone!
 Please share how it works and how it doesn't. Both help.

 Fawaze, just made few changes to make this work with spark 1.6. Can
 you please try building from branch *spark_1.6*

 thanks,
 rohitk



 On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber >>> > wrote:

> It's super amazing  i see it was tested on spark 2.0.0 and
> above, what about Spark 1.6 which is still part of Cloudera's main 
> versions?
>
> We have a vast Spark applications with version 1.6.0
>
> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau <
> hol...@pigscanfly.ca> wrote:
>
>> Super exciting! I look forward to digging through it this weekend.
>>
>> On Wed, Mar 21, 2018 at 9:33 PM ☼ R Nair (रविशंकर नायर) <
>> ravishankar.n...@gmail.com> wrote:
>>
>>> Excellent. You filled a missing link.
>>>
>>> Best,
>>> Passion
>>>
>>> On Wed, Mar 21, 2018 at 11:36 PM, Rohit Karlupia <
>>> roh...@qubole.com> wrote:
>>>
 Hi,

 Happy to announce the availability of Sparklens as open source
 project. It helps in understanding the  scalability limits of spark
 applications and can be a useful guide on the path towards tuning
 applications for lower runtime or cost.

 Please clone from here: https://github.com/qubole/sparklens
 Old blogpost: https://www.qubole.c
 om/blog/introducing-quboles-spark-tuning-tool/

 thanks,
 rohitk

 PS: Thanks for the patience. It took couple of months to get
 back on this.





>>> --
>> Twitter: https://twitter.com/holdenkarau
>>
>
>

>>>
>>>
>>> --
>>> Shmuel Blitz
>>> Big Data Developer
>>> Email: shmuel.bl...@similarweb.com
>>> www.similarweb.com
>>> 

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

[Shmuel Blitz]

I ran it on a single job.
SparkLens has an overhead on the job duration. I'm not ready to enable it
by default on all our jobs.

Attached is the output.

Still trying to understand what exactly it means.

On Sun, Mar 25, 2018 at 10:40 AM, Fawze Abujaber  wrote:

> Nice!
>
> Shmuel, Were you able to run on a cluster level or for a specific job?
>
> Did you configure it on the spark-default.conf?
>
> On Sun, 25 Mar 2018 at 10:34 Shmuel Blitz 
> wrote:
>
>> Just to let you know, I have managed to run SparkLens on our cluster.
>>
>> I switched to the spark_1.6 branch, and also compiled against the
>> specific image of Spark we are using (cdh5.7.6).
>>
>> Now I need to figure out what the output means... :P
>>
>> Shmuel
>>
>> On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber 
>> wrote:
>>
>>> Quick question:
>>>
>>> how to add the  --jars /path/to/sparklens_2.11-0.1.0.jar to the
>>> spark-default conf, should it be using:
>>>
>>> spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i
>>> should use spark.jars option? anyone who could give an example how it
>>> should be, and if i the path for the jar should be an hdfs path as i'm
>>> using it in cluster mode.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 23, 2018 at 6:33 AM, Fawze Abujaber 
>>> wrote:
>>>
 Hi Shmuel,

 Did you compile the code against the right branch for Spark 1.6.

 I tested it and it looks working and now i'm testing the branch for a
 wide tests, Please use the branch for Spark 1.6

 On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz <
 shmuel.bl...@similarweb.com> wrote:

> Hi Rohit,
>
> Thanks for sharing this great tool.
> I tried running a spark job with the tool, but it failed with an 
> *IncompatibleClassChangeError
> *Exception.
>
> I have opened an issue on Github.(https://github.com/qub
> ole/sparklens/issues/1)
>
> Shmuel
>
> On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz <
> shmuel.bl...@similarweb.com> wrote:
>
>> Thanks.
>>
>> We will give this a try and report back.
>>
>> Shmuel
>>
>> On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia 
>> wrote:
>>
>>> Thanks everyone!
>>> Please share how it works and how it doesn't. Both help.
>>>
>>> Fawaze, just made few changes to make this work with spark 1.6. Can
>>> you please try building from branch *spark_1.6*
>>>
>>> thanks,
>>> rohitk
>>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber 
>>> wrote:
>>>
 It's super amazing  i see it was tested on spark 2.0.0 and
 above, what about Spark 1.6 which is still part of Cloudera's main 
 versions?

 We have a vast Spark applications with version 1.6.0

 On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau >>> > wrote:

> Super exciting! I look forward to digging through it this weekend.
>
> On Wed, Mar 21, 2018 at 9:33 PM ☼ R Nair (रविशंकर नायर) <
> ravishankar.n...@gmail.com> wrote:
>
>> Excellent. You filled a missing link.
>>
>> Best,
>> Passion
>>
>> On Wed, Mar 21, 2018 at 11:36 PM, Rohit Karlupia <
>> roh...@qubole.com> wrote:
>>
>>> Hi,
>>>
>>> Happy to announce the availability of Sparklens as open source
>>> project. It helps in understanding the  scalability limits of spark
>>> applications and can be a useful guide on the path towards tuning
>>> applications for lower runtime or cost.
>>>
>>> Please clone from here: https://github.com/qubole/sparklens
>>> Old blogpost: https://www.qubole.com/blog/introducing-quboles-sp
>>> ark-tuning-tool/
>>>
>>> thanks,
>>> rohitk
>>>
>>> PS: Thanks for the patience. It took couple of months to get
>>> back on this.
>>>
>>>
>>>
>>>
>>>
>> --
> Twitter: https://twitter.com/holdenkarau
>


>>>
>>
>>
>> --
>> Shmuel Blitz
>> Big Data Developer
>> Email: shmuel.bl...@similarweb.com
>> www.similarweb.com
>> 
>> 
>> 
>>
>
>
>
> --
> Shmuel Blitz
> Big Data Developer
> Email: shmuel.bl...@similarweb.com
> www.similarweb.com
> 
> 
> 
>


>>>
>>
>>
>> --
>> Shmuel Blitz
>> Big Data Developer
>> Email: shmuel.bl...@similarweb.com
>> www.similarweb.com
>> 
>> 
>> 
>>
>


-- 
Shmuel Blitz
Big Data Developer
Email: s

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

[Fawze Abujaber]

Nice!

Shmuel, Were you able to run on a cluster level or for a specific job?

Did you configure it on the spark-default.conf?

On Sun, 25 Mar 2018 at 10:34 Shmuel Blitz 
wrote:

> Just to let you know, I have managed to run SparkLens on our cluster.
>
> I switched to the spark_1.6 branch, and also compiled against the specific
> image of Spark we are using (cdh5.7.6).
>
> Now I need to figure out what the output means... :P
>
> Shmuel
>
> On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber  wrote:
>
>> Quick question:
>>
>> how to add the  --jars /path/to/sparklens_2.11-0.1.0.jar to the
>> spark-default conf, should it be using:
>>
>> spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i
>> should use spark.jars option? anyone who could give an example how it
>> should be, and if i the path for the jar should be an hdfs path as i'm
>> using it in cluster mode.
>>
>>
>>
>>
>> On Fri, Mar 23, 2018 at 6:33 AM, Fawze Abujaber 
>> wrote:
>>
>>> Hi Shmuel,
>>>
>>> Did you compile the code against the right branch for Spark 1.6.
>>>
>>> I tested it and it looks working and now i'm testing the branch for a
>>> wide tests, Please use the branch for Spark 1.6
>>>
>>> On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz <
>>> shmuel.bl...@similarweb.com> wrote:
>>>
 Hi Rohit,

 Thanks for sharing this great tool.
 I tried running a spark job with the tool, but it failed with an 
 *IncompatibleClassChangeError
 *Exception.

 I have opened an issue on Github.(https://github.com/
 qubole/sparklens/issues/1)

 Shmuel

 On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz <
 shmuel.bl...@similarweb.com> wrote:

> Thanks.
>
> We will give this a try and report back.
>
> Shmuel
>
> On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia 
> wrote:
>
>> Thanks everyone!
>> Please share how it works and how it doesn't. Both help.
>>
>> Fawaze, just made few changes to make this work with spark 1.6. Can
>> you please try building from branch *spark_1.6*
>>
>> thanks,
>> rohitk
>>
>>
>>
>> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber 
>> wrote:
>>
>>> It's super amazing  i see it was tested on spark 2.0.0 and
>>> above, what about Spark 1.6 which is still part of Cloudera's main 
>>> versions?
>>>
>>> We have a vast Spark applications with version 1.6.0
>>>
>>> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau 
>>> wrote:
>>>
 Super exciting! I look forward to digging through it this weekend.

 On Wed, Mar 21, 2018 at 9:33 PM ☼ R Nair (रविशंकर नायर) <
 ravishankar.n...@gmail.com> wrote:

> Excellent. You filled a missing link.
>
> Best,
> Passion
>
> On Wed, Mar 21, 2018 at 11:36 PM, Rohit Karlupia <
> roh...@qubole.com> wrote:
>
>> Hi,
>>
>> Happy to announce the availability of Sparklens as open source
>> project. It helps in understanding the  scalability limits of spark
>> applications and can be a useful guide on the path towards tuning
>> applications for lower runtime or cost.
>>
>> Please clone from here: https://github.com/qubole/sparklens
>> Old blogpost: https://www.qubole.com/blog/introducing-quboles-
>> spark-tuning-tool/
>>
>> thanks,
>> rohitk
>>
>> PS: Thanks for the patience. It took couple of months to get back
>> on this.
>>
>>
>>
>>
>>
> --
 Twitter: https://twitter.com/holdenkarau

>>>
>>>
>>
>
>
> --
> Shmuel Blitz
> Big Data Developer
> Email: shmuel.bl...@similarweb.com
> www.similarweb.com
> 
> 
> 
>



 --
 Shmuel Blitz
 Big Data Developer
 Email: shmuel.bl...@similarweb.com
 www.similarweb.com
 
 
 

>>>
>>>
>>
>
>
> --
> Shmuel Blitz
> Big Data Developer
> Email: shmuel.bl...@similarweb.com
> www.similarweb.com
> 
> 
> 
>

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

[Shmuel Blitz]

Just to let you know, I have managed to run SparkLens on our cluster.

I switched to the spark_1.6 branch, and also compiled against the specific
image of Spark we are using (cdh5.7.6).

Now I need to figure out what the output means... :P

Shmuel

On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber  wrote:

> Quick question:
>
> how to add the  --jars /path/to/sparklens_2.11-0.1.0.jar to the
> spark-default conf, should it be using:
>
> spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i should
> use spark.jars option? anyone who could give an example how it should be,
> and if i the path for the jar should be an hdfs path as i'm using it in
> cluster mode.
>
>
>
>
> On Fri, Mar 23, 2018 at 6:33 AM, Fawze Abujaber  wrote:
>
>> Hi Shmuel,
>>
>> Did you compile the code against the right branch for Spark 1.6.
>>
>> I tested it and it looks working and now i'm testing the branch for a
>> wide tests, Please use the branch for Spark 1.6
>>
>> On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz <
>> shmuel.bl...@similarweb.com> wrote:
>>
>>> Hi Rohit,
>>>
>>> Thanks for sharing this great tool.
>>> I tried running a spark job with the tool, but it failed with an 
>>> *IncompatibleClassChangeError
>>> *Exception.
>>>
>>> I have opened an issue on Github.(https://github.com/qub
>>> ole/sparklens/issues/1)
>>>
>>> Shmuel
>>>
>>> On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz <
>>> shmuel.bl...@similarweb.com> wrote:
>>>
 Thanks.

 We will give this a try and report back.

 Shmuel

 On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia 
 wrote:

> Thanks everyone!
> Please share how it works and how it doesn't. Both help.
>
> Fawaze, just made few changes to make this work with spark 1.6. Can
> you please try building from branch *spark_1.6*
>
> thanks,
> rohitk
>
>
>
> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber 
> wrote:
>
>> It's super amazing  i see it was tested on spark 2.0.0 and above,
>> what about Spark 1.6 which is still part of Cloudera's main versions?
>>
>> We have a vast Spark applications with version 1.6.0
>>
>> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau 
>> wrote:
>>
>>> Super exciting! I look forward to digging through it this weekend.
>>>
>>> On Wed, Mar 21, 2018 at 9:33 PM ☼ R Nair (रविशंकर नायर) <
>>> ravishankar.n...@gmail.com> wrote:
>>>
 Excellent. You filled a missing link.

 Best,
 Passion

 On Wed, Mar 21, 2018 at 11:36 PM, Rohit Karlupia >>> > wrote:

> Hi,
>
> Happy to announce the availability of Sparklens as open source
> project. It helps in understanding the  scalability limits of spark
> applications and can be a useful guide on the path towards tuning
> applications for lower runtime or cost.
>
> Please clone from here: https://github.com/qubole/sparklens
> Old blogpost: https://www.qubole.com/blog/introducing-quboles-sp
> ark-tuning-tool/
>
> thanks,
> rohitk
>
> PS: Thanks for the patience. It took couple of months to get back
> on this.
>
>
>
>
>
 --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
>>
>


 --
 Shmuel Blitz
 Big Data Developer
 Email: shmuel.bl...@similarweb.com
 www.similarweb.com
 
 
 

>>>
>>>
>>>
>>> --
>>> Shmuel Blitz
>>> Big Data Developer
>>> Email: shmuel.bl...@similarweb.com
>>> www.similarweb.com
>>> 
>>> 
>>> 
>>>
>>
>>
>


-- 
Shmuel Blitz
Big Data Developer
Email: shmuel.bl...@similarweb.com
www.similarweb.com