Parts of it are indeed written in Java. You probably want to reach out to
the developers list to talk about changing Spark.
On Sat, Nov 3, 2018, 11:42 AM Soheil Pourbafrani Hi, I want to customize some part of Spark. I was wondering if there any
> Spark source is written in Java language, or all
Thanks,
I don't need sources for learning Spark.
I need some sources in Java language that I can implement new functions.
For example, Spark has HashTF and I want to customize that in such a way:
public static class newHashingTF implements Something
On Sat, Nov 3, 2018 at 10:39 PM Chris Olivier
that’s a great link. thanks!
On Sat, Nov 3, 2018 at 11:55 AM Jean Georges Perrin wrote:
> I would take this one very closely to my heart :)
>
> Look at:
> https://github.com/jgperrin/net.jgp.labs.spark
>
> And if the examples are too weird, have a look at:
> http://jgp.net/book published at
I would take this one very closely to my heart :)
Look at:
https://github.com/jgperrin/net.jgp.labs.spark
And if the examples are too weird, have a look at:
http://jgp.net/book published at Manning
Feedback appreciated!
jg
> On Nov 3, 2018, at 12:30, Jeyhun Karimov wrote:
>
> Hi Soheil,
Hi Soheil,
>From the spark github repo, you can find some classes implemented in Java:
https://github.com/apache/spark/search?l=java
Cheers,
Jeyhun
On Sat, Nov 3, 2018 at 6:42 PM Soheil Pourbafrani
wrote:
> Hi, I want to customize some part of Spark. I was wondering if there any
> Spark
Hi, I want to customize some part of Spark. I was wondering if there any
Spark source is written in Java language, or all the sources are in Scala
language?
You could have used two separate pools with different weights for ETL and rest
jobs, when ETL pool weights is about 1 and Rest weight is 1000, anytime a Rest
Job comes in, it allocate all the resources. Details:
https://spark.apache.org/docs/latest/job-scheduling.html Sent using Zoho Mail
Hi,
What does your Spark deployment architecture looks like? Standalone? Yarn?
Mesos? Kubernetes? Those have resource managers (not middlewares) that allow to
implement scenarios as you want to achieve.
In any case you can try the FairScheduler of any of those solutions.
Best regards
> Am
On Sat, Nov 03, 2018 at 02:04:01AM -0700, conner wrote:
> My solution is to find a good way to divide the spark cluster resource
> into two.
What about yarn and its queue management system ?
--
nicolas
-
To unsubscribe
Hi,
I use spark cluster to run ETL jobs and analysis computation about the data
after elt stage.
The elt jobs can keep running for several hours, but analysis computation is
a short-running job which can finish in a few seconds.
The dilemma I entrapped is that my application runs in a single JVM
10 matches
Mail list logo
