Re: [k8s] Spark operator (the Java one)

[Yinan Li]

Hi Erik,

I agree with what you said about the community benefits of supporting
operators for Spark. However, that doesn't necessarily mean the operators
need and should be part of Spark core.

> *) The Kubernetes ecosystem is evolving toward adopting operators as the
de facto standard for deploying and manipulating software resources on a
kube cluster. Supporting an out-of-the-box operator will increase the
attractiveness of Spark for users and stakeholders in the Kubernetes
ecosystem and maximize future uptake; it will continue to keep the barrier
to entry low for Spark on Kubernetes.

Making an operator be part of Spark core is not necessarily a prerequisite
or condition for building strong community support for it. There are lots
of other third-party projects around Spark that have strong community
support behind them. Kubernetes itself is moving towards a model that more
and more functionalities are built out of core through CRDs, for good
reasons. There are a lot of benefits with this model. We can still build a
strong community for the existing operators for Spark, while using separate
build/test setups, release schedules, and enhancement planning.

> *) It represents an additional channel for exposing kube-specific
features, that might otherwise need to be plumbed through spark-submit or
the k8s backend.

It's applicable regardless of if the operators are maintained as part of
Spark core or not, with the maturity of Kubernetes features around CRD
support and webhooks. The GCP Spark operator supports a lot of additional
pod/container configs using a webhook, and this approach seems pretty
successful so far.


On Wed, Oct 16, 2019 at 7:10 AM Erik Erlandson  wrote:

>
> Folks have (correctly) pointed out that an operator does not need to be
> coupled to the Apache Spark project. However, I believe there are some
> strategic community benefits to supporting a Spark operator that should be
> weighed against the costs of maintaining one.
>
> *) The Kubernetes ecosystem is evolving toward adopting operators as the
> de facto standard for deploying and manipulating software resources on a
> kube cluster. Supporting an out-of-the-box operator will increase the
> attractiveness of Spark for users and stakeholders in the Kubernetes
> ecosystem and maximize future uptake; it will continue to keep the barrier
> to entry low for Spark on Kubernetes.
>
> *) An operator provides a unified and idiomatic kube front-end not just
> for spark job submissions, but also standalone spark clusters in the cloud,
> the spark history server and eventually the modernized shuffle service,
> when that is completed.
>
> *) It represents an additional channel for exposing kube-specific
> features, that might otherwise need to be plumbed through spark-submit or
> the k8s backend.
>
> Cheers,
> Erik
>
> On Thu, Oct 10, 2019 at 9:23 PM Yinan Li  wrote:
>
>> +1. This and the GCP Spark Operator, although being very useful for k8s
>> users, are not something needed by all Spark users, not even by all Spark
>> on k8s users.
>>
>>
>> On Thu, Oct 10, 2019 at 6:34 PM Stavros Kontopoulos <
>> stavros.kontopou...@lightbend.com> wrote:
>>
>>> Hi all,
>>>
>>> I also left a comment on the PR with more details. I dont see why the
>>> java operator should be maintained by the Spark project.
>>> This is an interesting project and could thrive on its own as an
>>> external operator project.
>>>
>>> Best,
>>> Stavros
>>>
>>> On Thu, Oct 10, 2019 at 7:51 PM Sean Owen  wrote:
>>>
>>>> I'd have the same question on the PR - why does this need to be in the
>>>> Apache Spark project vs where it is now? Yes, it's not a Spark package
>>>> per se, but it seems like this is a tool for K8S to use Spark rather
>>>> than a core Spark tool.
>>>>
>>>> Yes of course all the packages, licenses, etc have to be overhauled,
>>>> but that kind of underscores that this is a dump of a third party tool
>>>> that works fine on its own?
>>>>
>>>> On Thu, Oct 10, 2019 at 9:30 AM Jiri Kremser 
>>>> wrote:
>>>> >
>>>> > Hello,
>>>> >
>>>> >
>>>> > Spark Operator is a tool that can deploy/scale and help with
>>>> monitoring of Spark clusters on Kubernetes. It follows the operator pattern
>>>> [1] introduced by CoreOS so it watches for changes in custom resources
>>>> representing the desired state of the clusters and does the steps to
>>>> achieve this state in the Kubernetes by using the K8s client. It’s written
>>>> in Java and there is an overlap with the spark depe

Re: [k8s] Spark operator (the Java one)

[Yinan Li]

+1. This and the GCP Spark Operator, although being very useful for k8s
users, are not something needed by all Spark users, not even by all Spark
on k8s users.


On Thu, Oct 10, 2019 at 6:34 PM Stavros Kontopoulos <
stavros.kontopou...@lightbend.com> wrote:

> Hi all,
>
> I also left a comment on the PR with more details. I dont see why the java
> operator should be maintained by the Spark project.
> This is an interesting project and could thrive on its own as an external
> operator project.
>
> Best,
> Stavros
>
> On Thu, Oct 10, 2019 at 7:51 PM Sean Owen  wrote:
>
>> I'd have the same question on the PR - why does this need to be in the
>> Apache Spark project vs where it is now? Yes, it's not a Spark package
>> per se, but it seems like this is a tool for K8S to use Spark rather
>> than a core Spark tool.
>>
>> Yes of course all the packages, licenses, etc have to be overhauled,
>> but that kind of underscores that this is a dump of a third party tool
>> that works fine on its own?
>>
>> On Thu, Oct 10, 2019 at 9:30 AM Jiri Kremser  wrote:
>> >
>> > Hello,
>> >
>> >
>> > Spark Operator is a tool that can deploy/scale and help with monitoring
>> of Spark clusters on Kubernetes. It follows the operator pattern [1]
>> introduced by CoreOS so it watches for changes in custom resources
>> representing the desired state of the clusters and does the steps to
>> achieve this state in the Kubernetes by using the K8s client. It’s written
>> in Java and there is an overlap with the spark dependencies (logging, k8s
>> client, apache-commons-*, fasterxml-jackson, etc.). The operator contains
>> also metadata that allows it to deploy smoothly using the operatorhub.io
>> [2]. For a very basic info, check the readme on the project page including
>> the gif :) Other unique feature to this operator is the ability (it’s
>> optional) to compile itself to a native image using GraalVM compiler to be
>> able to start fast and have a very low memory footprint.
>> >
>> >
>> > We would like to contribute this project to Spark’s code base. It can’t
>> be distributed as a spark package, because it’s not a library that can be
>> used from Spark environment. So if you are interested, the directory under
>> resource-managers/kubernetes/spark-operator/ could be a suitable
>> destination.
>> >
>> >
>> > The current repository is radanalytics/spark-operator [2] on GitHub and
>> it contains also a test suite [3] that verifies if the operator can work
>> well on K8s (using minikube) and also on OpenShift. I am not sure how to
>> transfer those tests in case you would be interested in those as well.
>> >
>> >
>> > I’ve already opened the PR [5], but it got closed, so I am opening the
>> discussion here first. The PR contained old package names with our
>> organisation called radanalytics.io but we are willing to change that to
>> anything that will be more aligned with the existing Spark conventions,
>> same holds for the license headers in all the source files.
>> >
>> >
>> > jk
>> >
>> >
>> >
>> > [1]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
>> >
>> > [2]: https://operatorhub.io/operator/radanalytics-spark
>> >
>> > [3]: https://github.com/radanalyticsio/spark-operator
>> >
>> > [4]: https://travis-ci.org/radanalyticsio/spark-operator
>> >
>> > [5]: https://github.com/apache/spark/pull/26075
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>

Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

[Yinan Li]

+1 (non-binding)

On Mon, Jun 17, 2019 at 1:58 PM Ryan Blue  wrote:

> +1 (non-binding)
>
> On Sun, Jun 16, 2019 at 11:11 PM Dongjoon Hyun 
> wrote:
>
>> +1
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Sun, Jun 16, 2019 at 9:41 PM Saisai Shao 
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Thanks
>>> Saisai
>>>
>>> Imran Rashid  于2019年6月15日周六 上午3:46写道：
>>>
 +1 (binding)

 I think this is a really important feature for spark.

 First, there is already a lot of interest in alternative shuffle
 storage in the community.  There is already a lot of interest in
 alternative shuffle storage, from dynamic allocation in kubernetes, to even
 just improving stability in standard on-premise use of Spark.  However,
 they're often stuck doing this in forks of Spark, and in ways that are not
 maintainable (because they copy-paste many spark internals) or are
 incorrect (for not correctly handling speculative execution & stage
 retries).

 Second, I think the specific proposal is good for finding the right
 balance between flexibility and too much complexity, to allow incremental
 improvements.  A lot of work has been put into this already to try to
 figure out which pieces are essential to make alternative shuffle storage
 implementations feasible.

 Of course, that means it doesn't include everything imaginable; some
 things still aren't supported, and some will still choose to use the older
 ShuffleManager api to give total control over all of shuffle.  But we know
 there are a reasonable set of things which can be implemented behind the
 api as the first step, and it can continue to evolve.

 On Fri, Jun 14, 2019 at 12:13 PM Ilan Filonenko 
 wrote:

> +1 (non-binding). This API is versatile and flexible enough to handle
> Bloomberg's internal use-cases. The ability for us to vary implementation
> strategies is quite appealing. It is also worth to note the minimal 
> changes
> to Spark core in order to make it work. This is a very much needed 
> addition
> within the Spark shuffle story.
>
> On Fri, Jun 14, 2019 at 9:59 AM bo yang  wrote:
>
>> +1 This is great work, allowing plugin of different sort shuffle
>> write/read implementation! Also great to see it retain the current Spark
>> configuration
>> (spark.shuffle.manager=org.apache.spark.shuffle.YourShuffleManagerImpl).
>>
>>
>> On Thu, Jun 13, 2019 at 2:58 PM Matt Cheah 
>> wrote:
>>
>>> Hi everyone,
>>>
>>>
>>>
>>> I would like to call a vote for the SPIP for SPARK-25299
>>> , which proposes
>>> to introduce a pluggable storage API for temporary shuffle data.
>>>
>>>
>>>
>>> You may find the SPIP document here
>>> 
>>> .
>>>
>>>
>>>
>>> The discussion thread for the SPIP was conducted here
>>> 
>>> .
>>>
>>>
>>>
>>> Please vote on whether or not this proposal is agreeable to you.
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>> -Matt Cheah
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

[Yinan Li]

+1

On Fri, Mar 1, 2019 at 12:37 PM Tom Graves 
wrote:

> +1 for the SPIP.
>
> Tom
>
> On Friday, March 1, 2019, 8:14:43 AM CST, Xingbo Jiang <
> jiangxb1...@gmail.com> wrote:
>
>
> Hi all,
>
> I want to call for a vote of SPARK-24615
> . It improves Spark by
> making it aware of GPUs exposed by cluster managers, and hence Spark can
> match GPU resources with user task requests properly. The proposal
> 
>  and production doc
> 
>  was
> made available on dev@ to collect input. Your can also find a design
> sketch at SPARK-27005 .
>
> The vote will be up for the next 72 hours. Please reply with your vote:
>
> +1: Yeah, let's go forward and implement the SPIP.
> +0: Don't really care.
> -1: I don't think this is a good idea because of the following technical
> reasons.
>
> Thank you!
>
> Xingbo
>

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

[Yinan Li]

Yep, the Kerberos support for k8s is in the master but not in branch-2.4. I
see no reason to get the integration tests into 2.4, which depend on the
feature in the master.

On Tue, Oct 16, 2018 at 9:32 AM Rob Vesse  wrote:

> Right now the Kerberos support for Spark on K8S is only on master AFAICT
> i.e. the feature is not present on branch-2.4
>
>
>
> Therefore I don’t see any point in adding the tests into branch-2.4 unless
> the plan is to also merge the Kerberos support to branch-2.4
>
>
>
> Rob
>
>
>
> *From: *Erik Erlandson 
> *Date: *Tuesday, 16 October 2018 at 16:47
> *To: *dev 
> *Subject: *[DISCUSS][K8S][TESTS] Include Kerberos integration tests for
> Spark 2.4
>
>
>
> I'd like to propose including integration testing for Kerberos on the
> Spark 2.4 release:
>
> https://github.com/apache/spark/pull/22608
>
>
>
> Arguments in favor:
>
> 1) it improves testing coverage on a feature important for integrating
> with HDFS deployments
>
> 2) its intersection with existing code is small - it consists primarily of
> new testing code, with a bit of refactoring into 'main' and 'test'
> sub-trees. These new tests appear stable.
>
> 3) Spark 2.4 is still in RC, with outstanding correctness issues.
>
>
>
> The argument 'against' that I'm aware of would be the relatively large
> size of the PR. I believe this is considered above, but am soliciting
> community feedback before committing.
>
> Cheers,
>
> Erik
>
>
>

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

[Yinan Li]

> You can do this manually yourself via kubectl cp so it should be possible
to programmatically do this since it looks like this is just a tar piped
into a kubectl exec.   This would keep the relevant logic in the Kubernetes
specific client which may/may not be desirable depending on whether we’re
looking to just fix this for K8S or more generally.  Of course there is
probably a fair bit of complexity in making this work but does that sound
like something worth exploring?

Yes, kubectl cp is able to copy files from your local machine into a
container in a pod. However, the pod must be up and running for this to
work. So if you want to use this to upload dependencies to the driver pod,
the driver pod must already be up and running. So you may not even have a
chance to upload the dependencies at this point.

On Mon, Oct 8, 2018 at 6:36 AM Rob Vesse  wrote:

> Folks, thanks for all the great input. Responding to various points raised:
>
>
>
> Marcelo/Yinan/Felix –
>
>
>
> Yes, client mode will work.  The main JAR will be automatically
> distributed and --jars/--files specified dependencies are also distributed
> though for --files user code needs to use the appropriate Spark APIs to
> resolve the actual path i.e. SparkFiles.get()
>
>
>
> However client mode can be awkward if you want to mix spark-submit
> distribution with mounting dependencies via volumes since you may need to
> ensure that dependencies appear at the same path both on the local
> submission client and when mounted into the executors.  This mainly applies
> to the case where user code does not use SparkFiles.get() and simply tries
> to access the path directly.
>
>
>
> Marcelo/Stavros –
>
>
>
> Yes I did give the other resource managers too much credit.  From my past
> experience with Mesos and Standalone I had thought this wasn’t an issue but
> going back and looking at what we did for both of those it appears we were
> entirely reliant on the shared file system (whether HDFS, NFS or other
> POSIX compliant filesystems e.g. Lustre).
>
>
>
> Since connectivity back to the client is a potential stumbling block for
> cluster mode I wander if it would be better to think in reverse i.e. rather
> than having the driver pull from the client have the client push to the
> driver pod?
>
>
>
> You can do this manually yourself via kubectl cp so it should be possible
> to programmatically do this since it looks like this is just a tar piped
> into a kubectl exec.   This would keep the relevant logic in the Kubernetes
> specific client which may/may not be desirable depending on whether we’re
> looking to just fix this for K8S or more generally.  Of course there is
> probably a fair bit of complexity in making this work but does that sound
> like something worth exploring?
>
>
>
> I hadn’t really considered the HA aspect, a first step would be to get the
> basics working and then look at the HA aspect.  Although if the above
> theoretical approach is practical that could simply be part of restarting
> the driver.
>
>
>
> Rob
>
>
>
>
>
> *From: *Felix Cheung 
> *Date: *Sunday, 7 October 2018 at 23:00
> *To: *Yinan Li , Stavros Kontopoulos <
> stavros.kontopou...@lightbend.com>
> *Cc: *Rob Vesse , dev 
> *Subject: *Re: [DISCUSS][K8S] Local dependencies with Kubernetes
>
>
>
> Jars and libraries only accessible locally at the driver is fairly
> limited? Don’t you want the same on all executor?
>
>
>
>
>
>
> --
>
> *From:* Yinan Li 
> *Sent:* Friday, October 5, 2018 11:25 AM
> *To:* Stavros Kontopoulos
> *Cc:* rve...@dotnetrdf.org; dev
> *Subject:* Re: [DISCUSS][K8S] Local dependencies with Kubernetes
>
>
>
> > Just to be clear: in client mode things work right? (Although I'm not
> really familiar with how client mode works in k8s - never tried it.)
>
>
>
> If the driver runs on the submission client machine, yes, it should just
> work. If the driver runs in a pod, however, it faces the same problem as in
> cluster mode.
>
>
>
> Yinan
>
>
>
> On Fri, Oct 5, 2018 at 11:06 AM Stavros Kontopoulos <
> stavros.kontopou...@lightbend.com> wrote:
>
> @Marcelo is correct. Mesos does not have something similar. Only Yarn does
> due to the distributed cache thing.
>
> I have described most of the above in the the jira also there are some
> other options.
>
>
>
> Best,
>
> Stavros
>
>
>
> On Fri, Oct 5, 2018 at 8:28 PM, Marcelo Vanzin <
> van...@cloudera.com.invalid> wrote:
>
> On Fri, Oct 5, 2018 at 7:54 AM Rob Vesse  wrote:
> > Ideally this would all just be handled automatically for users in the
> way that all other resou

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

[Yinan Li]

> Just to be clear: in client mode things work right? (Although I'm not
really familiar with how client mode works in k8s - never tried it.)

If the driver runs on the submission client machine, yes, it should just
work. If the driver runs in a pod, however, it faces the same problem as in
cluster mode.

Yinan

On Fri, Oct 5, 2018 at 11:06 AM Stavros Kontopoulos <
stavros.kontopou...@lightbend.com> wrote:

> @Marcelo is correct. Mesos does not have something similar. Only Yarn does
> due to the distributed cache thing.
> I have described most of the above in the the jira also there are some
> other options.
>
> Best,
> Stavros
>
> On Fri, Oct 5, 2018 at 8:28 PM, Marcelo Vanzin <
> van...@cloudera.com.invalid> wrote:
>
>> On Fri, Oct 5, 2018 at 7:54 AM Rob Vesse  wrote:
>> > Ideally this would all just be handled automatically for users in the
>> way that all other resource managers do
>>
>> I think you're giving other resource managers too much credit. In
>> cluster mode, only YARN really distributes local dependencies, because
>> YARN has that feature (its distributed cache) and Spark just uses it.
>>
>> Standalone doesn't do it (see SPARK-4160) and I don't remember seeing
>> anything similar on the Mesos side.
>>
>> There are things that could be done; e.g. if you have HDFS you could
>> do a restricted version of what YARN does (upload files to HDFS, and
>> change the "spark.jars" and "spark.files" URLs to point to HDFS
>> instead). Or you could turn the submission client into a file server
>> that the cluster-mode driver downloads files from - although that
>> requires connectivity from the driver back to the client.
>>
>> Neither is great, but better than not having that feature.
>>
>> Just to be clear: in client mode things work right? (Although I'm not
>> really familiar with how client mode works in k8s - never tried it.)
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>
>
>

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

[Yinan Li]

Agreed with Marcelo that this is not a unique problem to Spark on k8s. For
a lot of organizations, hosting dependencies on HDFS seems the choice. One
option that the Spark Operator

does is to automatically upload application dependencies on the submission
client machine to a user-specified S3 or GCS bucket and substitute the
local dependencies with the remote ones. But regardless of which option to
use to stage local dependencies, it generally only works for small ones
like jars or small config/data files.

Yinan

On Fri, Oct 5, 2018 at 10:28 AM Marcelo Vanzin 
wrote:

> On Fri, Oct 5, 2018 at 7:54 AM Rob Vesse  wrote:
> > Ideally this would all just be handled automatically for users in the
> way that all other resource managers do
>
> I think you're giving other resource managers too much credit. In
> cluster mode, only YARN really distributes local dependencies, because
> YARN has that feature (its distributed cache) and Spark just uses it.
>
> Standalone doesn't do it (see SPARK-4160) and I don't remember seeing
> anything similar on the Mesos side.
>
> There are things that could be done; e.g. if you have HDFS you could
> do a restricted version of what YARN does (upload files to HDFS, and
> change the "spark.jars" and "spark.files" URLs to point to HDFS
> instead). Or you could turn the submission client into a file server
> that the cluster-mode driver downloads files from - although that
> requires connectivity from the driver back to the client.
>
> Neither is great, but better than not having that feature.
>
> Just to be clear: in client mode things work right? (Although I'm not
> really familiar with how client mode works in k8s - never tried it.)
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Python kubernetes spark 2.4 branch

[Yinan Li]

Can you give more details on how you ran your app, did you build your own
image, and which image are you using?

On Tue, Sep 25, 2018 at 10:23 AM Garlapati, Suryanarayana (Nokia -
IN/Bangalore)  wrote:

> Hi,
>
> I am trying to run spark python testcases on k8s based on tag
> spark-2.4-rc1. When the dependent files are passed through the --py-files
> option, they are not getting resolved by the main python script. Please let
> me know, is this a known issue?
>
>
>
> Regards
>
> Surya
>
>
>

Re: [DISCUSS][K8S] Supporting advanced pod customisation

[Yinan Li]

Thanks for bring this up. My opinion on this is this feature is really
targeting advanced use cases that need more customization than what the
basic k8s-related Spark config properties offer. So I think it's fair to
assume that users who would like to use this feature know the risks and are
responsible for making sure the supplied pod template is valid. Clear
documentation on which fields of the pod spec are customizable and which
are not is critical. With that being said, however, I agree that it's not a
good experience for end users if the pod spec generated from a template
plus overlay of customization from other Spark config properties and the
k8s backend itself is invalid. Ideally, I think option #3 is favorable and
the k8s backend code or the code for building the pod spec specifically
should perform validation on the generated pod spec, e.g., json schema
validation at bare minimum. I'm not sure how well this is supported by the
client (farbric8 java client) we use though. Regarding option #2, I
personally think the risks and complexity it brings far outweigh the
benefits and it is much more prone to errors. For example, the k8s backend
code sets some environment variables in the container that get used by the
entrypoint script. If the backend skips the building of the pod spec, the
end users are responsible for properly populating them and maybe other
stuffs. So my opinion is I think initially #1 plus good documentation is
the way to go, and we can improve on this in subsequent releases to move it
towards #3.

Yinan

On Wed, Sep 19, 2018 at 9:12 AM Rob Vesse  wrote:

> Hey all
>
>
>
> For those following the K8S backend you are probably aware of SPARK-24434
> [1] (and PR 22416 [2]) which proposes a mechanism to allow for advanced pod
> customisation via pod templates.  This is motivated by the fact that
> introducing additional Spark configuration properties for each aspect of
> pod specification a user might wish to customise was becoming unwieldy.
>
>
>
> However I am concerned that the current implementation doesn’t go far
> enough and actually limits the utility of the proposed new feature.  The
> problem stems from the fact that the implementation simply uses the pod
> template as a base and then Spark attempts to build a pod spec on top of
> that.  As the code that does this doesn’t do any kind of validation or
> inspection of the incoming template it is possible to provide a template
> that causes Spark to generate an invalid pod spec ultimately causing the
> job to be rejected by Kubernetes.
>
>
>
> Now clearly Spark code cannot attempt to account for every possible
> customisation that a user may attempt to make via pod templates nor should
> it be responsible for ensuring that the user doesn’t start from an invalid
> template in the first place.  However it seems like we could be more
> intelligent in how we build our pod specs to avoid generating invalid specs
> in cases where we have a clear use case for advanced customisation.  For
> example the current implementation does not allow users to customise the
> volumes used to back SPARK_LOCAL_DIRS to better suit the compute
> environment the K8S cluster is running on and trying to do so with a pod
> template will result in an invalid spec due to duplicate volumes.
>
>
>
> I think there are a few ways the community could address this:
>
>
>
>1. Status quo – provide the pod template feature as-is and simply tell
>users that certain customisations are never supported and may result in
>invalid pod specs
>2. Provide the ability for advanced users to explicitly skip pod spec
>building steps they know interfere with their pod templates via
>configuration properties
>3. Modify the pod spec building code to be aware of known desirable
>user customisation points and avoid generating  invalid specs in those 
> cases
>
>
>
> Currently committers seem to be going for Option 1.  Personally I would
> like to see the community adopt option 3 but have already received
> considerable pushback when I proposed that in one of my PRs hence the
> suggestion of the compromise option 2.  Yes this still has the possibility
> of ending up with invalid specs if users are over-zealous in the spec
> building steps they disable but since this is a power user feature I think
> this would be a risk power users would be willing to assume.  If we are
> going to provide features for power users we should avoid unnecessarily
> limiting the utility of those features.
>
>
>
> What do other K8S folks think about this issue?
>
>
>
> Thanks,
>
>
>
> Rob
>
>
>
> [1] https://issues.apache.org/jira/browse/SPARK-24434
>
> [2] https://github.com/apache/spark/pull/22146
>
>
>

Re: [VOTE] SPARK 2.4.0 (RC1)

[Yinan Li]

FYI: SPARK-23200 has been resolved.

On Tue, Sep 18, 2018 at 8:49 AM Felix Cheung 
wrote:

> If we could work on this quickly - it might get on to future RCs.
>
>
>
> --
> *From:* Stavros Kontopoulos 
> *Sent:* Monday, September 17, 2018 2:35 PM
> *To:* Yinan Li
> *Cc:* Xiao Li; eerla...@redhat.com; van...@cloudera.com.invalid; Sean
> Owen; Wenchen Fan; dev
> *Subject:* Re: [VOTE] SPARK 2.4.0 (RC1)
>
> Hi Xiao,
>
> I just tested it, it seems ok. There are some questions about which
> properties we should keep when restoring the config. Otherwise it looks ok
> to me.
> The reason this should go in 2.4 is that streaming on k8s is something
> people want to try day one (or at least it is cool to try) and since 2.4
> comes with k8s support being refactored a lot,
> it would be disappointing not to have it in...IMHO.
>
> Best,
> Stavros
>
> On Mon, Sep 17, 2018 at 11:13 PM, Yinan Li  wrote:
>
>> We can merge the PR and get SPARK-23200 resolved if the whole point is to
>> make streaming on k8s work first. But given that this is not a blocker for
>> 2.4, I think we can take a bit more time here and get it right. With that
>> being said, I would expect it to be resolved soon.
>>
>> On Mon, Sep 17, 2018 at 11:47 AM Xiao Li  wrote:
>>
>>> Hi, Erik and Stavros,
>>>
>>> This bug fix SPARK-23200 is not a blocker of the 2.4 release. It sounds
>>> important for the Streaming on K8S. Could the K8S oriented committers speed
>>> up the reviews?
>>>
>>> Thanks,
>>>
>>> Xiao
>>>
>>> Erik Erlandson  于2018年9月17日周一 上午11:04写道：
>>>
>>>>
>>>> I have no binding vote but I second Stavros’ recommendation for
>>>> spark-23200
>>>>
>>>> Per parallel threads on Py2 support I would also like to propose
>>>> deprecating Py2 starting with this 2.4 release
>>>>
>>>> On Mon, Sep 17, 2018 at 10:38 AM Marcelo Vanzin
>>>>  wrote:
>>>>
>>>>> You can log in to https://repository.apache.org and see what's wrong.
>>>>> Just find that staging repo and look at the messages. In your case it
>>>>> seems related to your signature.
>>>>>
>>>>> failureMessageNo public key: Key with id: () was not able to be
>>>>> located on http://gpg-keyserver.de/. Upload your public key and try
>>>>> the operation again.
>>>>> On Sun, Sep 16, 2018 at 10:00 PM Wenchen Fan 
>>>>> wrote:
>>>>> >
>>>>> > I confirmed that
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285
>>>>> is not accessible. I did it via ./dev/create-release/do-release-docker.sh
>>>>> -d /my/work/dir -s publish , not sure what's going wrong. I didn't see any
>>>>> error message during it.
>>>>> >
>>>>> > Any insights are appreciated! So that I can fix it in the next RC.
>>>>> Thanks!
>>>>> >
>>>>> > On Mon, Sep 17, 2018 at 11:31 AM Sean Owen 
>>>>> wrote:
>>>>> >>
>>>>> >> I think one build is enough, but haven't thought it through. The
>>>>> >> Hadoop 2.6/2.7 builds are already nearly redundant. 2.12 is probably
>>>>> >> best advertised as a 'beta'. So maybe publish a no-hadoop build of
>>>>> it?
>>>>> >> Really, whatever's the easy thing to do.
>>>>> >> On Sun, Sep 16, 2018 at 10:28 PM Wenchen Fan 
>>>>> wrote:
>>>>> >> >
>>>>> >> > Ah I missed the Scala 2.12 build. Do you mean we should publish a
>>>>> Scala 2.12 build this time? Current for Scala 2.11 we have 3 builds: with
>>>>> hadoop 2.7, with hadoop 2.6, without hadoop. Shall we do the same thing 
>>>>> for
>>>>> Scala 2.12?
>>>>> >> >
>>>>> >> > On Mon, Sep 17, 2018 at 11:14 AM Sean Owen 
>>>>> wrote:
>>>>> >> >>
>>>>> >> >> A few preliminary notes:
>>>>> >> >>
>>>>> >> >> Wenchen for some weird reason when I hit your key in gpg
>>>>> --import, it
>>>>> >> >> asks for a passphrase. When I skip it, it's fine, gpg can still
>>>>> verify
>>>>> >> >> the signature. No issue there really.
>>>

Re: [VOTE] SPARK 2.4.0 (RC1)

[Yinan Li]

We can merge the PR and get SPARK-23200 resolved if the whole point is to
make streaming on k8s work first. But given that this is not a blocker for
2.4, I think we can take a bit more time here and get it right. With that
being said, I would expect it to be resolved soon.

On Mon, Sep 17, 2018 at 11:47 AM Xiao Li  wrote:

> Hi, Erik and Stavros,
>
> This bug fix SPARK-23200 is not a blocker of the 2.4 release. It sounds
> important for the Streaming on K8S. Could the K8S oriented committers speed
> up the reviews?
>
> Thanks,
>
> Xiao
>
> Erik Erlandson  于2018年9月17日周一 上午11:04写道：
>
>>
>> I have no binding vote but I second Stavros’ recommendation for
>> spark-23200
>>
>> Per parallel threads on Py2 support I would also like to propose
>> deprecating Py2 starting with this 2.4 release
>>
>> On Mon, Sep 17, 2018 at 10:38 AM Marcelo Vanzin
>>  wrote:
>>
>>> You can log in to https://repository.apache.org and see what's wrong.
>>> Just find that staging repo and look at the messages. In your case it
>>> seems related to your signature.
>>>
>>> failureMessageNo public key: Key with id: () was not able to be
>>> located on http://gpg-keyserver.de/. Upload your public key and try
>>> the operation again.
>>> On Sun, Sep 16, 2018 at 10:00 PM Wenchen Fan 
>>> wrote:
>>> >
>>> > I confirmed that
>>> https://repository.apache.org/content/repositories/orgapachespark-1285
>>> is not accessible. I did it via ./dev/create-release/do-release-docker.sh
>>> -d /my/work/dir -s publish , not sure what's going wrong. I didn't see any
>>> error message during it.
>>> >
>>> > Any insights are appreciated! So that I can fix it in the next RC.
>>> Thanks!
>>> >
>>> > On Mon, Sep 17, 2018 at 11:31 AM Sean Owen  wrote:
>>> >>
>>> >> I think one build is enough, but haven't thought it through. The
>>> >> Hadoop 2.6/2.7 builds are already nearly redundant. 2.12 is probably
>>> >> best advertised as a 'beta'. So maybe publish a no-hadoop build of it?
>>> >> Really, whatever's the easy thing to do.
>>> >> On Sun, Sep 16, 2018 at 10:28 PM Wenchen Fan 
>>> wrote:
>>> >> >
>>> >> > Ah I missed the Scala 2.12 build. Do you mean we should publish a
>>> Scala 2.12 build this time? Current for Scala 2.11 we have 3 builds: with
>>> hadoop 2.7, with hadoop 2.6, without hadoop. Shall we do the same thing for
>>> Scala 2.12?
>>> >> >
>>> >> > On Mon, Sep 17, 2018 at 11:14 AM Sean Owen 
>>> wrote:
>>> >> >>
>>> >> >> A few preliminary notes:
>>> >> >>
>>> >> >> Wenchen for some weird reason when I hit your key in gpg --import,
>>> it
>>> >> >> asks for a passphrase. When I skip it, it's fine, gpg can still
>>> verify
>>> >> >> the signature. No issue there really.
>>> >> >>
>>> >> >> The staging repo gives a 404:
>>> >> >>
>>> https://repository.apache.org/content/repositories/orgapachespark-1285/
>>> >> >> 404 - Repository "orgapachespark-1285 (staging: open)"
>>> >> >> [id=orgapachespark-1285] exists but is not exposed.
>>> >> >>
>>> >> >> The (revamped) licenses are OK, though there are some minor
>>> glitches
>>> >> >> in the final release tarballs (my fault) : there's an extra
>>> directory,
>>> >> >> and the source release has both binary and source licenses. I'll
>>> fix
>>> >> >> that. Not strictly necessary to reject the release over those.
>>> >> >>
>>> >> >> Last, when I check the staging repo I'll get my answer, but, were
>>> you
>>> >> >> able to build 2.12 artifacts as well?
>>> >> >>
>>> >> >> On Sun, Sep 16, 2018 at 9:48 PM Wenchen Fan 
>>> wrote:
>>> >> >> >
>>> >> >> > Please vote on releasing the following candidate as Apache Spark
>>> version 2.4.0.
>>> >> >> >
>>> >> >> > The vote is open until September 20 PST and passes if a majority
>>> +1 PMC votes are cast, with
>>> >> >> > a minimum of 3 +1 votes.
>>> >> >> >
>>> >> >> > [ ] +1 Release this package as Apache Spark 2.4.0
>>> >> >> > [ ] -1 Do not release this package because ...
>>> >> >> >
>>> >> >> > To learn more about Apache Spark, please see
>>> http://spark.apache.org/
>>> >> >> >
>>> >> >> > The tag to be voted on is v2.4.0-rc1 (commit
>>> 1220ab8a0738b5f67dc522df5e3e77ffc83d207a):
>>> >> >> > https://github.com/apache/spark/tree/v2.4.0-rc1
>>> >> >> >
>>> >> >> > The release files, including signatures, digests, etc. can be
>>> found at:
>>> >> >> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-bin/
>>> >> >> >
>>> >> >> > Signatures used for Spark RCs can be found in this file:
>>> >> >> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >> >> >
>>> >> >> > The staging repository for this release can be found at:
>>> >> >> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1285/
>>> >> >> >
>>> >> >> > The documentation corresponding to this release can be found at:
>>> >> >> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-docs/
>>> >> >> >
>>> >> >> > The list of bug fixes going into 2.4.0 can be found at the
>>> following URL:
>>> >> >> > https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>> >> >> >

Spark on Kubernetes plan for 2.4

[Yinan Li]

On behalf of folks who work on Spark on Kubernetes, I would like to share a
doc

on the plan for Spark on Kubernetes features and changes for the upcoming
2.4 release. Please take a look if you are interested. Feedback and
comments are highly appreciated. Thanks!

Yinan

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

[Yinan Li]

Yes, the PR allows you to set say 1.5. The New configuration property
defaults to spark.executor.cores, which defaults to 1.

On Fri, Mar 30, 2018, 3:03 PM Kimoon Kim <kim...@pepperdata.com> wrote:

> David, glad it helped! And thanks for your clear example.
>
> > The only remaining question would then be what a sensible default for
> *spark.kubernetes.executor.cores *would be. Seeing that I wanted more
> than 1 and Yinan wants less, leaving it at 1 night be best.
>
> 1 as default SGTM.
>
> Thanks,
> Kimoon
>
> On Fri, Mar 30, 2018 at 1:38 PM, David Vogelbacher <
> dvogelbac...@palantir.com> wrote:
>
>> Thanks for linking that PR Kimoon.
>>
>>
>> It actually does mostly address the issue I was referring to. As the
>> issue <https://github.com/apache-spark-on-k8s/spark/issues/352> I linked
>> in my first email states, one physical cpu might not be enough to execute a
>> task in a performant way.
>>
>>
>>
>> So if I set *spark.executor.cores=1* and *spark.task.cpus=1* , I will
>> get 1 core from Kubernetes and execute one task per Executor and run into
>> performance problems.
>>
>> Being able to specify `spark.kubernetes.executor.cores=1.2` would fix the
>> issue (1.2 is just an example).
>>
>> I am curious as to why you, Yinan, would want to use this property to
>> request less than 1 physical cpu (that is how it sounds to me on the PR).
>>
>> Do you have testing that indicates that less than 1 physical CPU is
>> enough for executing tasks?
>>
>>
>>
>> In the end it boils down to the question proposed by Yinan:
>>
>> > A relevant question is should Spark on Kubernetes really be opinionated
>> on how to set the cpu request and limit and even try to determine this
>> automatically?
>>
>>
>>
>> And I completely agree with your answer Kimoon, we should provide
>> sensible defaults and make it configurable, as Yinan’s PR does.
>>
>> The only remaining question would then be what a sensible default for 
>> *spark.kubernetes.executor.cores
>> *would be. Seeing that I wanted more than 1 and Yinan wants less,
>> leaving it at 1 night be best.
>>
>>
>>
>> Thanks,
>>
>> David
>>
>>
>>
>> *From: *Kimoon Kim <kim...@pepperdata.com>
>> *Date: *Friday, March 30, 2018 at 4:28 PM
>> *To: *Yinan Li <liyinan...@gmail.com>
>> *Cc: *David Vogelbacher <dvogelbac...@palantir.com>, "
>> dev@spark.apache.org" <dev@spark.apache.org>
>> *Subject: *Re: [Kubernetes] Resource requests and limits for Driver and
>> Executor Pods
>>
>>
>>
>> I see. Good to learn the interaction between spark.task.cpus and
>> spark.executor.cores. But am I right to say that PR #20553 can be still
>> used as an additional knob on top of those two? Say a user wants 1.5 core
>> per executor from Kubernetes, not the rounded up integer value 2?
>>
>>
>>
>> > A relevant question is should Spark on Kubernetes really be opinionated
>> on how to set the cpu request and limit and even try to determine this
>> automatically?
>>
>>
>>
>> Personally, I don't see how this can be auto-determined at all. I think
>> the best we can do is to come up with sensible default values for the most
>> common case, and provide and well-document other knobs for edge cases.
>>
>>
>> Thanks,
>>
>> Kimoon
>>
>>
>>
>> On Fri, Mar 30, 2018 at 12:37 PM, Yinan Li <liyinan...@gmail.com> wrote:
>>
>> PR #20553 [github.com]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_pull_20553=DwMFaQ=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=BFXcJr3WTIvmlY-gtaiCO5QK4bLix2sgwDDpPfrZKoE=TrCA4oIVKyN3M_ExqpHr7bbhi14uvoEaspPwclIJI4M=jqIG5lO5tnV3K3SDPPxw2bEHs0i6cltoaLh8K39JTTQ=>
>>  is
>> more for allowing users to use a fractional value for cpu requests. The
>> existing spark.executor.cores is sufficient for specifying more than one
>> cpus.
>>
>>
>>
>> > One way to solve this could be to request more than 1 core from
>> Kubernetes per task. The exact amount we should request is unclear to me
>> (it largely depends on how many threads actually get spawned for a task).
>>
>> A good indication is spark.task.cpus, and on average how many tasks are
>> expected to run by a single executor at any point in time. If each executor
>> is only expected to run one task at most at any point in time,
>> spark.executor.cores can be set to be equal to spark.

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

[Yinan Li]

PR #20553  is more for allowing
users to use a fractional value for cpu requests. The existing
spark.executor.cores is sufficient for specifying more than one cpus.

> One way to solve this could be to request more than 1 core from
Kubernetes per task. The exact amount we should request is unclear to me
(it largely depends on how many threads actually get spawned for a task).

A good indication is spark.task.cpus, and on average how many tasks are
expected to run by a single executor at any point in time. If each executor
is only expected to run one task at most at any point in time,
spark.executor.cores can be set to be equal to spark.task.cpus.

A relevant question is should Spark on Kubernetes really be opinionated on
how to set the cpu request and limit and even try to determine this
automatically?

On Fri, Mar 30, 2018 at 11:40 AM, Kimoon Kim  wrote:

> > Instead of requesting `[driver,executor].memory`, we should just
> request `[driver,executor].memory + [driver,executor].memoryOverhead `. I
> think this case is a bit clearer than the CPU case, so I went ahead and
> filed an issue  with
> more details and made a PR .
>
> I think this suggestion makes sense.
>
> > One way to solve this could be to request more than 1 core from
> Kubernetes per task. The exact amount we should request is unclear to me
> (it largely depends on how many threads actually get spawned for a task).
>
> I wonder if this is being addressed by PR #20553
>  written by Yinan. Yinan?
>
> Thanks,
> Kimoon
>
> On Thu, Mar 29, 2018 at 5:14 PM, David Vogelbacher <
> dvogelbac...@palantir.com> wrote:
>
>> Hi,
>>
>>
>>
>> At the moment driver and executor pods are created using the following
>> requests and limits:
>>
>>
>>
>> *CPU*
>>
>> *Memory*
>>
>> *Request*
>>
>> [driver,executor].cores
>>
>> [driver,executor].memory
>>
>> *Limit*
>>
>> Unlimited (but can be specified using spark.[driver,executor].cores)
>>
>> [driver,executor].memory + [driver,executor].memoryOverhead
>>
>>
>>
>> Specifying the requests like this leads to problems if the pods only get
>> the requested amount of resources and nothing of the optional (limit)
>> resources, as it can happen in a fully utilized cluster.
>>
>>
>>
>> *For memory:*
>>
>> Let’s say we have a node with 100GiB memory and 5 pods with 20 GiB memory
>> and 5 GiB memoryOverhead.
>>
>> At the beginning all 5 pods use 20 GiB of memory and all is well. If a
>> pod then starts using its overhead memory it will get killed as there is no
>> more memory available, even though we told spark
>>
>> that it can use 25 GiB of memory.
>>
>>
>>
>> Instead of requesting `[driver,executor].memory`, we should just request
>> `[driver,executor].memory + [driver,executor].memoryOverhead `.
>>
>> I think this case is a bit clearer than the CPU case, so I went ahead and
>> filed an issue  with
>> more details and made a PR .
>>
>>
>>
>> *For CPU:*
>>
>> As it turns out, there can be performance problems if we only have
>> `executor.cores` available (which means we have one core per task). This
>> was raised here 
>> and is the reason that the cpu limit was set to unlimited.
>>
>> This issue stems from the fact that in general there will be more than
>> one thread per task, resulting in performance impacts if there is only one
>> core available.
>>
>> However, I am not sure that just setting the limit to unlimited is the
>> best solution because it means that even if the Kubernetes cluster can
>> perfectly satisfy the resource requests, performance might be very bad.
>>
>>
>>
>> I think we should guarantee that an executor is able to do its work well
>> (without performance issues or getting killed - as could happen in the
>> memory case) with the resources it gets guaranteed from Kubernetes.
>>
>>
>>
>> One way to solve this could be to request more than 1 core from
>> Kubernetes per task. The exact amount we should request is unclear to me
>> (it largely depends on how many threads actually get spawned for a task).
>>
>> We would need to find a way to determine this somehow automatically or at
>> least come up with a better default value than 1 core per task.
>>
>>
>>
>> Does somebody have ideas or thoughts on how to solve this best?
>>
>>
>>
>> Best,
>>
>> David
>>
>
>

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

[Yinan Li]

Hi David,

Regarding cpu limit, in Spark 2.3, we do have the following config
properties to specify cpu limit for the driver and executors. See
http://spark.apache.org/docs/latest/running-on-kubernetes.html.

spark.kubernetes.driver.limit.cores
spark.kubernetes.executor.limit.cores

On Thu, Mar 29, 2018 at 5:14 PM, David Vogelbacher <
dvogelbac...@palantir.com> wrote:

> Hi,
>
>
>
> At the moment driver and executor pods are created using the following
> requests and limits:
>
>
>
> *CPU*
>
> *Memory*
>
> *Request*
>
> [driver,executor].cores
>
> [driver,executor].memory
>
> *Limit*
>
> Unlimited (but can be specified using spark.[driver,executor].cores)
>
> [driver,executor].memory + [driver,executor].memoryOverhead
>
>
>
> Specifying the requests like this leads to problems if the pods only get
> the requested amount of resources and nothing of the optional (limit)
> resources, as it can happen in a fully utilized cluster.
>
>
>
> *For memory:*
>
> Let’s say we have a node with 100GiB memory and 5 pods with 20 GiB memory
> and 5 GiB memoryOverhead.
>
> At the beginning all 5 pods use 20 GiB of memory and all is well. If a pod
> then starts using its overhead memory it will get killed as there is no
> more memory available, even though we told spark
>
> that it can use 25 GiB of memory.
>
>
>
> Instead of requesting `[driver,executor].memory`, we should just request
> `[driver,executor].memory + [driver,executor].memoryOverhead `.
>
> I think this case is a bit clearer than the CPU case, so I went ahead and
> filed an issue  with
> more details and made a PR .
>
>
>
> *For CPU:*
>
> As it turns out, there can be performance problems if we only have
> `executor.cores` available (which means we have one core per task). This
> was raised here 
> and is the reason that the cpu limit was set to unlimited.
>
> This issue stems from the fact that in general there will be more than one
> thread per task, resulting in performance impacts if there is only one core
> available.
>
> However, I am not sure that just setting the limit to unlimited is the
> best solution because it means that even if the Kubernetes cluster can
> perfectly satisfy the resource requests, performance might be very bad.
>
>
>
> I think we should guarantee that an executor is able to do its work well
> (without performance issues or getting killed - as could happen in the
> memory case) with the resources it gets guaranteed from Kubernetes.
>
>
>
> One way to solve this could be to request more than 1 core from Kubernetes
> per task. The exact amount we should request is unclear to me (it largely
> depends on how many threads actually get spawned for a task).
>
> We would need to find a way to determine this somehow automatically or at
> least come up with a better default value than 1 core per task.
>
>
>
> Does somebody have ideas or thoughts on how to solve this best?
>
>
>
> Best,
>
> David
>

Re: Build issues with apache-spark-on-k8s.

[Yinan Li]

For 2.3, the dockerfile is under kubernetes/ in the tarball, not under the
directory where you started the build. Once you successfully build, copy
the tarball out, untar it, and you should see the directory kubernetes/ in
it.

On Thu, Mar 29, 2018 at 3:00 AM, Atul Sowani  wrote:

> Thanks all for responding and helping me with the build issue. I tried
> building the code at git://github.com/apache/spark.git (master branch) in
> my ppc64le Ubuntu 16.04 VM and it failed. I tried building a specific
> branch (branch-2.2) using following command:
>
> build/mvn -DskipTests -Pkubernetes clean package install
>
> This builds it successfully, but again I do not see "dockerfiles" and
> "jars" directories anywhere. This behaviour is exactly same as observed
> with source code at https://github.com/apache-spark-on-k8s/spark
>
> Any advise on how to proceed on this? As far as possible, I need to build
> v2.2.
>
> Thanks,
> Atul.
>
>
>
> On Wed, Mar 28, 2018 at 8:06 PM, Anirudh Ramanathan <
> ramanath...@google.com> wrote:
>
>> As Lucas said, those directories are generated and copied when you run a
>> full maven build with the -Pkubernetes flag specified (or use instructions
>> in  https://spark.apache.org/docs/latest/building-spark.html
>> #building-a-runnable-distribution).
>>
>> Also, using the Kubernetes integration in the  main Apache Spark project
>> is recommended. The fork https://github.com/apache-spark-on-k8s/spark/
>> will be retired once we finish upstreaming all those features in Spark 2.4.
>>
>>
>> On Wed, Mar 28, 2018, 6:42 AM Lucas Kacher  wrote:
>>
>>> Are you building on the fork or on the official release now? I built
>>> v2.3.0 from source w/out issue. One thing I noticed is that I needed to run
>>> the build-image command from the bin which was placed in dist/ as opposed
>>> to the one in the repo (as that's how it copies the necessary targets).
>>>
>>> (Failed to reply-all to the list).
>>>
>>> On Wed, Mar 28, 2018 at 4:30 AM, Atul Sowani  wrote:
>>>
 Hi,

 I built apache-spark-on-k8s from source on Ubuntu 16.04 and it got
 built without errors. Next, I wanted to create docker images, so as
 explained at https://apache-spark-on-k8s.github.io/userdocs/running-
 on-kubernetes.html I used sbin/build-push-docker-images.sh to create
 those. While using this script I came across 2 issues:

 1. It references "dockerfiles" directory which should be in "spark",
 however this directory is missing. I created "dockerfiles" directory and
 copied Dockerfiles from resource-managers/kuberne
 tes/docker-minimal-bundle

 2, spark-base dockerfile expects to have some JAR files present in a
 directory called "jars" - this directory is missing. I tried rebuilding the
 code but this directory is not getting generated if it is supposed to be.

 My doubt is, if this is a genuine/known issue or am I missing out some
 build steps?

 Thanks,
 Atul.


>>>
>>>
>>> --
>>>
>>> *Lucas Kacher*Senior Engineer
>>> -
>>> vsco.co 
>>> New York, NY
>>> 818.512.5239 <(818)%20512-5239>
>>>
>>
>

Re: Kubernetes: why use init containers?

[Yinan Li]

> Sorry, but what are those again? So far all the benefits are already
> provided by spark-submit...

1. Retries of init-containers are automatically supported by k8s through
pod restart policies. For this point, sorry I'm not sure how spark-submit
achieves this.
2. The ability to use credentials that are not shared with the main
containers.
3. Not only the user code, but Spark internal code like Executor won't be
run if the init-container fails.
4. Easier to build tooling around k8s events/status of the init-container
in case of failures as it's doing exactly one thing: downloading
dependencies.

There could be others that I'm not aware of.



On Wed, Jan 10, 2018 at 2:21 PM, Marcelo Vanzin <van...@cloudera.com> wrote:

> On Wed, Jan 10, 2018 at 2:16 PM, Yinan Li <liyinan...@gmail.com> wrote:
> > but we can not rule out the benefits init-containers bring either.
>
> Sorry, but what are those again? So far all the benefits are already
> provided by spark-submit...
>
> > Again, I would suggest we look at this more thoroughly post 2.3.
>
> Actually, one of the reasons why I brought this up is that we should
> remove init containers from 2.3 unless they're really required for
> something.
>
> Simplifying the code is not the only issue. The init container support
> introduces a whole lot of user-visible behavior - like config options
> and the execution of a completely separate container that the user can
> customize. If removed later, that could be considered a breaking
> change.
>
> So if we ship 2.3 without init containers and add them later if
> needed, it's a much better world than flipping that around.
>
> --
> Marcelo
>

Re: Kubernetes: why use init containers?

[Yinan Li]

> 1500 less lines of code trump all of the arguments given so far for
> what the init container might be a good idea.

We can also reduce the #lines of code by simply refactoring the code in
such as way that a lot of code can be shared between configuration of the
main container and that of the ini-container. Actually we have been
discussing this as one of the things to do right after the 2.3 release and
we do have a Jira ticket to track it. It's probably true that none of the
arguments we made are convincing enough, but we can not rule out the
benefits init-containers bring either.

Again, I would suggest we look at this more thoroughly post 2.3.

On Wed, Jan 10, 2018 at 2:06 PM, Marcelo Vanzin <van...@cloudera.com> wrote:

> On Wed, Jan 10, 2018 at 2:00 PM, Yinan Li <liyinan...@gmail.com> wrote:
> > I want to re-iterate on one point, that the init-container achieves a
> clear
> > separation between preparing an application and actually running the
> > application. It's a guarantee provided by the K8s admission control and
> > scheduling components that if the init-container fails, the main
> container
> > won't be run. I think this is definitely positive to have. In the case
> of a
> > Spark application, the application code and driver/executor code won't
> even
> > be run if the init-container fails to localize any of the dependencies
>
> That is also the case with spark-submit... (can't download
> dependencies -> spark-submit fails before running user code).
>
> > Note that we are not blindly opposing getting rid of the init-container,
> > it's just that there's still valid reasons to keep it for now
>
> I'll flip that around: I'm not against having an init container if
> it's serving a needed purpose, it's just that nobody is able to tell
> me what that needed purpose is.
>
> 1500 less lines of code trump all of the arguments given so far for
> what the init container might be a good idea.
>
> --
> Marcelo
>

Re: Kubernetes: why use init containers?

[Yinan Li]

I want to re-iterate on one point, that the init-container achieves a clear
separation between preparing an application and actually running the
application. It's a guarantee provided by the K8s admission control and
scheduling components that if the init-container fails, the main container
won't be run. I think this is definitely positive to have. In the case of a
Spark application, the application code and driver/executor code won't even
be run if the init-container fails to localize any of the dependencies. The
result is that it's much easier for users to figure out what's wrong if
their applications fail to run: they can tell if the pods are initialized
or not and if not, simply check the status/logs of the init-container.
Another argument I want to make is we can easily make the init-container to
be able to exclusively use certain credentials for downloading dependencies
that are not appropriate to be visible in the main containers and therefore
should not be shared. This is not achievable using the Spark canonical way.
K8s has built-in support for dynamically injecting containers into pods
through the admission control process. One use case would be for cluster
operators to inject an init-container (e.g., through a admission webhook)
for downloading certain dependencies that require certain
access-restrictive credentials.

Note that we are not blindly opposing getting rid of the init-container,
it's just that there's still valid reasons to keep it for now, particularly
given that we don't have a solid around client mode yet. Also given that we
have been using it in our fork for over a year, we are definitely more
confident on the current way of handling remote dependencies as it's been
tested more thoroughly. Since getting rid of the init-container is such a
significant change, I would suggest that we defer making a decision on if
we should get rid of it to 2.4 so we have a more thorough understanding of
the pros and cons.

On Wed, Jan 10, 2018 at 1:48 PM, Marcelo Vanzin  wrote:

> On Wed, Jan 10, 2018 at 1:47 PM, Matt Cheah  wrote:
> >> With a config value set by the submission code, like what I'm doing to
> prevent client mode submission in my p.o.c.?
> >
> > The contract for what determines the appropriate scheduler backend to
> instantiate is then going to be different in Kubernetes versus the other
> cluster managers.
>
> There is no contract for how to pick the appropriate scheduler. That's
> a decision that is completely internal to the cluster manager code
>
> --
> Marcelo
>

Re: Kubernetes: why use init containers?

[Yinan Li]

The init-container is required for use with the resource staging server (
https://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md#resource-staging-server).
The resource staging server (RSS) is a spark-on-k8s component running in a
Kubernetes cluster for staging submission client local dependencies to
Spark pods. The init-container is responsible for downloading the
dependencies from the RSS. We haven't upstream the RSS code yet, but this
is a value add component for Spark on K8s as a way for users to use
submission local dependencies without resorting to other mechanisms that
are not immediately available on most Kubernetes clusters, e.g., HDFS. We
do plan to upstream it in the 2.4 timeframe. Additionally, the
init-container is a Kubernetes native way of making sure that the
dependencies are localized before the main driver/executor containers are
started. IMO, this guarantee is positive to have and it helps achieve
separation of concerns. So IMO, I think the init-container is a valuable
component and should be kept.

On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas  wrote:

> I’d like to point out the output of “git show —stat” for that diff:
> 29 files changed, 130 insertions(+), 1560 deletions(-)
>
> +1 for that and generally for the idea of leveraging spark-submit.
>
> You can argue that executors downloading from
> external servers would be faster than downloading from the driver, but
> I’m not sure I’d agree - it can go both ways.
>
> On a tangentially related note, one of the main reasons spark-ec2
>  is so slow to launch clusters is
> that it distributes files like the Spark binaries to all the workers via
> the master. Because of that, the launch time scaled with the number of
> workers requested .
>
> When I wrote Flintrock , I got a
> large improvement in launch time over spark-ec2 simply by having all the
> workers download the installation files in parallel from an external host
> (typically S3 or an Apache mirror). And launch time became largely
> independent of the cluster size.
>
> That may or may not say anything about the driver distributing application
> files vs. having init containers do it in parallel, but I’d be curious to
> hear more.
>
> Nick
> ​
>
> On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan 
> 
> wrote:
>
>> We were running a change in our fork which was similar to this at one
>> point early on. My biggest concerns off the top of my head with this change
>> would be localization performance with large numbers of executors, and what
>> we lose in terms of separation of concerns. Init containers are a standard
>> construct in k8s for resource localization. Also how this approach affects
>> the HDFS work would be interesting.
>>
>> +matt +kimoon
>> Still thinking about the potential trade offs here. Adding Matt and
>> Kimoon who would remember more about our reasoning at the time.
>>
>>
>> On Jan 9, 2018 5:22 PM, "Marcelo Vanzin"  wrote:
>>
>>> Hello,
>>>
>>> Me again. I was playing some more with the kubernetes backend and the
>>> whole init container thing seemed unnecessary to me.
>>>
>>> Currently it's used to download remote jars and files, mount the
>>> volume into the driver / executor, and place those jars in the
>>> classpath / move the files to the working directory. This is all stuff
>>> that spark-submit already does without needing extra help.
>>>
>>> So I spent some time hacking stuff and removing the init container
>>> code, and launching the driver inside kubernetes using spark-submit
>>> (similar to how standalone and mesos cluster mode works):
>>>
>>> https://github.com/vanzin/spark/commit/k8s-no-init
>>>
>>> I'd like to point out the output of "git show --stat" for that diff:
>>>  29 files changed, 130 insertions(+), 1560 deletions(-)
>>>
>>> You get massive code reuse by simply using spark-submit. The remote
>>> dependencies are downloaded in the driver, and the driver does the job
>>> of service them to executors.
>>>
>>> So I guess my question is: is there any advantage in using an init
>>> container?
>>>
>>> The current init container code can download stuff in parallel, but
>>> that's an easy improvement to make in spark-submit and that would
>>> benefit everybody. You can argue that executors downloading from
>>> external servers would be faster than downloading from the driver, but
>>> I'm not sure I'd agree - it can go both ways.
>>>
>>> Also the same idea could probably be applied to starting executors;
>>> Mesos starts executors using "spark-class" already, so doing that
>>> would both improve code sharing and potentially simplify some code in
>>> the k8s backend.
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe e-mail: 

Re: Kubernetes backend and docker images

[Yinan Li]

This is neat. With some code cleanup and as long as users can still use
custom driver/executor/init-container images if they want to, I think this
is great to have. I don't think there's a particular reason why having a
single image wouldn't work. Thanks for doing this!

On Fri, Jan 5, 2018 at 5:06 PM, Marcelo Vanzin  wrote:

> Hey all, especially those working on the k8s stuff.
>
> Currently we have 3 docker images that need to be built and provided
> by the user when starting a Spark app: driver, executor, and init
> container.
>
> When the initial review went by, I asked why do we need 3, and I was
> told that's because they have different entry points. That never
> really convinced me, but well, everybody wanted to get things in to
> get the ball rolling.
>
> But I still think that's not the best way to go. I did some pretty
> simple hacking and got things to work with a single image:
>
> https://github.com/vanzin/spark/commit/k8s-img
>
> Is there a reason why that approach would not work? You could still
> create separate images for driver and executor if wanted, but there's
> no reason I can see why we should need 3 images for the simple case.
>
> Note that the code there can be cleaned up still, and I don't love the
> idea of using env variables to propagate arguments to the container,
> but that works for now.
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Announcing Spark on Kubernetes release 0.5.0

[Yinan Li]

The Spark on Kubernetes development community is pleased to announce
release 0.5.0
of Apache Spark with Kubernetes as a native scheduler back-end!

This release includes a few bug fixes and the following features:

   - Spark R support
   - Kubernetes 1.8 support
   - Mounts emptyDir volumes for temporary directories on executors in
   static allocation mode

The full release notes are available here:
https://github.com/apache-spark-on-k8s/spark/releases/
tag/v2.2.0-kubernetes-0.5.0

Community resources for Spark on Kubernetes are available at:

   - Slack: https://kubernetes.slack.com
   - User Docs: https://apache-spark-on-k8s.github.io/userdocs/
   - GitHub: https://github.com/apache-spark-on-k8s/spark