Re: [Internet]Re: Improving Dynamic Allocation Logic for Spark 4+

-of-pods#section-3e8-8n8-hdh On Fri, Aug 25, 2023 at 10:08 PM Mich Talebzadeh wrote: > Hi Qian, > > How in practice have you implemented image caching for the driver and > executor pods respectively? > > Thanks > > On Thu, 24 Aug 2023 at 02:44, Qian Sun wrote: > >>

Re: [Internet]Re: Improving Dynamic Allocation Logic for Spark 4+

memory":"1433Mi"},"name":"spark-kubernetes-driver"}]},"output... >>>>>>> >>>>>>> autopilot.gke.io/warden-version: 2.7.41 >>>>>>> >>>>>>> >>>>>>> >>>>>>> This is on spark 3.4.1 with Java 11 both the host running >>>>>>> spark-submit and the docker itself >>>>>>> >>>>>>> >>>>>>> >>>>>>> I am not sure how relevant this is to this discussion but it looks >>>>>>> like a kind of blocker for now. What config params can help here and >>>>>>> what >>>>>>> can be done? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> >>>>>>> Solutions Architect/Engineering Lead >>>>>>> >>>>>>> London >>>>>>> >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>> >>>>>>>view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>> for any loss, damage or destruction of data or any other property which >>>>>>> may >>>>>>> arise from relying on this email's technical content is explicitly >>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>> damages >>>>>>> arising from such loss, damage or destruction. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, 7 Aug 2023 at 22:39, Holden Karau >>>>>>> wrote: >>>>>>> >>>>>>> Oh great point >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 7, 2023 at 2:23 PM bo yang wrote: >>>>>>> >>>>>>> Thanks Holden for bringing this up! >>>>>>> >>>>>>> >>>>>>> >>>>>>> Maybe another thing to think about is how to make dynamic allocation >>>>>>> more friendly with Kubernetes and disaggregated shuffle storage? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 7, 2023 at 1:27 PM Holden Karau >>>>>>> wrote: >>>>>>> >>>>>>> So I wondering if there is interesting in revisiting some of how >>>>>>> Spark is doing it's dynamica allocation for Spark 4+? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Some things that I've been thinking about: >>>>>>> >>>>>>> >>>>>>> >>>>>>> - Advisory user input (e.g. a way to say after X is done I know I >>>>>>> need Y where Y might be a bunch of GPU machines) >>>>>>> >>>>>>> - Configurable tolerance (e.g. if we have at most Z% over target >>>>>>> no-op) >>>>>>> >>>>>>> - Past runs of same job (e.g. stage X of job Y had a peak of K) >>>>>>> >>>>>>> - Faster executor launches (I'm a little fuzzy on what we can do >>>>>>> here but, one area for example is we setup and tear down an RPC >>>>>>> connection >>>>>>> to the driver with a blocking call which does seem to have some locking >>>>>>> inside of the driver at first glance) >>>>>>> >>>>>>> >>>>>>> >>>>>>> Is this an area other folks are thinking about? Should I make an >>>>>>> epic we can track ideas in? Or are folks generally happy with today's >>>>>>> dynamic allocation (or just busy with other things)? >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> >>>>>>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> -- Regards, Qian Sun

Re: Executor metrics are missing on Prometheus sink

b/spark-dashboard ) > > An additional comment is that there is room for having more sinks > available for Apache Spark metrics, notably for InfluxDB and for Prometheus > (gateway), if someone is interested in working on that. > > > > Best, > > Luca > > > > > > *Fr

Executor metrics are missing on prometheus sink

-instance--executor *How to expose executor metrics on spark exeuctors pod?* *Any help will be appreciated.* -- Regards, Qian Sun

[DISCUSS] SPIP: Introduce Chaos Experiments in Apache Spark

-- Best! Qian Sun

Re: Welcome Yikun Jiang as a Spark committer

ut a lot of effort into stabilizing and optimizing the builds > so we all can work together in Apache Spark more > efficiently and effectively. He's also driving the SPIP for Docker > official image in Apache Spark as well for users and developers. > Please join me in welcoming Yikun! > > -- Best! Qian SUN

Re: [VOTE] SPIP: Support Docker Official Image for Spark

ssues.apache.org/jira/browse/SPARK-40513> > > Please vote on the SPIP for the next 72 hours: > > [ ] +1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don’t think this is a good idea because … > > -- Best! Qian SUN

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

docker images maintenance effort (such as > frequently rebuilding, image security update) of the Apache Spark community. > > See more in SPIP DOC: > https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o > > cc: Ruifeng (co-author) and Hyukjin (shepherd) > > Regards, > Yikun > -- Best! Qian SUN

Re: Welcoming three new PMC members

gt; >>> The Spark PMC recently voted to add three new PMC members. Join me in >>> welcoming them to their new roles! >>> >>> New PMC members: Huaxin Gao, Gengliang Wang and Maxim Gekk >>> >>> The Spark PMC >>> >> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> > -- Best! Qian SUN

Re: Welcome Xinrong Meng as a Spark committer

Congratulations Xinrong! Regards, Qian SUN Yang,Jie(INF) 于2022年8月9日周二 17:10写道： > Congratulations! > > > Regards, > > Yang Jie > > > > > > *发件人**: *Hyukjin Kwon > *日期**: *2022年8月9日 星期二 16:12 > *收件人**: *dev > *主题**: *Welcome Xinrong Meng as a Spar

Re: Contributions and help needed in SPARK-40005

Sure, I will do it. SPARK-40010 <https://issues.apache.org/jira/browse/SPARK-40010> is built to track progress. Hyukjin Kwon gurwls...@gmail.com <http://mailto:gurwls...@gmail.com> 于2022年8月9日周二 10:58写道： Please go ahead. Would be very appreciated. > > On Tue, 9 Aug 2022 at 11:5

Re: Contributions and help needed in SPARK-40005

das.DataFrame.pivot. >>>There are many API that misses parameters in PySpark, e.g., >>> DataFrame.union >>> >>> Here is one example PR I am working on: >>> https://github.com/apache/spark/pull/37437 >>> I can't do it all by myself. Any help, review, and contributions >>> would be welcome and appreciated. >>> >>> Thank you all in advance. >>> >> -- Best! Qian SUN

Re: spark driver with OOM due to org.apache.spark.status.ElementTrackingStore

.ui.retainedTasks 100 > > I'll set this, spark.ui.dagGraph.retainedRootRDDs, as well. > > Any other advice for this? > > Thanks > Jason > > On Wed, 3 Aug 2022 at 15:56, Qian Sun wrote: > >> Hi Jason >> LiveUI initializes ElementTrackingStore with InMemorySt

Re: spark driver with OOM due to org.apache.spark.status.ElementTrackingStore

Hi Jason LiveUI initializes ElementTrackingStore with InMemoryStore, so it has OOM risk. /** * Create an in-memory store for a live application. */ def createLiveStore( conf: SparkConf, appStatusSource: Option[AppStatusSource] = None): AppStatusStore = { val store = new

Re: [PSA] Please rebase and sync your master branch in your forked repository

Thank you Hyukjin > 2022年6月21日 上午7:45，Hyukjin Kwon 写道： > > After https://github.com/apache/spark/pull/36922 > gets merged, it requires your > fork's master branch to be synced to the latest master branch in Apache > Spark. Otherwise, builds would

Re: Stickers and Swag

GOOD! Could these are mailed to China? > 2022年6月14日 下午2:04，Xiao Li 写道： > > Hi, all, > > The ASF has an official store at RedBubble > that Apache Community > Development (ComDev) runs. If you are interested in buying Spark Swag, 70 > products

Maven Test blocks with TransportCipherSuite

] at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) ~[surefire-booter-3.0.0-M5.jar:3.0.0-M5] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) ~[surefire-booter-3.0.0-M5.jar:3.0.0-M5] Anyone with same exception? -- Best! Qian SUN

Re: SIGMOD System Award for Apache Spark

Congratulations !!! > 2022年5月13日 上午3:44，Matei Zaharia 写道： > > Hi all, > > We recently found out that Apache Spark received > the SIGMOD System Award this year, > given by SIGMOD (the ACM’s data management research organization) to > impactful

Re: PR builder not working now

as >>>> https://github.com/apache/spark/pull/36157/checks?check_run_id=5984075130 >>>> because we rely on that. >>>> >>>> To check the PR builder's status, we should manually find the workflow >>>> run in PR author's repository for now by going to: >>>> https://github.com/[PR AUTHOR >>>> ID]/spark/actions/workflows/build_and_test.yml >>>> >>> -- Best! Qian SUN

Re: [How To] run test suites for specific module

stOnly org.apache.spark.scheduler.DAGSchedulerSuite Hope this helps Best regards, Qian Sun Fangjia Shen 于2022年1月25日周二 07:44写道： > Hello all, > > How do you run Spark's test suites when you want to test the correctness > of your code? Is there a way to run a specific test suite for Spark? For > exampl

Re: [VOTE] Release Spark 3.2.1 (RC1)

+1 Looks good. All integration tests passed. Qian > 2022年1月11日 上午2:09，huaxin gao 写道： > > Please vote on releasing the following candidate as Apache Spark version > 3.2.1. > > The vote is open until Jan. 13th at 12 PM PST (8 PM UTC) and passes if a > majority > +1 PMC votes are cast, with

Re: Log4j 1.2.17 spark CVE

My understanding is that we don’t need to do anything. Log4j2-core not used in spark. > 2021年12月13日 下午12:45，Pralabh Kumar 写道： > > Hi developers, users > > Spark is built using log4j 1.2.17 . Is there a plan to upgrade based on > recent CVE detected ? > > > Regards > Pralabh kumar