Re: EMR + celeborn

t to false: spark.celeborn.client.spark.push.unsafeRow.fastWrite.enabled=false Regards, Keyong Zhou rahul gidwani 于2024年6月18日周二 04:53写道： > Hello, > > I was wondering if anyone here has tried running celeborn in k8s for spark > on EMR? Our spark clusters run in EMR currently and want to uti

Necessary config for AWS Spark

Dear Celeborn users, If you are running AWS Spark with Celeborn, please make sure the following spark config is added, or there might be undefined behavior: spark.celeborn.client.spark.push.unsafeRow.fastWrite.enabled=false Regards, Keyong Zhou

[ANNOUNCE] Add Mridul Muralidharan as new committer

submission process. This should enable better productivity. A PMC member helps manage and guide the direction of the project. Please join me in congratulating Mridul Muralidharan! Regards, Keyong Zhou

Re: How does Celeborn handles FetchFailed exception

ailure, conf.clientRpcRegisterShuffleRpcAskTimeout(), ClassTag$.MODULE$.apply(PbReportShuffleFetchFailureResponse.class)); ... } cc Erik, Mridul [1] https://docs.google.com/document/d/1dkG6fww3g99VAb1wkphNlUES_MpngVPNg8601chmVp8 [2] https://lists.apache.org/thread/nogsgc61qh2zomdvhfbs4b0y88s3qtqg Regards, Keyon

[ANNOUNCE] Add Chandni Singh as new committer

submission process. This should enable better productivity. A (P)PMC member helps manage and guide the direction of the project. Please join me in congratulating Chandni Singh! Thanks, Keyong Zhou

Re: Celeborn for Spark 3.2 + JDK17

I think so. Do you mind sending a PR to Celeborn? :) Regards, Keyong Zhou Curtis Howard 于2024年3月12日 周二22:58写道： > Thanks Keyong, > > I was able to build the 'client' JARs for Spark 3.2 / JDK21, and run > simple tests; however I did face a runtime error similar

Re: Celeborn for Spark 3.2 + JDK17

Hi Curtis, With this PR https://github.com/apache/incubator-celeborn/pull/2385 you can compile with JDK21 using following commands: ./build/make-distribution.sh -Pspark-3.5 -Pjdk-21 Regards, Keyong Zhou Curtis Howard 于2024年3月9日周六 02:38写道： > Thank you Keyong! > > Related to this, ha

Re: Celeborn for Spark 3.2 + JDK17

Hi Curtis, Thanks for reaching out! No there is no known blockers for Celeborn + Spark 3.2 + JDK17, and I think there is a good chance that it could be used successfully. Any problems with your test, feel free to let us know :) Regards, Keyong Zhou Curtis Howard 于2024年3月8日周五 22:23写道： >

Re: Celeborn Spark dynamic allocation support (in Spark versions < 3.5)

, Keyong Zhou Curtis Howard 于2024年3月8日周五 22:32写道： > Hi, > > The documentation mentions Spark 3.5 or greater is required for Spark > dynamic allocation to be supported through Celeborn: > https://celeborn.apache.org/docs/latest/deploy/#spark-configuration > # Support Spark Dynamic R

[ANNOUNCE] Add Xiaofeng Jiang as new committer

submission process. This should enable better productivity. A (P)PMC member helps manage and guide the direction of the project. Please join me in congratulating Xiaofeng Jiang! Thanks, Keyong Zhou

[ANNOUNCE] Add Yihe Li as new committer

submission process. This should enable better productivity. A (P)PMC member helps manage and guide the direction of the project. Please join me in congratulating Yihe Li! Thanks, Keyong Zhou

[ANNOUNCE] Add zhongqiangchen(Zhongqiang Chen) as new committer

tinue contributing to the project, pushing Celeborn to the next level together with all contributors of the community! Also, we are looking forward to add more and more committers to our project :) Thanks! Keyong Zhou

[ANNOUNCE] Add rexxiong(Jiashu Xiong) as new committer

o the project, pushing Celeborn to the next level together with all contributors of the community! Also, we are looking forward to add more and more committers to our project :) Thanks! Keyong Zhou