This is an automated email from the ASF dual-hosted git repository. chengpan pushed a commit to branch 0.3.1 in repository https://gitbox.apache.org/repos/asf/incubator-celeborn-website.git
commit 38681d611324d5b54051d390628a607c9b333ba2 Author: Cheng Pan <[email protected]> AuthorDate: Fri Oct 13 22:17:10 2023 +0800 Release 0.3.1-incubating --- .github/workflows/site.yaml | 12 +- docs/community/news.md | 1 + docs/community/release_notes/release_note_0.3.1.md | 167 +++++++++++++++++++++ docs/download.md | 11 +- mkdocs.yml | 2 +- 5 files changed, 185 insertions(+), 8 deletions(-) diff --git a/.github/workflows/site.yaml b/.github/workflows/site.yaml index ec1c18c..6d38f7f 100644 --- a/.github/workflows/site.yaml +++ b/.github/workflows/site.yaml @@ -70,16 +70,16 @@ jobs: git add . git commit -m 'docs/0.2.1-incubating' - run: | - wget https://github.com/apache/incubator-celeborn/archive/refs/tags/v0.3.0-incubating.tar.gz - tar -xzf v0.3.0-incubating.tar.gz - cd incubator-celeborn-0.3.0-incubating + wget https://github.com/apache/incubator-celeborn/archive/refs/tags/v0.3.1-incubating.tar.gz + tar -xzf v0.3.1-incubating.tar.gz + cd incubator-celeborn-0.3.1-incubating mkdocs build cd .. mkdir -p docs - if [ -d docs/0.3.0-incubating ]; then rm -r docs/0.3.0-incubating; fi - mv incubator-celeborn-0.3.0-incubating/site docs/0.3.0-incubating + if [ -d docs/0.3.1-incubating ]; then rm -r docs/0.3.1-incubating; fi + mv incubator-celeborn-0.3.1-incubating/site docs/0.3.1-incubating git add . - git commit -m 'docs/0.3.0-incubating' + git commit -m 'docs/0.3.1-incubating' - run: | echo 'publish:' >> .asf.yaml echo ' whoami: asf-site' >> .asf.yaml diff --git a/docs/community/news.md b/docs/community/news.md index 3b40767..f28104e 100644 --- a/docs/community/news.md +++ b/docs/community/news.md @@ -18,6 +18,7 @@ license: | | Date | Title | Brief | |-----------------|---------------------------------------------------------|-------------------------------------------------------------------------------------------------| +| 2023 October 13 | Release 0.3.1-incubating | Celeborn release 0.3.1-incubating. | | 2023 August 4 | New Committer: Fu chen | We are happy to announce Fu Chen becomes a new Celeborn committer. | | 2023 July 27 | Release 0.3.0-incubating | Celeborn release 0.3.0-incubating. | | 2023 March 29 | Release 0.2.1-incubating | Celeborn release 0.2.1-incubating. | diff --git a/docs/community/release_notes/release_note_0.3.1.md b/docs/community/release_notes/release_note_0.3.1.md new file mode 100644 index 0000000..c5de3b6 --- /dev/null +++ b/docs/community/release_notes/release_note_0.3.1.md @@ -0,0 +1,167 @@ +--- +hide: + - navigation + +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +# Apache Celeborn(Incubating) 0.3.1 Release Notes + +## Highlight + +- Flink supports split partitions +- Kubernetes deployment enhancements +- Metrics and Grafana Dashboard improvements +- Prefer to use jemalloc for memory allocation +- Globally disable thread-local cache in the shared PooledByteBufAllocator +- Fix wrongly delete running app shuffle data +- Fix occasionally data reading error when AQE is enabled + +### Improvement + +[CELEBORN-152] Add config to limit max workers when offering slots +[CELEBORN-468] Timeout useless lostWorkers/shutdownWorkers meta +[CELEBORN-498] Add new config for DfsPartitionReader's chunk size +[CELEBORN-627] Support split partitions +[CELEBORN-656] Should also refine log about return HARD_SPLIT in handlePushMergedData +[CELEBORN-712] Fix Utils.makeReducerKey +[CELEBORN-726] Amend method names +[CELEBORN-752] Separate local read test +[CELEBORN-752] Support read local shuffle file for spark +[CELEBORN-760] Convert OpenStream and StreamHandler to Pb +[CELEBORN-770] Convert BacklogAnnouncement, BufferStreamEnd, ReadAddCredit to PB +[CELEBORN-771] Convert PushDataHandShake, RegionFinish, RegionStart to PB +[CELEBORN-796] Support for globally disable thread-local cache in the shared PooledByteBufAllocator +[CELEBORN-798] Revert Add heartbeat from client to LifecycleManager to clean +[CELEBORN-804] ShuffleManager stop should set shuffleClient to null +[CELEBORN-807] Adjust shutdown worker logs in LifecycleManager +[CELEBORN-809] Directly use isDriver passed from SparkEnv +[CELEBORN-812] Cleanup SendBufferPool if idle for long +[CELEBORN-819] Worker close should pass close status to support handle graceful shutdown and decommission +[CELEBORN-820] Merge service shutdown and close method +[CELEBORN-827] Eliminate unnecessary chunksBeingTransferred calculation +[CELEBORN-830] Add spark integration test to verify fallback with workers unavailable +[CELEBORN-830] Check available workers in CelebornShuffleFallbackPolicyRunner +[CELEBORN-832] Support use RESTful API to trigger worker decommission +[CELEBORN-837] Add silencer plugin to suppress deprecated warnings +[CELEBORN-838] Add custom mvn flag to celeborn +[CELEBORN-846] Remove unused updateReleaseSlotsMeta in master side +[CELEBORN-847] Support use RESTful API to trigger worker exit and exitImmediately +[CELEBORN-852] Add active connection count metrics to grafana dashboard +[CELEBORN-852] Adding new metrics to record the number of registered … +[CELEBORN-863] Persist committed file infos to support worker recovery +[CELEBORN-874] Enrich Fetch log +[CELEBORN-876] Enhance log to find out failed workers if data lost +[CELEBORN-878] Convert all IOException to PartitionUnRetryAbleException when openStream/read file +[CELEBORN-882] Add `Pause Push Data Time Count` Metrics & Dashboard Panel +[CELEBORN-883] Optimized configuration checks during MemoryManager initialization +[CELEBORN-886] Support multiple celeborn clusters in the same K8s namespace +[CELEBORN-888] Tweak the logic and add unit tests for the MemoryManager#currentServingState method +[CELEBORN-892] Fix statistics error of commitFiles method +[CELEBORN-897] Set celeborn.network.memory.allocator.allowCache default to false +[CELEBORN-900] Disable jemalloc in non-docker environment +[CELEBORN-900] Prefer to use jemalloc for memory allocation +[CELEBORN-901] Add support for Scala 2.13 +[CELEBORN-902] Associate Celeborn.storage.dir with volumes in Helm values.yml +[CELEBORN-904] Bump Spark in spark-3.3 profile from 3.3.2 to 3.3.3 +[CELEBORN-905] Redraw the flowchart backpressure.svg after worker pause logic is reconstructed +[CELEBORN-908] Tweak pause and resume logic && add unit test MemoryManager memory check thread +[CELEBORN-920] Worker sends its load to Master through heartbeat +[CELEBORN-922] Improve celeborn shuffle maanger fallback log message +[CELEBORN-924] Change log level to error in PartitionFilesSorter upon Exception +[CELEBORN-933] Add metrics about active shuffle data size +[CELEBORN-934] Make the log description in switchServingState more precise +[CELEBORN-942] Release script supports uploading Nexus +[CELEBORN-943] Pre-create CelebornInputStreams in CelebornShuffleReader +[CELEBORN-945] Change ShutdownHook's timeout for decommission +[CELEBORN-950] Change CelebornShuffleReader log level and information +[CELEBORN-956] Modify parameter passing in AbstractRemoteShuffleInputGateFactory +[CELEBORN-957] Simplify nano time duration calculation +[CELEBORN-958] Log DNS resolution result +[CELEBORN-959] Use Java API to obtain disk capacity information instead of `df` command +[CELEBORN-960] Exclude workers without healthy disks +[CELEBORN-962] Add check DiskInfo#Status in PushDataHandler#checkDiskFull +[CELEBORN-963] Add WORKDIR in celeborn Dockerfile +[CELEBORN-964] Simplify read process output to prevent leak +[CELEBORN-968] Make volume name dynamic in StatefulSet in Helm chart +[CELEBORN-969] Allow user set priorityClass used by celeborn pods +[CELEBORN-971] Should update Charts appVersion when we update project version +[CELEBORN-975] Refactor the check logic to stop the celeborn master and worker +[CELEBORN-976] Introduce script to check master and worker status +[CELEBORN-979] Reduce default disk Check Interval +[CELEBORN-981] Imrpove enable graceful shutdown tips +[CELEBORN-982] Improve RPC bind port tips +[CELEBORN-986] Use formatted log instead of string concat +[CELEBORN-1007] Improve JVM metrics naming and add ThreadStates metrics +[CELEBORN-1008] Adjust push/fetch timeout checker thread pool and tasks +[CELEBORN-1014] Output log with bound address and port +[MINOR] Add an alternative for CLIENT_RESERVE_SLOTS_RACKAWAE_ENABLED +[MINOR] Exclude sbt files from git + +### Stability and Bug Fix + +[CELEBORN-656] Fix wrong message call when revive return STAGE_END +[CELEBORN-788] Update callback's location should also update the PushState to keep consistent +[CELEBORN-788] Update latest PartitionLocation before retry PushData +[CELEBORN-804] ShuffleClient should cleanup shuffle infos when trigger unregisterShuffle +[CELEBORN-806] Correct the conf key `celeborn.data.io.threads` within the class `ShuffleClientImpl` +[CELEBORN-819] Fix worker graceful shutdown exitKind set +[CELEBORN-844] Fix incorrect config name in ConfigEntity checkvalue method and format message +[CELEBORN-845] Sort memory counter won't decrease after sort failed +[CELEBORN-846] Fix broken link caused by unknown RPC +[CELEBORN-881] StorageManager clean up thread may delete new app directories +[CELEBORN-890] PushHandler should check whether FileWriter has closed to avoid data lost +[CELEBORN-899] Fix potential NPE in ShuffleClientImpl#revive +[CELEBORN-917] Record read metric should be compatible with Gluten shuffle serde +[CELEBORN-926] Enabled GRACEFUL SHUTDOWN, will meet IllegalMonitorStateException +[CELEBORN-928] Don't stop LocalFlusher when notify error +[CELEBORN-932] Fix worker register after gracefaully restart +[CELEBORN-946] Record read metric should be compatible with Gluten shuffle dependency +[CELEBORN-961] Catch exception when constructing Worker +[CELEBORN-972] Enhance workingdirDiskCapacity unit parsing and fix ConfigMap is not effected for workerStatefuleSet +[CELEBORN-988] Add config option to control original unsorted file deletion in `PartitionFilesSorter` +[CELEBORN-988] Rename config key `celeborn.worker.sortPartition.lazyRemovalOfOriginalFiles.enabled` +[CELEBORN-990] Add exception handler when calling CelebornHadoopUtils.getHadoopFS +[CELEBORN-1005] Clean expired app dirs will delete the running application +[CELEBORN-1013] Shutdown master if initialized failed + +### Documentation + +[CELEBORN-810] Fix some typos and grammar +[CELEBORN-811] Refine monitoring doc +[CELEBORN-814] Merge upgrade doc to Deployment tab and add TOC +[CELEBORN-909] Restore titles in migration guide +[CELEBORN-941] Fix incorrect deploy doc +[CELEBORN-944] Add link about cluster planning +[CELEBORN-984] ShutdownWorkers API documentation +[CELEBORN-1003] Correct the LICENSE and NOTICE for shaded client jars +[CELEBORN-1007] Update Migration Guide +[CELEBORN-1009] CELEBORN_PREFER_JEMALLOC +[MINOR] Fix configuration version +[MINOR] Fix typo in CelebornConf + +## Credits +Thanks to the following contributors who helped to review and commit to Apache Celeborn(Incubating) 0.3.1-incubating version: + +| Contributors | | | | | | +|--------------|----------------|---------------|---------------|----------------|-------------| +| Angerszhuuuu | Aravind Patnam | Cheng Pan | Fu Chen | Jun He | Keyong Zhou | +| Melody | Shuang | SteNicholas | Xiduo You | camper42 | caojiaqing | +| hongzhaoyang | jiang13021 | jiaoqingbo | liangyongyuan | lishiyucn | mingji | +| onebox-li | sunjunjie | sychen | xiyu.zk | zhongqiang.czq | zwangsheng | +| 宪英 | | | | | | + + + diff --git a/docs/download.md b/docs/download.md index 0ffa370..a4ca81c 100644 --- a/docs/download.md +++ b/docs/download.md @@ -21,6 +21,16 @@ license: | The latest version is {{ stable_version }}. +### 0.3.1-incubating (2023-10-13) + +[release note](community/release_notes/release_note_0.3.1.md) + +| | Download from ASF | Checksum | Signature | +|:-----------:|:-------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------:|:---------:| +| Source Code | [src](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.1-incubating/apache-celeborn-0.3.1-incubating-source.tgz) | [sha512](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.1-incubating/apache-celeborn-0.3.1-incubating-source.tgz.sha512) | [asc](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.1-incubating/apache-celeborn-0.3.1-incubating-source.tgz.asc) | +| Binary | [bin](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.1-incubating/apache-celeborn-0.3.1-incubating-bin.tgz) | [sha512](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.1-incubating/apache-celeborn-0.3.1-incubating-bin.tgz.sha512) | [asc](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.1-incubating/apache-celeborn-0.3.1-incubating-bin.tgz.asc) | + + ### 0.3.0-incubating (2023-07-27) [release note](community/release_notes/release_note_0.3.0.md) @@ -31,7 +41,6 @@ The latest version is {{ stable_version }}. | Binary | [bin](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.0-incubating/apache-celeborn-0.3.0-incubating-bin.tgz) | [sha512](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.0-incubating/apache-celeborn-0.3.0-incubating-bin.tgz.sha512) | [asc](https://www.apache.org/dyn/closer.lua/incubator/celeborn/celeborn-0.3.0-incubating/apache-celeborn-0.3.0-incubating-bin.tgz.asc) | - ### 0.2.1-incubating (2023-03-29) [release note](community/release_notes/release_note_0.2.1.md) diff --git a/mkdocs.yml b/mkdocs.yml index 9aa79a2..af676e0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -51,7 +51,7 @@ markdown_extensions: extra: version: 0.4.0-SNAPSHOT - stable_version: 0.3.0 + stable_version: 0.3.1 social: - icon: fontawesome/brands/github
