This is an automated email from the ASF dual-hosted git repository.
xianjingfeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle-website.git
The following commit(s) were added to refs/heads/master by this push:
new fdb00dd add release notes for 0.8.0
fdb00dd is described below
commit fdb00dda2cd59280081723932d043bb46e0a49b8
Author: 01390972 <[email protected]>
AuthorDate: Mon Oct 9 11:42:00 2023 +0800
add release notes for 0.8.0
---
download/release-notes-0.8.0.md | 279 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 279 insertions(+)
diff --git a/download/release-notes-0.8.0.md b/download/release-notes-0.8.0.md
new file mode 100644
index 0000000..3443586
--- /dev/null
+++ b/download/release-notes-0.8.0.md
@@ -0,0 +1,279 @@
+---
+title: Release Notes 0.8.0
+sidebar_position: 995
+---
+
+# Uniffle Release 0.8.0
+
+## Highlight
+
+- Support TEZ
+- Introduce Netty for shuffle data transmission
+- Use off heap memory to store shuffle data.
+- Introduce REST API for cluster management.
+- Introduce command line for cluster management.
+
+## ChangeLog
+
+* [#1175] fix(netty): Retry failed with StacklessClosedChannelException after
channel closed (#1181)
+* [#1177] improvement: Reduce the write time of tasks (#1179)
+* [MINOR] docs: Fix spark.serializer in README and client_guide (#1180)
+* [#1173] fix: incorrect shuffle server status (#1174)
+* [#1164] refactor: Exposing the getDataLen method for ShuffleDataResult
(#1170)
+* [#1165][FOLLOWUP] Fix some spelling mistakes. (#1172)
+* [#1165] improvement(tez): Unregister shuffle data after completing the
execution of a DAG. (#1166)
+* [#1013] fix(tez): Wait when return MapOutput of type wait (#1063)
+* [#1062] fix(tez): Fix to use TezIdHelper to getTaskAttemptId instead of
default IdHelper. (#1078)
+* [#1161] improvement(netty): Reduce the data copy when accessing data buffer
len (#1162)
+* [#722] test: cleanup residue files in tmp directory after tests (#1134)
+* [#1108] feat(server): Add labels with disk path for
`total_localfile_write_data` metrics. (#1160)
+* [#1152] fix: Direct memory may leak in shuffle server. (#1154)
+* [#1155] fix(netty): io.netty.util.internal.OutOfDirectMemoryError. (#1151)
+* [#1051] fix(mr): Ensure configurations in both mapred-site.xml and
dynamic_client.conf take effect. (#1112)
+* [#1127] fix(netty): incorrect bytebuf release for ShuffleBlockInfo data in
client side (#1150)
+* [#1132] improvement(spark): Unregister shuffle explicitly when Spark
application is stopped. #1139
+* [#1044] chore: Add LICENSE-binary (#1054)
+* [#1143] docs: Correct sequence number text by reducing paragraph indentation
by 1 space (#1144)
+* [#1124] docs(tez): Add the document of config option
`tez.rss.client.send.thread.num` (#1142)
+* [#1124] docs(tez): Add the document for tez-client (#1140)
+* [#837] feat: Display information of all application through Cli. (#964)
+* [#951] fix rss-server Docker image building bug. (#1027)
+* [#1115] improvement(tez&mr): Unregister shuffle explicitly when application
is stopped. (#1131)
+* [#1111] fix: Shuffle server can not delete remote storage path of secured
HDFS cluster (#1122)
+* [#1124] docs: modify the document for tez-client (#1126)
+* [#1114] feat: introduce hdfs host as the total_hadoop_write_data metric
label (#1107)
+* [#1109] fix(tez): Fix the user of remote storage. (#1128)
+* [#1124] docs: Add the document for tez-client (#1125)
+* [#1081] fix(tez): shuffle can not read the data which is flushed to hdfs
(#1118)
+* [#299] improvement: Make config type of RSS_STORAGE_TYPE as enum (#1123)
+* [#1045] fix(server): shuffle server may hang when restart worker due to
multi require-momery and no require-momery release (#1058)
+* [#1060] doc: Add the document for Netty. (#1116)
+* [#434] refactor: New utility method to cover dynamic class loading in
RSSUtils (#1104)
+* [#1006] feat(spark): Support Spark 3.4 (#1082)
+* [#1043] chore: Add NOTICE-binary (#1097)
+* [#1101] improvement(tez): Release server resources as soon as possible in
RssDAGAppMaster. (#1102)
+* [MINOR]Fix(tez) add output mapOutputRecordCounter metrics (#1093)
+* [#1074] feat: Introduce the metric of `local_storage_service_used_space`
(#1075)
+* Revert "[#299] improvement: Make config type of RSS_STORAGE_TYPE as enum
(#1052)" (#1106)
+* [#299] improvement: Make config type of RSS_STORAGE_TYPE as enum (#1052)
+* [#1100] improvement(mr): Fail fast in SortWriteBufferManager when failed to
send shuffle data to shuffle server. (#1103)
+* [#1095] doc: Add slack link to the readme.md (#1099)
+* [#1066] fix: The Jetty server fails to start when compiled with JDK 8 but
runtime version is JDK 11. (#1067)
+* [#1070] fix(tez): shuffle server may leak if not register remote storage.
(#1076)
+* [#1068] feat(tez): Fail fast in client when failed to send data to server.
(#1069)
+* [MINOR] fix(mr): fix default value (#1035)
+* [#1064] improvement(tez): Make shuffle data send thread pool configurable in
WriteBufferManager. (#1065)
+* [MINOR] doc: Add JDK benchmark tests (#1059)
+* [#477] feat(spark): Fix `rss.resubmit.stage` does not support dynamic client
conf. (#1050)
+* [#1011] feat(tez): Avoid recompute succeeded task. (#1033)
+* [MINOR] chore(asf): stop forwarding GitHub issues to dev mail list (#1032)
+* [#1048] fix(coord): ExcludeNodes does not take effect when the coordinator
restarts. (#1049)
+* [#640] feat(netty): Metric system for netty server (#1041)
+* [#133] feat(netty): local shuffle read support zero-copy. (#1047)
+* [MINOR] doc: fix metrics document (#1040)
+* [#889] improvement: set default value of
`rss.server.max.concurrency.of.per-partition.write` to `30`. (#1037)
+* [#889] improvement: Modify default value of single buffer flush. (#1039)
+* [#889] improvement: Modify the default value of the
`rss.coordinator.select.partition.strategy` parameter to `CONTINUOUS`. (#1036)
+* [#133] feat(netty): integration-test supports netty. (#1008)
+* [#1018] test(tez) RssUnorderedPartitionedKVOutputTest add close func unit
test (#1034)
+* [MINOR] Add new collabrators (#1031)
+* [#808] feat(spark): ensure thread safe and data consistency when spill (#848)
+* [#1019] test(tez) RssOrderedPartitionedKVOutputTest add close func unit test
(#1025)
+* [#477] feat(spark): support getShuffleResult throws FetchFailedException.
(#1004)
+* [#973] improvement: Make shuffle manager client RPC timeout configurable
(#1017)
+* [#846] feat(cli): Service's start & stop & restart through Cli. (#925)
+* [#388][FOLLOWUP] fix: Fix the flaky test GrpcServerTest (#1023)
+* [#993] feat(tez): Optimize the method of obtain the application attemptId
(#1021)
+* [MINOR] refactor: use general method to check the remote storage existence
(#980)
+* [#972 ] fix(tez): Add output mapOutputByteCounter metrics (#1016)
+* [MINOR] test(tez): add storage type configs in TezRemoteShuffleManagerTest
(#1015)
+* [#999] improvement: apply REST principles in the urls of the http interfaces
(#1000)
+* [#992] fix(tez): convertTaskAttemptIdToLong should not consider appattemptId
(#1007)
+* [#1010]chore: enable and enforce spotless (#1009)
+* [#991] Improvement(tez): TezRemoteShuffleManager support secure cluster.
(#1005)
+* [#957] feat(tez): Tez examples integration test (#982)
+* [#1001] improvement: support get all metrics by one request (#1002)
+* [#757] feat(server): separate flush thread pools for different storage
types. (#775)
+* [#989] Bug(tez): parition class is not set for RssUnorderedKVOutput. (#994)
+* [#864] feat(server): Introduce Jersey to strengthen REST API (#939)
+* [#986] [Improvement][tez] Optimize the method of obtain the vertex id. (#990)
+* [MINOR] docs: update the document and scripts of build tez. (#997)
+* [MINOR] Add new collabrators (#995)
+* [#983] improvement(tez): Optimize tez client delivery configuration (#985)
+* [MINOR] docs: update document for tez client plugin. (#987)
+* [#768] [Follow Up] feat(cli): Cli method for blacklist update. (#968)
+* [#978][Improvement] Provides a tool class to format CLI output content (#979)
+* [#965] feat(tez): support remote shuffle for tez framework (#966)
+* [#961] refactor(coordinator): Improve Coordinator Log Format. (#962)
+* [#940] improvement: Optimize columnar shuffle integration (#958)
+* [#886] fix(tez): Tez Client may lost data or throw exception when
rss.storage.type without MEMORY. (#976)
+* [MINOR] fix(tez): fix thread factory name (#975)
+* [#854][FOLLOWUP] feat(tez): Add Rss Input Class to begin Tez input task
(#949)
+* [#956] refactor: Changes the Boolean flag that determines whether a Node is
healthy to a state (#959)
+* [#854][FOLLOWUP] feat(tez): Add RssShuffleManager to run and manager shuffle
work (#947)
+* [#381]imporvement(server): Check JAVA_HOME and HADOOP_HOME in
start-shuffle-server.sh (#954)
+* [#768] feat(cli): Cli method for blacklist update (#931)
+* [#937] feat: Add rest api for servernode list of losing connection and
unhealthy (#938)
+* [#940] feat: Support columnar shuffle with gluten (#950)
+* [#854][FOLLOWUP] feat(tez): Add RssShuffle to handle event and generate
fetch task (#929)
+* [#854][FOLLOWUP] feat(tez): Add RssShuffleScheduler to run and manager
shuffle work (#948)
+* [#590][part-1] ManagedBuffer instead ByteBuf to hold ShuffleData (#906)
+* [#854][FOLLOWUP] feat(tez): Add Simple Fetched Allocator to allocation
memory or disk for shuffle data (#922)
+* [#855] feat(tez): Support Tez Output RssUnorderedKVOutput (#944)
+* [#855] feat(tez): Support Tez Output UnorderedPartitionedKVOutput (#943)
+* [#854][FOLLOWUP] feat(tez): Add RssTezFetcherTask to fetch data from worker
for OrderedInput (#935)
+* [MINOR] Add new collaborators (#942)
+* [#343] improvement(build): Shade Netty Native lib (#924)
+* [#855] feat(tez): Support Tez Output OrderedPartitionedKVOutput (#930)
+* [#933] fix incorrect metric grpc_server_connection_number (#934)
+* [MINOR] fix: Fix kerberos ut error caused by Config.singleton is not
refresh. (#932)
+* [#854][FOLLOWUP] feat(tez): add RssTezFetcher to fetch data from worker.
(#920)
+* [#819 ] feat(tez): Tez ApplicationMaster supporting RemoteShuffle (#918)
+* [#927] Improvement: improve the control of server heartbeat (#928)
+* [MINOR] improvement(mr): Add @Test to activate test case (#923)
+* [MINOR] fix: Error logs upload fail (#917)
+* [#872][FOLLOWUP] feat(tez): Add UmbilicalUtils to get Worker info from AM
(#919)
+* [#872][FOLLOWUP] feat(tez): Modify utils and add test case (#916)
+* [#872][FOLLOWUP] feat(tez): Add the common and utils class (#894)
+* [#872] feat(tez): Get parameter from Inputcontext then provide util function
(#915)
+* [#908] feat(tez): Write byte array shuffle data to MapOutput (#909)
+* [MINOR] refactor: Fix tez workflow and checkstyle (#911)
+* [MINOR] docs: update document and build script for Hadoop-3.2 (#912)
+* [#896] Improvement: Support Hadoop-3.2 (#897)
+* [#881] fix: Ensure LocalStorageMeta disk size is correctly updated when
events are processed (#902)
+* [#417] refactor: Eliminate raw use of parameterized class (#891)
+* [MINOR] docs: Add benchmark results (#904)
+* [MINOR] refactor: Add overwrite annotation (#905)
+* (jersey2) [#895] improvement: Rename Hdfs*.java to Hadoop*.java to support
other Hadoop FS-compatible distributed filesystem (#898)
+* [MINOR] fix: Fix LocalStorageManager divide by zero exception (#900)
+* [#133] feat(netty): Fix IllegalReferenceCountException. (#899)
+* [#886] fix(mr): MR Client may lost data or throw exception when
rss.storage.type without MEMORY. (#887)
+* [#872] feat(tez): Add the common and utils class (#890)
+* [#715] fix(mr): The container does not exit because shuffleclient is not
closed (#882)
+* [#133] feat(netty): Implement ShuffleServer interface. (#879)
+* [#884] improvement: Make start and stop scripts executable under the bin
folder (#885)
+* [MINOR] Remove unused code of shuffle upload (#883)
+* [MINOR] chore: delete checkstyle-suppressions.xml (#878)
+* [#493] improvement: replace `putIfAbsent` with `computeIfAbsent` to avoid
performance loss in some critical paths (#876)
+* [MINOR] refactor: Reduce the usage of memory in the `ShuffleWriter` (#877)
+* [#863] feat(coordinator): support comments in exclude node files (#874)
+* [#774] docs: Fix the metadata.annotations: Too long in `install.md` (#853)
+* [MINOR] chore(ci): Add Tez pipeline (#862)
+* [#593][FOLLOWUP] Fix zstd compress dest ByteBuffer position
+* [#859][Improvement] Set MALLOC_ARENA_MAX in start-shuffle-server.sh (#860)
+* [#596][FOLLOWUP] Index data support offheap read (#852)
+* [#133] feat(netty): Introduce ShuffleServerGrpcNettyClient. (#839)
+* [#770] feat(cli): Introduce apache.commons.cli basic framework (#833)
+* [#414] feat(client): support specifying per-partition's max concurrency to
write in client side (#815)
+* [#841] feat(config): Support deprecated and fallback keys for ConfigOptions
(#843)
+* fix(client): disable spark memory spill (#844)
+* build(operator): update clusterrole of controller and webhook (#842)
+* [Improvement] Make codec to be a singleton. (#840)
+* [#593][part-1] feat: Codec compress support ByteBuffer (#830)
+* [#133] feat(netty): Rewrite protocol. (#826)
+* [MINOR] Remove unused config `SHUFFLE_EXPIRED_TIMEOUT_MS` (#835)
+* [#827] feat(operator): support generating hpa objects (#828)
+* [MINOR] Modify MD format (#829)
+* [#778] feat: Separate `ShuffleServer` metrics through tags (#812)
+* [MINOR] Add `WORKDIR` (#823)
+* [#416] feat(hdfs): lazy initialization of hdfsShuffleWriteHandler when
per-partition concurrent write is enabled (#816)
+* [#596] feat(netty): Use off heap memory to read HDFS data (#806)
+* [#779] feat: Grpc server support random port (#820)
+* [MINOR] chore: update import order rule to make scala a separated group
(#818)
+* [#804] improvement: Optimize CRC calculation of ByteBuffer (#805)
+* [#477][part-1] feat: support stage resubmit in spark clients (#787)
+* [MINOR] test: Fix the flaky test `GrpcServerTest` (#789)
+* [BUG][MINOR] Fix scripts compatible with jdk8 & jdk11 (#798)
+* [#796] bug:fix the issues of MetricReporter (#797)
+* [#799] feat: use storage host label for remote storage write metrics (#800)
+* [MINOR] fix: the description of Spark patches in the README.md (#801)
+* [#794] feat(operator): support delete ShuffleServer pod with Evicted status
(#795)
+* [#720][FOLLOW-UP] Correct the shuffle server id (#792)
+* [#782]refactor: restrict rss.rpc.server.type to an enum (#783)
+* [#720] feat(netty): support random port for netty (#723)
+* [#584] feat(netty): Add transport client pool for netty (#771)
+* [MINOR] fix: set the default app number in `AccessAppQuotaChecker` (#786)
+* [#477][part-0] feat: add ShuffleManagerServer impl (#777)
+* chore(spark): remove noisy log in client side (#776)
+* [#772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
+* [#706] feat(spark): support spill to avoid memory deadlock (#714)
+* [#755] refactor: Add the method of creating thread pool (#767)
+* [#716] improvement(operator): support specifying imagePullSecrets (#765)
+* [#519] Speed up ConcurrentHashMap#computeIfAbsent (#766)
+* [MINOR] chore(operator): remove useless configuration items in configuration
example (#764)
+* [#133] feat(netty): Add Encoder and Decoder. (#742)
+* [#762] If the storage path is not exist, get file store for its parent.
(#763)
+* [MINOR] test: do not wrap test in try-catch block (#746)
+* [MINOR] chore: Remove committers from collaborators (#754)
+* [#752] refactor: replace RuntimeException with RssException (#753)
+* [MINOR] test: untracked files created in ShuffleFlushManagerTest (#745)
+* [#750] feat: add RssFetchFailedException (#751)
+* [#747] feat(tez): Add Tez Framework (#748)
+* [#736] feat(storage): best effort to write same hdfs file when no race
condition (#744)
+* [#719] feat(netty): Optimize allocation strategy (#739)
+* [#669] improvement: refresh application when reading memory data (#741)
+* [#133] feat(netty): Add Netty Utils (#727)
+* [MINOR] test: fix CoordinatorGrpcTest#shuffleServerHeartbeatTest on Linux
SSD platform (#738)
+* [#729] improvement: use `foreach` when iterate over Roaring64NavigableMap
for better performance (#730)
+* [#733] test: fix LocalStorageManagerTest#testGetLocalStorageInfo on Linux
SSD platform (#734)
+* Revert "[MINOR] test: fix tempdir leak in KerberizedHdfs tests (#721)" (#732)
+* [MINOR] log(client): update log level from INFO to DEBUG to avoid noisy
(#725)
+* [MINOR] chore(ci): test compile with java.version 11 and 17 (#728)
+* [#625] improvement: Package sun.security.krb5 is not visible in Java 11 and
17. (#726)
+* [#133] feat(netty): Add StreamServer. (#718)
+* [MINOR] test: test class name should end with Test (#724)
+* [MINOR] test: fix tempdir leak in KerberizedHdfs tests (#721)
+* [#711] feat(netty): Add Netty port information for Shuffle Server (#712)
+* [#564] test(operator): add end-to-end test (#581)
+* [#615] improvement: Reduce task binary by removing 'partitionToServers' from
RssShuffleHandle (#637)
+* [MINOR] Removed unused methods and variable (#702)
+* [#483] test: fix flaky test ShuffleServerFaultToleranceTest (#705)
+* [#708] test: do not assume hostname of hdfs mini-cluster (#709)
+* [MINOR] test: fix static field initialized before TempDir field injected
(#707)
+* [#80][Part-3] feat: add REST API for decommisson (#684)
+* [MINOR] fix: Add method `close` for ApplicationManager (#704)
+* [MINOR] improvement: Reduce the size of Spark patch (#699)
+* [#674] feat(docker): use JDK11 as the default java version in Dockerfile
(#683)
+* chore: remove unused log info (#700)
+* build(deps): bump golang.org/x/net in /deploy/kubernetes/operator (#676)
+* [#585] fix: avoid unbound variable errors in start-shuffle-server.sh
+* [#691] fix(test): flaky test CoordinatorMetricsTest#testCoordinatorMetrics
+* [#697] fix: use the naive equals method to avoid introducing additional
dependencies (#698)
+* [#585] feat(netty): Add MaxDirectMemorySize option for shuffle Server (#690)
+* [#397] docs: add the usage of AccessQuotaChecker (#692)
+* [MINOR] Use multithreading to detect multiple disks (#687)
+* [#678] improvement: Write hdfs files asynchronously when `detectStorage`
(#680)
+* [#645][Improvement] feat(operator): support manager parameter configuration
(#670)
+* [#671] feat(coordinator): Metrics of the number of apps submitted by users
(#672)
+* [#675] fix: filter no space exception in checkStorageReadAndWrite (#677)
+* [#80][Part-2] feat: Add RPC logic and heartbeat logic for decommisson (#663)
+* feat: introduce storage manager selector to support more selector strategy
(#621)
+* [MINOR] test: address unchecked conversions (#624)
+* [#642]feat(server): better default options for shuffle server (#662)
+* [#665] feat(client): keep consistent with vanilla spark when key or value is
null (#666)
+* [MINOR] docs: fix flaky-test-report.yml (#664)
+* [#408] test: fix memory check failure in
ShuffleBufferManagerTest#bufferSizeTest (#657)
+* [#612] test: cleanup shuffleServer instance for each test (#658)
+* [#659] fix(server): fix NPE of ShuffleDataFlushEvent's underStorage in some
cases (#660)
+* feat(coordinator): heap size configurable and add gc log (#656)
+* [MINOR] docs: update the project introduction in README (#653)
+* [#647][FOLLOWUP] set coordinator id before ApplicationMaster (#654)
+* [#649] test: remove mini-cluster in ClientConfManagerTest (#650)
+* [ISSUE-576] Increase the timeout and remove the initialization time (#646)
+* [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE (#644)
+* [#647] fix: Multiple coordinator produce conflicts when they delect the same
file (#648)
+* [#545][FOLLOWUP]feat(operator): support specifying custom affinity &
tolerations (#641)
+* [MINOR] refactor: address unchecked conversions (#623)
+* [#80][Part-1] feat: Add decommisson logic to shuffle server (#606)
+* [#630] feat(client): Disable the localShuffleReader by default in Spark3
client (#636)
+* [#632]fix: respect volumes in rss spec (#634)
+* [#600] chore(operator): change JDK base from openjdk to eclipse-temurin
(#617)
+* [#626][FOLLOWUP] chore(ci): fix typo in build.yml (#633)
+* [#627] fix(operator): support specifying custom ports (#629)
+* [#626] chore(ci): skip build operator if no code changes (#628)
+* [#580] chore(ci): disable parallel build in maven (#631)
+* feat: support downloads latest hadoop archives and mirror url (#619)
+* [FOLLOWUP] fix: don't recreate base dir if it's already existed (#616)
+* [#551] docs: update templates for flaky test and pull request (#588)
+* [MINOR] fix: allow `mountPoint` not containing '/'
+* [#613] test: SimpleClusterManagerTest#updateExcludeNodesTest (#614)
