This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new f6f8e657de Enhance README with AI-native data storage details (#4369)
f6f8e657de is described below
commit f6f8e657de0ca447f6819249e521f5653ff33fcd
Author: Liang Chen <[email protected]>
AuthorDate: Mon Oct 6 18:46:52 2025 +0800
Enhance README with AI-native data storage details (#4369)
Expanded on the AI-native data storage capabilities and challenges in data
preparation for CarbonData.
---
README.md | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index f138497305..d2f3c8a52e 100644
--- a/README.md
+++ b/README.md
@@ -17,8 +17,16 @@
<img src="/docs/images/CarbonData_logo.png" width="200" height="40">
-Apache CarbonData is an indexed columnar data store solution for fast
analytics on big data platform, e.g. Apache Hadoop, Apache Spark, etc.
+- AI-native data storage for CarbonData's new scope, In AI projects, data
scientists and engineers spend 80% of their time on data preparation.
Traditional storage presents numerous bottlenecks in this process:
+ - Data silos: Training data may be scattered across data lakes, data
warehouses, file systems, object storage, and other locations, making
integration difficult.
+ - Performance bottlenecks:Training phase: High-speed, low-latency data
throughput is required to feed GPUs to avoid expensive GPU resources sitting
idle.
+ - Inference phase: High-concurrency, low-latency vector similarity search
capabilities are required.
+ - Complex data formats: AI processes data types far beyond tables, including
unstructured data (images, videos, text, audio) and semi-structured data (JSON,
XML). Traditional databases have limited capabilities for processing and
querying such data.
+ - Lack of metadata management: The lack of effective management of rich
metadata such as data versions, lineage, annotation information, and
experimental parameters leads to poor experimental reproducibility.
+ - Vectorization requirements: Modern AI models (such as large language
models) convert all data into vector embeddings. Traditional storage cannot
efficiently store and retrieve high-dimensional vectors.
+
+- In the previous releases Apache CarbonData is an indexed columnar data store
solution for fast analytics on big data platform, e.g. Apache Hadoop, Apache
Spark, etc.
You can find the latest CarbonData document and learn more at:
[https://carbondata.apache.org](https://carbondata.apache.org/)
