Re: Discussion : Propose CarbonData project to consider as AI-nativedata storage

[Jacky Li]

Hi Liang,


Totally agree, especially the train data preparation requires processing 
multi-modal data type, which is very different from big data engine like Apache 
Hive and Spark.
In my opinion, not only the data type, data format is different but also the 
data catalog support is different. The data catalog for AI needs to support 
free-style data files and folder orgnization instead of tranditional database 
and table, schema support.


If community is interest in driving CarbonData towards this direction, we can 
discuss more detail and draft a plan for the community.


Regards,
Jacky Li




Jacky Li
jacky.li...@qq.com



        



         原始邮件
         
       
发件人：Liang Chen <chenliang...@apache.org&gt;
发件时间：2025年8月21日 19:23
收件人：dev <dev@carbondata.apache.org&gt;
主题：Re: Discussion : Propose CarbonData project to consider as AI-nativedata 
storage



       
For&nbsp;AI,&nbsp;require&nbsp;multi&nbsp;type&nbsp;high&nbsp;efficient&nbsp;storage.

Liang&nbsp;Chen&nbsp;<chenliang...@apache.org&gt;&nbsp;于2025年8月21日周四&nbsp;10:41写道：

&gt;&nbsp;Dear&nbsp;Dev
&gt;
&gt;&nbsp;I&nbsp;propose&nbsp;the&nbsp;CarbonData&nbsp;project&nbsp;to&nbsp;consider&nbsp;as&nbsp;AI-native&nbsp;data&nbsp;storage,
&gt;&nbsp;the&nbsp;new&nbsp;journey&nbsp;is&nbsp;super&nbsp;suitable&nbsp;for&nbsp;carbondata.
&gt;
&gt;&nbsp;What&nbsp;is&nbsp;AI-native&nbsp;data&nbsp;storage
&gt;
&gt;&nbsp;AI-native&nbsp;data&nbsp;storage&nbsp;is&nbsp;a&nbsp;data&nbsp;storage&nbsp;and&nbsp;management&nbsp;system&nbsp;designed
&gt;&nbsp;and&nbsp;built&nbsp;specifically&nbsp;for&nbsp;the&nbsp;needs&nbsp;of&nbsp;artificial&nbsp;intelligence&nbsp;(AI)
&gt;&nbsp;workloads,&nbsp;particularly&nbsp;machine&nbsp;learning&nbsp;and&nbsp;deep&nbsp;learning.&nbsp;Its&nbsp;core
&gt;&nbsp;concept&nbsp;is&nbsp;to&nbsp;transform&nbsp;data&nbsp;storage&nbsp;from&nbsp;a&nbsp;passive,&nbsp;isolated&nbsp;component&nbsp;of
&gt;&nbsp;the&nbsp;AI&nbsp;process&nbsp;into&nbsp;an&nbsp;active,&nbsp;intelligent,&nbsp;and&nbsp;deeply&nbsp;integrated
&gt;&nbsp;infrastructure.
&gt;&nbsp;Why&nbsp;AI-native&nbsp;data&nbsp;storage&nbsp;for&nbsp;CarbonData's&nbsp;new&nbsp;scope
&gt;
&gt;&nbsp;In&nbsp;AI&nbsp;projects,&nbsp;data&nbsp;scientists&nbsp;and&nbsp;engineers&nbsp;spend&nbsp;80%&nbsp;of&nbsp;their&nbsp;time&nbsp;on
&gt;&nbsp;data&nbsp;preparation.&nbsp;Traditional&nbsp;storage&nbsp;presents&nbsp;numerous&nbsp;bottlenecks&nbsp;in&nbsp;this
&gt;&nbsp;process:
&gt;
&gt;&nbsp;Data&nbsp;silos:&nbsp;Training&nbsp;data&nbsp;may&nbsp;be&nbsp;scattered&nbsp;across&nbsp;data&nbsp;lakes,&nbsp;data
&gt;&nbsp;warehouses,&nbsp;file&nbsp;systems,&nbsp;object&nbsp;storage,&nbsp;and&nbsp;other&nbsp;locations,&nbsp;making
&gt;&nbsp;integration&nbsp;difficult.
&gt;
&gt;&nbsp;Performance&nbsp;bottlenecks:
&gt;
&gt;&nbsp;Training&nbsp;phase:&nbsp;High-speed,&nbsp;low-latency&nbsp;data&nbsp;throughput&nbsp;is&nbsp;required&nbsp;to
&gt;&nbsp;feed&nbsp;GPUs&nbsp;to&nbsp;avoid&nbsp;expensive&nbsp;GPU&nbsp;resources&nbsp;sitting&nbsp;idle.
&gt;
&gt;&nbsp;Inference&nbsp;phase:&nbsp;High-concurrency,&nbsp;low-latency&nbsp;vector&nbsp;similarity&nbsp;search
&gt;&nbsp;capabilities&nbsp;are&nbsp;required.
&gt;
&gt;&nbsp;Complex&nbsp;data&nbsp;formats:&nbsp;AI&nbsp;processes&nbsp;data&nbsp;types&nbsp;far&nbsp;beyond&nbsp;tables,&nbsp;including
&gt;&nbsp;unstructured&nbsp;data&nbsp;(images,&nbsp;videos,&nbsp;text,&nbsp;audio)&nbsp;and&nbsp;semi-structured&nbsp;data
&gt;&nbsp;(JSON,&nbsp;XML).&nbsp;Traditional&nbsp;databases&nbsp;have&nbsp;limited&nbsp;capabilities&nbsp;for&nbsp;processing
&gt;&nbsp;and&nbsp;querying&nbsp;such&nbsp;data.
&gt;
&gt;&nbsp;Lack&nbsp;of&nbsp;metadata&nbsp;management:&nbsp;The&nbsp;lack&nbsp;of&nbsp;effective&nbsp;management&nbsp;of&nbsp;rich
&gt;&nbsp;metadata&nbsp;such&nbsp;as&nbsp;data&nbsp;versions,&nbsp;lineage,&nbsp;annotation&nbsp;information,&nbsp;and
&gt;&nbsp;experimental&nbsp;parameters&nbsp;leads&nbsp;to&nbsp;poor&nbsp;experimental&nbsp;reproducibility.
&gt;
&gt;&nbsp;Vectorization&nbsp;requirements:&nbsp;Modern&nbsp;AI&nbsp;models&nbsp;(such&nbsp;as&nbsp;large&nbsp;language
&gt;&nbsp;models)&nbsp;convert&nbsp;all&nbsp;data&nbsp;into&nbsp;vector&nbsp;embeddings.&nbsp;Traditional&nbsp;storage&nbsp;cannot
&gt;&nbsp;efficiently&nbsp;store&nbsp;and&nbsp;retrieve&nbsp;high-dimensional&nbsp;vectors.
&gt;
&gt;
&gt;&nbsp;Regards
&gt;
&gt;&nbsp;Liang
&gt;