Hi Liang，
AI technology has broad prospects, and large model technology is in full
swing
If we combine AI technology to automatically tune Carbon's parameters,
including some predictions, we will become more user-friendly
It's very visionary to exert force here。
On 2023/10/19
Hi Liang,
Agree on your point, to get CarbonData integrated with AI and Machine
Learning, which could help with predictive analytics and also automated
data cleaning.
Some other potential features that we could consider for next roadmap could
be Data versioning, TimeTravel and upgrading Spark,
Hey Liang and Xu Bo,
AI seems to be a good direction to move forward.
Ray is also a good option to integrate Carbondata with. It is getting quite
popular and has a strong place in the ML stack.
I suggest upgrading to newer spark versions as they have many good features
for AI/ML.
Also we should
Agree, CarbonData was focus on bigdata before, and only has very less
integration with AI, such as PyCarbon, which support PyTorch and TensorFlow
read data from CarbonData. AI is very popular recently, and has many customer
need unified data format and storage for bigdata and AI.
I suggest:
As you know, Carbondata as datastore and dataformat already be quite good
and mature.
I want to create the thread via mailing list to open discuss what are the
next milestones of carbondata project？
One proposal from my side: we should consider how to integrate with AI
computing engine？
Regards
