Agree, CarbonData was focus on bigdata before, and only has very less integration with AI, such as PyCarbon, which support PyTorch and TensorFlow read data from CarbonData. AI is very popular recently, and has many customer need unified data format and storage for bigdata and AI. I suggest: 1. support developer tools integrade CarbonData, such as jupyter notebook and zepplin, 2. improve usability of CarbonData, such as support run CarbonData on docker and kubernetes easily 3.support/enhance different AI framework integrate CarbonData, such as TensorFlow/PyTorch/Ray
I hope CarbonData can become unified data format and datastore for bigdata,warehouse and AI, User can use the same data with CarbonData in different compute engine,such as spark/flink/tensorflow/Pytorch On 2023/10/19 07:58:14 Liang Chen wrote: > As you know, Carbondata as datastore and dataformat already be quite good > and mature. > I want to create the thread via mailing list to open discuss what are the > next milestones of carbondata project？ > One proposal from my side: we should consider how to integrate with AI > computing engine？ > > Regards > Liang >
