Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

The Design is too complex and we may spend to much time and people to develop it. Can we simplify it and just support streaming first? -- View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-New-Feature-Streaming-Ingestion

Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

Hi Jacky, Thanks for your comments. I guess i should have uploaded in google doc format instead of pdf, somehow google doc messes up all the diagrams if I copy paste and i have not figured the way to fix it. Anyway, I apologize for the inconvenience for those wanted to add in-line comments in the

Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

ppend > > offsets and metadata information. > > > > Phase -3 : Add support for external streaming frameworks such as Kafka > > streaming using spark structured steaming, maintain > > topics/partitions/offsets and support fault tolerance . > > > > Phase-4 :

Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

Hi Aniket, Comment inline And I have put some review comment in the PDF here: https://drive.google.com/file/d/0B5vjWGChUwXdSUV0OTFkTGE4am8/view?usp=sharing > 在 2017年3月29日，上午7:10，Aniket Adnaik

Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

titions/offsets and support fault tolerance . > > Phase-4 : Add support to other streaming frameworks , such as flink , beam > etc. > > Phase-5: Future support for in-memory cache for buffering streaming data, > support for union with Spark Structured streaming to serve d

Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

Hi Jacky, Please see my comments below; 1. In this phase, is it still using columnar format? Save to a file for every mini batch? If so, it is only readable after the file has been closed and some metadata need to be kept to indicate the availability of the new file. AA >> yes, for initial phase

Re: [DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

Hi Aniket, This feature looks great, the overall plan also seems fine to me. Thanks for proposing it. And I have some doubts inline. > 在 2017年3月27日，下午6:34，Aniket Adnaik 写道： > > Hi All, > > I would like to open up a discussion for new feature to support streaming >

[DISCUSSION]: (New Feature) Streaming Ingestion into CarbonData

Hi All, I would like to open up a discussion for new feature to support streaming ingestion in CarbonData. Please refer to design document(draft) in the link below. https://drive.google.com/file/d/0B71_EuXTdDi8MlFDU2tqZU9BZ3M /view?usp=sharing Your comments/suggestions are welcome. Here