Re: [Discussion] Merging carbonindex files for each segments and across segments

This feature will be released in 1.4.1 -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Discussion] Merging carbonindex files for each segments and across segments

Hi Dev, Currently, Merge index feature is not complete and stable. It has some gaps also, for some of the features like pre-aggregate and streaming, merge index was not supported when it was implemented. We were not able to stabilize and use this feature then. With this discussion, will again

Re: [Discussion] Merging carbonindex files for each segments and across segments

Yes, Jin Zhou. Merge all index files to one in a segment would be useful feature. it would significantly improve query performance. Regards Liang Jin Zhou wrote > Hi, ravipesala > > Thank you for your proposal, merging index file is a very useful feature > as > we have already met serious

回复： [Discussion] Merging carbonindex files for each segments and across segments

A very good feature! I think case 1 and case 2 can be handle We can merge data files and index files after we insert into hdfs automaticlly. In case 1: if the data files are not small, there will be 100 data files and 1 index file. if the data files are small, there will be

Re: [Discussion] Merging carbonindex files for each segments and across segments

If we already have many carbonindex files in cluster, how to merge them, any tool or command will be available ? or we need to reload the data. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Discussion] Merging carbonindex files for each segments and across segments

Hi Ravindra, I doubt whether Level 2 merge is required, if the intention is to solve problem of case 2, user can perform data compaction, so that both data and index will be merged using level 1 merge. So it can avoid both small data file and small index file, right? Regards, Jacky Li > 在

Re: [Discussion] Merging carbonindex files for each segments and across segments

+1 for this proposal and solution, thanks, Ravi Regards Liang 2017-10-20 19:13 GMT+05:30 Ravindra Pesala : > Hi, > > Problem : > The first-time query of carbon becomes very slow. It is because of reading > many small carbonindex files and cache to the driver at the first

[Discussion] Merging carbonindex files for each segments and across segments

Hi, Problem : The first-time query of carbon becomes very slow. It is because of reading many small carbonindex files and cache to the driver at the first time. Many carbonindex files are created in two cases Case 1: Loading data in large cluster For example, if the cluster size is 100