Re: [SUGGESTION]Support Decoder based fallback mechanism in local dictionary

2018-08-27 Thread akashrn5
As of now i will code as user property, and we can take desicion once we get the performance report with this. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [DISCUSSION] Implement file-level Min/Max index for streaming segment

2018-08-27 Thread manish gupta
+1 Regards Manish Gupta On Mon, 27 Aug 2018 at 9:47 AM, Kumar Vishal wrote: > +1 > Regards > Kumar Vishal > > On Mon, 27 Aug 2018 at 07:15, xm_zzc <441586...@qq.com> wrote: > > > +1. > > > > > > > > -- > > Sent from: > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [SUGGESTION]Support Decoder based fallback mechanism in local dictionary

2018-08-27 Thread manish gupta
+1 @Akash..I suggest not to expose any property to the user for this. The design should support this decision based on the property but to expose it to the end user, this decision can be taken once you complete your performance testing. Regards Manish Gupta On Mon, 27 Aug 2018 at 1:57 PM, Kumar

Re: [SUGGESTION]Support Decoder based fallback mechanism in local dictionary

2018-08-27 Thread Kumar Vishal
+1 @ xuchuanyin This will not impact data map writing flow as actual column page will be cleared only after consuming all the records by data map writer, there will not be any change in that area. -Regards Kumar Vishal , On Mon, Aug 27, 2018 at 1:01 PM xuchuanyin wrote: > This means, no need

Re: [SUGGESTION]Support Decoder based fallback mechanism in local dictionary

2018-08-27 Thread xuchuanyin
This means, no need to keep the actual data along with encoded data in encoded column page. --- A problem is that, currently index datamap needs the actual data to generate index. You may affect this procedure if you do not keep the actual data. -- Sent from:

[SUGGESTION]Support Decoder based fallback mechanism in local dictionary

2018-08-27 Thread Akash Nilugal
Hi all, Currently, when the fallback is initiated for a column page in case of local dictionary, we are keeping both encoded data and actual data in memory and then we form the new column page without dictionary encoding and then at last we free the Encoded Column Page. Because of this offheap