+1 for this :
-------------------------
May be we can have a try, after all it will be just one interface 
implementation for dictionary generation. We can have multiple 
implementations and then decide based on optimal performance. 

Regards
Liang


Ravindra Pesala wrote
> Hi Jacky/Jihong,
> 
> I agree that new dictionary values are less in case of incremental data
> load but that is completely depends on user data scenarios.  In some
> user scenarios new dictionary values may be more we cannot overrule that.
> And also for users convenience we should provide single pass solution with
> out insisting them to run external tool first. We can provide the option
> to
> run external tool first and provide dictionary to improve performance.
> 
> My opinion is better to use some professional distributed map like
> Hazlecast than Zookeeper + HDFS.  It is lite weight and does not require
> to
> have separate cluster, it can form the cluster within the executor jvm's .
> May be we can have a try, after all it will be just one interface
> implementation for dictionary generation. We can have multiple
> implementations and then decide based on optimal performance.
> 
> Regards,
> Ravi
> 
> On 15 October 2016 at 10:50, Jacky Li <

> jacky.likun@

> > wrote:
> 
>> Hi,
>>
>> I can offer one more approach for this discussion, since new dictionary
>> values are rare in case of incremental load (ensure first load having as
>> much dictionary value as possible), so synchronization should be rare. So
>> how about using Zookeeper + HDFS file to provide this service. This is
>> what
>> carbon is doing today, we can wrap Zookeeper + HDFS to provide the global
>> dictionary interface.
>> It has the benefit of
>> 1. automated: without bordering the user
>> 2. not introducing more dependency: we already using zookeeper and HDFS.
>> 3. performance? since new dictionary value and synchronization is rare.
>>
>> What do you think?
>>
>> Regards,
>> Jacky
>>
>> > 在 2016年10月15日,上午2:38,Jihong Ma <

> Jihong.Ma@

> > 写道:
>> >
>> > Hi Ravi,
>> >
>> > The major concern I have for generating global dictionary from scratch
>> with a single scan is performance, the way to handle an occasional update
>> to the dictionary is way simpler and cost effective in terms of
>> synchronization cost and refresh the global/local cache copy.
>> >
>> > There are a lot to worry about for distributed map, and leveraging KV
>> store is overkill if simply just for dictionary generation.
>> >
>> > Regards.
>> >
>> > Jihong
>> >
>> > -----Original Message-----
>> > From: Ravindra Pesala [mailto:

> ravi.pesala@

> ]
>> > Sent: Friday, October 14, 2016 11:03 AM
>> > To: dev
>> > Subject: Re: Discussion(New feature) regarding single pass data loading
>> solution.
>> >
>> > Hi Jihong,
>> >
>> > I agree, we can use external tool for first load, but for incremental
>> load
>> > we should have solution to add global dictionary. So this solution
>> should
>> > be enough to generate global dictionary even if user does not use
>> external
>> > tool for first time. That solution could be distributed map or KV
>> store.
>> >
>> > Regards,
>> > Ravi.
>> >
>> > On 14 October 2016 at 23:12, Jihong Ma <

> Jihong.Ma@

> > wrote:
>> >
>> >> Hi Liang,
>> >>
>> >> This tool is more or less like the first load, the first time after
>> table
>> >> is created, any subsequent loads/incremental loads will proceed and is
>> >> capable of updating the global dictionary when it encounters new
>> value,
>> >> this is easiest way of achieving 1 pass data loading process without
>> too
>> >> much overhead.
>> >>
>> >> Since this tool is only triggered once per table, not considered too
>> much
>> >> burden on the end users. Making global dictionary generation out of
>> the
>> way
>> >> of regular data loading is the key here.
>> >>
>> >> Jihong
>> >>
>> >> -----Original Message-----
>> >> From: Liang Chen [mailto:

> chenliang6136@

> ]
>> >> Sent: Thursday, October 13, 2016 5:39 PM
>> >> To: 

> dev@.apache

>> >> Subject: RE: Discussion(New feature) regarding single pass data
>> loading
>> >> solution.
>> >>
>> >> Hi jihong
>> >>
>> >> I am not sure that users can accept to use extra tool to do this work,
>> >> because provide tool or do scan at first time per table for most of
>> global
>> >> dict are same cost from users perspective, and maintain the dict file
>> also
>> >> be same cost, they always expecting that system can automatically and
>> >> internally generate dict file during loading data.
>> >>
>> >> Can we consider this:
>> >> first load: make scan to generate most of global dict file, then copy
>> this
>> >> file to each load node for subsequent loading
>> >>
>> >> Regards
>> >> Liang
>> >>
>> >>
>> >> Jihong Ma wrote
>> >>>>>>> the question is what would be the default implementation? Load
>> data
>> >> without dictionary?
>> >>>
>> >>> My thought is we can provide a tool to generate global dictionary
>> using
>> >>> sample data set, so the initial global dictionaries is available
>> before
>> >>> normal data loading. We shall be able to perform encoding based on
>> that,
>> >>> we only need to handle occasionally adding entries while loading. For
>> >>> columns specified with global dictionary encoding, but dictionary is
>> not
>> >>> placed before data loading, we error out and direct user to use the
>> tool
>> >>> first.
>> >>>
>> >>> Make sense?
>> >>>
>> >>> Jihong
>> >>>
>> >>> -----Original Message-----
>> >>> From: Ravindra Pesala [mailto:
>> >>
>> >>> ravi.pesala@
>> >>
>> >>> ]
>> >>> Sent: Thursday, October 13, 2016 1:12 AM
>> >>> To: dev
>> >>> Subject: Re: Discussion(New feature) regarding single pass data
>> loading
>> >>> solution.
>> >>>
>> >>> Hi Jihong/Aniket,
>> >>>
>> >>> In the current implementation of carbondata we are already handling
>> >>> external dictionary while loading the data.
>> >>> But here the question is what would be the default implementation?
>> Load
>> >>> data with out dictionary?
>> >>>
>> >>>
>> >>> Regards,
>> >>> Ravi
>> >>>
>> >>> On 13 October 2016 at 03:50, Aniket Adnaik <
>> >>
>> >>> aniket.adnaik@
>> >>
>> >>> > wrote:
>> >>>
>> >>>> Hi Ravi,
>> >>>>
>> >>>> 1. I agree with Jihong that creation of global dictionary should be
>> >>>> optional, so that it can be disabled to improve the load
>> performance.
>> >>>> User
>> >>>> should be made aware that using global dictionary may boost the
>> query
>> >>>> performance.
>> >>>> 2. We should have a generic interface to manage global dictionary
>> when
>> >>>> its
>> >>>> from external sources. In general, it is not a good idea to depend
>> on
>> >> too
>> >>>> many external tools.
>> >>>> 3. May be we should allow user to generate global dictionary
>> separately
>> >>>> through SQL command or similar. Something like materialized view.
>> This
>> >>>> means carbon should avoid using local dictionary and do late
>> >>>> materialization when global dictionary is present.
>> >>>> 4. May be we should think of some ways to create global dictionary
>> >> lazily
>> >>>> as we serve SELECT queries. Implementation may not be that straight
>> >>>> forward. Not sure if its worth the effort.
>> >>>>
>> >>>> Best Regards,
>> >>>> Aniket
>> >>>>
>> >>>>
>> >>>> On Tue, Oct 11, 2016 at 7:59 PM, Jihong Ma <
>> >>
>> >>> Jihong.Ma@
>> >>
>> >>> > wrote:
>> >>>>
>> >>>>>
>> >>>>> A rather straight option is allow user to supply global dictionary
>> >>>>> generated somewhere else or we build a separate tool just for
>> >>>> generating
>> >>>> as
>> >>>>> well updating dictionary. Then the general normal data loading
>> process
>> >>>> will
>> >>>>> encode columns with local dictionary if not supplied.  This should
>> >>>> cover
>> >>>>> majority of cases for low-medium cardinality column. For the cases
>> we
>> >>>> have
>> >>>>> to incorporate online dictionary update, use a lock mechanism to
>> sync
>> >>>> up
>> >>>>> should serve the purpose.
>> >>>>>
>> >>>>> In another words, generating global dictionary is an optional step,
>> >>>> only
>> >>>>> triggered when needed, not a default step as we do currently.
>> >>>>>
>> >>>>> Jihong
>> >>>>>
>> >>>>> -----Original Message-----
>> >>>>> From: Ravindra Pesala [mailto:
>> >>
>> >>> ravi.pesala@
>> >>
>> >>> ]
>> >>>>> Sent: Tuesday, October 11, 2016 2:33 AM
>> >>>>> To: dev
>> >>>>> Subject: Discussion(New feature) regarding single pass data loading
>> >>>>> solution.
>> >>>>>
>> >>>>> Hi All,
>> >>>>>
>> >>>>> This discussion is regarding single pass data load solution.
>> >>>>>
>> >>>>> Currently data is loading to carbon in 2 pass/jobs
>> >>>>> 1. Generating global dictionary using spark job.
>> >>>>> 2. Encode the data with dictionary values and create carbondata
>> >> files.
>> >>>>> This 2 pass solution has many disadvantages like it needs to read
>> the
>> >>>> data
>> >>>>> twice in case of csv files input or it needs to execute dataframe
>> >> twice
>> >>>> if
>> >>>>> data is loaded from dataframe.
>> >>>>>
>> >>>>> In order to overcome from above issues of 2 pass dataloading, we
>> can
>> >>>> have
>> >>>>> single pass dataloading and following are the alternate solutions.
>> >>>>>
>> >>>>> Use local dictionary
>> >>>>> Use local dictionary for each carbondata file while loading data,
>> but
>> >>>> it
>> >>>>> may lead to query performance degradation and more memory
>> footprint.
>> >>>>>
>> >>>>> Use KV store/distributed map.
>> >>>>> *HBase/Cassandra cluster : *
>> >>>>>  Dictionary data would be stored in KV store and generates the
>> >>>> dictionary
>> >>>>> value if it is not present in it. We all know the pros/cons of
>> Hbase
>> >>>> but
>> >>>>> following are few.
>> >>>>>  Pros : These are apache licensed
>> >>>>>         Easy to implement to store/retreive dictionary values.
>> >>>>>         Performance need to be evaluated.
>> >>>>>
>> >>>>>  Cons : Need to maintain seperate cluster for maintaining global
>> >>>>> dictionary.
>> >>>>>
>> >>>>> *Hazlecast distributed map : *
>> >>>>>  Dictionary data could be saved in distributed concurrent hash map
>> of
>> >>>>> hazlecast. It is in-memory map and partioned as per number of
>> nodes.
>> >>>> And
>> >>>>> even we can maintain the backups using sync/async functionality to
>> >>>> avoid
>> >>>>> the data loss when instance is down. We no need to maintain
>> seperate
>> >>>>> cluster for it as it can run on executor jvm itself.
>> >>>>>  Pros: It is apache licensed.
>> >>>>>        No need to maintain seperate cluster as instances can run in
>> >>>>> executor jvms.
>> >>>>>        Easy to implement and store/retreive dictionary values.
>> >>>>>        It is pure java implementation.
>> >>>>>        There is no master/slave concept and no single point
>> failure.
>> >>>>>
>> >>>>>  Cons: Performance need to be evaluated.
>> >>>>>
>> >>>>> *Redis distributed map : *
>> >>>>>    It is also in-memory map but it is coded in c language so we
>> >> should
>> >>>>> have java client libraries to interact with redis. Need to maintain
>> >>>>> seperate cluster for it. It also can partition the data.
>> >>>>>  Pros : More feature rich than Hazlecast.
>> >>>>>         Easy to implement and store/retreive dictionary values.
>> >>>>>  Cons : Need to maintain seperate cluster for maintaining global
>> >>>>> dictionary.
>> >>>>>         May not be suitable for big data stack.
>> >>>>>         It is BSD licensed (Not sure whether we can use or not)
>> >>>>>  Online performance figures says it is little slower than
>> hazlecast.
>> >>>>>
>> >>>>> Please let me know which would be best fit for our loading
>> solution.
>> >>>> And
>> >>>>> please add any other suitable solution if I missed.
>> >>>>> --
>> >>>>> Thanks & Regards,
>> >>>>> Ravi
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Thanks & Regards,
>> >>> Ravi
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context: http://apache-carbondata-
>> >> mailing-list-archive.1130556.n5.nabble.com/Discussion-New-
>> >> feature-regarding-single-pass-data-loading-solution-tp1761p1887.html
>> >> Sent from the Apache CarbonData Mailing List archive mailing list
>> archive
>> >> at Nabble.com.
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Ravi
>>
>>
>>
>>
> 
> 
> -- 
> Thanks & Regards,
> Ravi





--
View this message in context: 
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussion-New-feature-regarding-single-pass-data-loading-solution-tp1761p1984.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at 
Nabble.com.

Reply via email to