That's also possible. But when data grows, eventually where clause might add some overhead. Keeping only unprocessed data separately might speed up the process.
On Mon, Nov 13, 2017 at 6:27 PM, Sajith Ravindra <[email protected]> wrote: > > <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> > > On Mon, Nov 13, 2017 at 5:43 PM, Gihan Anuruddha <[email protected]> wrote: > >> Keeping a last processed data set in a separate table might be a valid >> use case for some scenario. Users can run an one job per day and gather >> last process data and could do some operations. This avoids the overhead to >> go through the entire data set. >> > > @Gihan, Gimanatha: But, is this a direct or intended use case of using > these two construts together. Isn't it more of a by product? > > If some user wants to process only subset of data wouldn't the user use a > "WHERE" clause? > >> >> On Mon, Nov 13, 2017 at 4:57 PM, Gimantha Bandara <[email protected]> >> wrote: >> >>> Hi, >>> >>> I don't think we have a valid use case of using OVERWRITE with >>> incremental processing unless we want to keep the last processed data set >>> in a separate table. >>> >>> On Mon, Nov 13, 2017 at 4:30 PM, Sajith Ravindra <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> Can there be a valid use case where we do an INSERT OVERWRITE to a >>>> table and then do incremental table commit? >>>> >>>> AFAIU, it leads to loss of data as incremental processing processes >>>> only unprocessed data and OVERWRITE statement drops previously processed >>>> data. >>>> >>>> Please advice. >>>> >>>> Thanks >>>> *,Sajith Ravindra* >>>> Associate Technical Lead >>>> WSO2 Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> mobile: +94 77 2273550 >>>> blog: http://sajithr.blogspot.com/ >>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>> >>> >>> >>> >>> -- >>> Gimantha Bandara >>> Software Engineer >>> WSO2. Inc : http://wso2.com >>> Mobile : +94714961919 <071%20496%201919> >>> >> >> >> >> -- >> W.G. Gihan Anuruddha >> Associate Technical Lead | WSO2, Inc. >> M: +94772272595 >> > > -- W.G. Gihan Anuruddha Associate Technical Lead | WSO2, Inc. M: +94772272595
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
