Keeping a last processed data set in a separate table might be a valid use case for some scenario. Users can run an one job per day and gather last process data and could do some operations. This avoids the overhead to go through the entire data set.
On Mon, Nov 13, 2017 at 4:57 PM, Gimantha Bandara <giman...@wso2.com> wrote: > Hi, > > I don't think we have a valid use case of using OVERWRITE with incremental > processing unless we want to keep the last processed data set in a separate > table. > > On Mon, Nov 13, 2017 at 4:30 PM, Sajith Ravindra <saji...@wso2.com> wrote: > >> Hi all, >> >> Can there be a valid use case where we do an INSERT OVERWRITE to a table >> and then do incremental table commit? >> >> AFAIU, it leads to loss of data as incremental processing processes only >> unprocessed data and OVERWRITE statement drops previously processed data. >> >> Please advice. >> >> Thanks >> *,Sajith Ravindra* >> Associate Technical Lead >> WSO2 Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 77 2273550 >> blog: http://sajithr.blogspot.com/ >> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >> > > > > -- > Gimantha Bandara > Software Engineer > WSO2. Inc : http://wso2.com > Mobile : +94714961919 <071%20496%201919> > -- W.G. Gihan Anuruddha Associate Technical Lead | WSO2, Inc. M: +94772272595
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
