On Tue, Jan 28, 2020 at 1:55 PM Dilip Kumar <dilipbal...@gmail.com> wrote: > > On Tue, Jan 28, 2020 at 1:30 PM Amit Kapila <amit.kapil...@gmail.com> wrote: > > > > On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbal...@gmail.com> wrote: > > > > > > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapil...@gmail.com> > > > wrote: > > > > > > > > > > It seems to me that we need to add all of this new handling because > > > > > > while taking the decision whether to stream or not we don't know > > > > > > whether the txn has changes that can't be streamed. One idea to > > > > > > make > > > > > > it work is that we identify it while decoding the WAL. I think we > > > > > > need to set a bit in the insert/delete WAL record to identify if the > > > > > > tuple belongs to a toast relation. This won't add any additional > > > > > > overhead in WAL and reduce a lot of complexity in the logical > > > > > > decoding > > > > > > and also decoding will be efficient. If this is feasible, then we > > > > > > can > > > > > > do the same for speculative insertions. > > > > > The Idea looks good to me. I will work on this. > > > > > > > > > > > > > One more thing we can do is to identify whether the tuple belongs to > > > > toast relation while decoding it. However, I think to do that we need > > > > to have access to relcache at that time and that might add some > > > > overhead as we need to do that for each tuple. Can we investigate > > > > what it will take to do that and if it is better than setting a bit > > > > during WAL logging. > > > > > > IMHO, for the catalog scan, we will have to start/stop the transaction > > > for each change. So do you want that we should evaluate its > > > performance? > > > > > > > No, I was not thinking about each change, but at the level of > > ReorderBufferTXN. > That means we will have to keep that transaction open until we decode > the commit WAL for that ReorderBufferTXN or you have anything else in > mind? >
or probably till we start streaming. > > > > > Also, during we get the change we might not have the > > > complete historic snapshot ready to fetch the rel cache entry. > > > > > > > Before decoding each change (say DecodeInsert), we call > > SnapBuildProcessChange. Isn't that sufficient? > Yeah, Right, we can get some recache entry based on the base snapshot. > And, that might be sufficient to know whether it's a toast relation or > not. > > > > Even, if the above is possible, I am not sure how good is it for each > > change we fetch rel cache entry, that is the point I was worried. > > We might not need to scan the catalog every time, we might get it from > the cache itself. > Right, but I am not completely sure if that is better than setting a bit in WAL record for toast tuples. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
