Abdullah, Do you think that issue is somehow related to the patch that you proposed? https://issues.apache.org/jira/browse/ASTERIXDB-1302 <https://issues.apache.org/jira/browse/ASTERIXDB-1302> It does not involve duplicates per se, but has the same issue with feed lock management.
> On Jan 7, 2016, at 09:12, abdullah alamoudi <[email protected]> wrote: > > So I have pushed the new change (Still under review). > > @Vignesh, > This change has the fix for the twitter adapter. > > @Jianfeng, > This change has the fix for the duplicate key issue. > > It can be checked out using > git fetch https://asterix-gerrit.ics.uci.edu/asterixdb > refs/changes/74/574/2 && git checkout FETCH_HEAD > > Amoudi, Abdullah. > > On Thu, Jan 7, 2016 at 7:42 PM, abdullah alamoudi <[email protected]> > wrote: > >> Found the issue. It had nothing to do with any of the three components. >> >> The problem was that: >>> 1. we remove the tuple causing the exception. >> didn't work as I expected and instead it removed all the tuples upto the >> offending record. >> I have fixed it and now it works like a charm. >> >> Pushing the fix in a few minutes. >> Abdullah. >> >> Amoudi, Abdullah. >> >> On Thu, Jan 7, 2016 at 6:16 PM, Mike Carey <[email protected]> wrote: >> >>> The general transaction handling for such an exception wrt locking and >>> aborts probably assumes that total bailouts are the answer. Thus, it may >>> leave messes that rollbacks are otherwise the answer to. Feeds and >>> transactions don't mix super well, it seems.... Watching how duplicate >>> keys work for insert from query statements may help you debug. If we >>> change >>> things to allow those to succeed for all non duplicate keys - which might >>> make more sense for that anyway. >>> On Jan 7, 2016 5:48 AM, "abdullah alamoudi" <[email protected]> wrote: >>> >>>> Today, as I was working on fixing handling of duplicate keys with feeds, >>>> everything seemed to work fine. here is what we do when we encounter a >>>> duplicate key exception. >>>> >>>> 1. we remove the tuple causing the exception. >>>> 2. we continue from where we stopped. >>>> >>>> The problem is that when I try to query the dataset after that to check >>>> and see which records made it into the dataset, I get a deadlock. >>>> >>>> I have looked at the stack trace (attached) and I think the threads in >>> the >>>> file are the relevant ones. Please, have a look and let me know if you >>> have >>>> a possible cause in mind. >>>> >>>> The threads are related to: >>>> 1. BufferCache. >>>> 2. Logging. >>>> 3. Locking. >>>> >>>> Let me know what you think. I can reproduce this bug. it happened on >>> 100% >>>> of my test runs. >>>> >>>> I will let you know when I solve it but it is taking longer than I >>> thought. >>>> >>>> Amoudi, Abdullah. >>>> >>> >> >> Best regards, Ildar
