[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169160#comment-17169160 ] Damian Warszawski commented on ATLAS-3398: -- Thanks [~amestry]. Of course, I don't mind. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Assignee: Ashutosh Mestry >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168967#comment-17168967 ] Ashutosh Mestry commented on ATLAS-3398: [~dwarszawski] I am not able to assign this issue to you. Your name does not show up in the list for some reason. I have assigned it to me for tracking. Hope that is OK. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Assignee: Ashutosh Mestry >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168314#comment-17168314 ] ASF subversion and git services commented on ATLAS-3398: Commit 20127459a59fd0a64966046df3a1b0f64363c4f7 in atlas's branch refs/heads/branch-2.0 from Damian Warszawski [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=2012745 ] ATLAS-3398: introduced consitency-lock configuration to avoid multiple entities with same unique attribute value Signed-off-by: Madhan Neethiraj (cherry picked from commit 80efe92630a9198ed4b4c564f6e78c8079045930) > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168312#comment-17168312 ] ASF subversion and git services commented on ATLAS-3398: Commit 80efe92630a9198ed4b4c564f6e78c8079045930 in atlas's branch refs/heads/master from Damian Warszawski [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=80efe92 ] ATLAS-3398: introduced consitency-lock configuration to avoid multiple entities with same unique attribute value Signed-off-by: Madhan Neethiraj > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168268#comment-17168268 ] Damian Warszawski commented on ATLAS-3398: -- [~mad...@apache.org], [~amestry] uploaded another patch to fix the unit tests. Please have a look on it. Thanks. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161884#comment-17161884 ] Damian Warszawski commented on ATLAS-3398: -- According to our observations that can be a race condition between hive-hook (kafka event) and profiler (org.apache.atlas:atlas-client-v2:2.0.0). It can be also the case with atlas-client which make implicit retries while calling Atlas API. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161588#comment-17161588 ] Ashutosh Mestry commented on ATLAS-3398: I suppose the way to duplicate would be to attempt to create entities via REST APIs? Test case would be to have entity payload with same _qualifiedName_ but different GUIDs and execute them at the same moment. > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161569#comment-17161569 ] Damian Warszawski commented on ATLAS-3398: -- Optional configuration to support locks on JanusGraph to ensure data consitency -> [https://reviews.apache.org/r/72695/] > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Labels: integrity > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
