Re: Review Request 74954: ATLAS-4846 : Source the atlas-env file in ATLAS import script to get all the argument
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74954/ --- (Updated April 12, 2024, 5:58 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4846 https://issues.apache.org/jira/browse/ATLAS-4846 Repository: atlas Description --- User has to run manually export command to get all the argument which are set, to run the Atlas import script. The problem is not with the Atlas service itself but with these scripts: /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-kafka.sh /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hbase.sh /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh Need to set argument before this script are being called. Diffs (updated) - addons/hbase-bridge/src/bin/import-hbase.sh 8ae735034 addons/hive-bridge/src/bin/import-hive.sh 951be5ba2 addons/impala-bridge/src/main/resources/import-impala.sh b440f2d05 addons/kafka-bridge/src/bin/import-kafka.sh 8fe27748a Diff: https://reviews.apache.org/r/74954/diff/3/ Changes: https://reviews.apache.org/r/74954/diff/2-3/ Testing --- Testing:- Manually create the cluster and ran the import script. Thanks, Paresh Devalia
Re: Review Request 74956: ATLAS-4847 : Export/Import : Atlas export fails and throws NullPointerException.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74956/#review226387 --- Ship it! Ship It! - Sheetal Shah On April 11, 2024, 4:03 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74956/ > --- > > (Updated April 11, 2024, 4:03 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4847 > https://issues.apache.org/jira/browse/ATLAS-4847 > > > Repository: atlas > > > Description > --- > > Steps to reproduce: > > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > > Result is it that it throws NullPointerException and export is failed. > > Problem: > > When an entity which has termed assigned was exported with changeMarker value > greater than value of lastModified time of that entity it resulted in > throwing NullPointerException. > > Solution: > > 1. NullPointerException occured because it was trying to fetch referred > entities of AtlasGlossary Object while performing export operation and as > AtlasGlossary didn't have any referred entities it gave raise this exception. > 2. This problem is solved by handling NullPointer before trying to fetch > referred entities. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 4615c6c2f > > > Diff: https://reviews.apache.org/r/74956/diff/1/ > > > Testing > --- > > Manual testing has been done for the same. > > > Thanks, > > Priyanshi Shah > >
Review Request 74956: ATLAS-4847 : Export/Import : Atlas export fails and throws NullPointerException.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74956/ --- Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and Sheetal Shah. Bugs: ATLAS-4847 https://issues.apache.org/jira/browse/ATLAS-4847 Repository: atlas Description --- Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Result is it that it throws NullPointerException and export is failed. Problem: When an entity which has termed assigned was exported with changeMarker value greater than value of lastModified time of that entity it resulted in throwing NullPointerException. Solution: 1. NullPointerException occured because it was trying to fetch referred entities of AtlasGlossary Object while performing export operation and as AtlasGlossary didn't have any referred entities it gave raise this exception. 2. This problem is solved by handling NullPointer before trying to fetch referred entities. Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 4615c6c2f Diff: https://reviews.apache.org/r/74956/diff/1/ Testing --- Manual testing has been done for the same. Thanks, Priyanshi Shah
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Problem is it throws NullPointerException and export is failed. was: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it throws NullPointerException and export is failed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Atlas web ui real show
Hello， After starting Atlas for the first time, 32 hive table messages were displayed on the web ui, and then I started to operate hive (create table and insert overwrite) Looking at the web ui again, there are still 32 hidden table information displayed; But I don't do anything. After restarting Atlas, it will update the hive table information to 33. How can Atlas display it in real time? Instead of restarting.
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. was: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. !image-2024-04-11-15-15-59-551.png! > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Attachment: image-2024-04-11-15-15-59-551.png > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it fails export and throws NullPointerException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. !image-2024-04-11-15-15-59-551.png! was: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Problem is it fails export and throws NullPointerException. > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: image-2024-04-11-15-15-59-551.png > > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > !image-2024-04-11-15-15-59-551.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Description: Steps to reproduce: 1. Create a hive_table entity and assign a term to it. 2. Perform 1st incremental export of that entity without using changeMarker. 3. Without any modification on that entity perform 2nd consecutive incremental export with changeMarker from previous export. Problem is it fails export and throws NullPointerException. > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > > Steps to reproduce: > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > Problem is it fails export and throws NullPointerException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Description: Repro steps: # First export + import with changeMarker “0” - worked as expected. # I tried to run 2nd export + import without any changes in atlas, 2nd export also worked and gave different changeMarker in output as expected. *But 2nd import failed* In other fetch Types this will work. So we also require similar behaviour in case of Incremental type also because this is very normal scenario and customer will hit this very frequently. > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4845.patch > > > Repro steps: > # First export + import with changeMarker “0” - worked as expected. > # I tried to run 2nd export + import without any changes in atlas, 2nd > export also worked and gave different changeMarker in output as expected. > *But 2nd import failed* > In other fetch Types this will work. So we also require similar behaviour in > case of Incremental type also because this is very normal scenario and > customer will hit this very frequently. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4845) Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
[ https://issues.apache.org/jira/browse/ATLAS-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4845: -- Attachment: ATLAS-4845.patch > Atlas Import is failing with fetchType: Incremental if there are no changes > between two consecutive runs > > > Key: ATLAS-4845 > URL: https://issues.apache.org/jira/browse/ATLAS-4845 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > Attachments: ATLAS-4845.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4847) Export/Import : Atlas export fails and throws NullPointerException.
[ https://issues.apache.org/jira/browse/ATLAS-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanshi Shah updated ATLAS-4847: -- Summary: Export/Import : Atlas export fails and throws NullPointerException. (was: Export/Import : Atlas export fails when glossary is assigned to exported entity with fetchtype : Incremental and changeMarker value greater than lastModified time of that entity.) > Export/Import : Atlas export fails and throws NullPointerException. > --- > > Key: ATLAS-4847 > URL: https://issues.apache.org/jira/browse/ATLAS-4847 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Priyanshi Shah >Assignee: Priyanshi Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4847) Export/Import : Atlas export fails when glossary is assigned to exported entity with fetchtype : Incremental and changeMarker value greater than lastModified time of that
Priyanshi Shah created ATLAS-4847: - Summary: Export/Import : Atlas export fails when glossary is assigned to exported entity with fetchtype : Incremental and changeMarker value greater than lastModified time of that entity. Key: ATLAS-4847 URL: https://issues.apache.org/jira/browse/ATLAS-4847 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Priyanshi Shah Assignee: Priyanshi Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
