Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72631/#review221107 --- Ship it! Ship It! - Nixon Rodrigues On July 1, 2020, 4:29 a.m., Nikhil Bonte wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72631/ > --- > > (Updated July 1, 2020, 4:29 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3871 > https://issues.apache.org/jira/browse/ATLAS-3871 > > > Repository: atlas > > > Description > --- > > AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for > ozone & hdfs path entites. > > This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls > path entities. > > Would add more tests for ozone as well. > > > Diffs > - > > common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java > 664bfb72b > > > Diff: https://reviews.apache.org/r/72631/diff/2/ > > > Testing > --- > > PC build > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console > (Finished: SUCCESS) > > > Thanks, > > Nikhil Bonte > >
Re: Review Request 72627: ATLAS-3867 : Relationship search API should have a provision to fetch custom attributes in search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72627/#review221106 --- Ship it! Ship It! - Sarath Subramanian On June 30, 2020, 10:26 p.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72627/ > --- > > (Updated June 30, 2020, 10:26 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3867 > https://issues.apache.org/jira/browse/ATLAS-3867 > > > Repository: atlas > > > Description > --- > > **Issue:** Attributes in search result of Relationship api are minimal. > > **WorkAround:** Relationship api request should have provision to specify > attributes to be present in search result. > > **Example Request:** > /v2/search/relationship?guid=ac9e04cc-f927-4334-af08-c83bc3733f5b&relation=columns&sortBy=name&sortOrder=ASCENDING&attributes=dcProfiledData > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java > e64c31522 > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > 4169e14ef > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 076284ec4 > > > Diff: https://reviews.apache.org/r/72627/diff/2/ > > > Testing > --- > > Manually tested > Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1994 > > > Thanks, > > Pinal Shah > >
Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72631/#review221105 --- Ship it! Ship It! - Sarath Subramanian On June 30, 2020, 9:29 p.m., Nikhil Bonte wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72631/ > --- > > (Updated June 30, 2020, 9:29 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3871 > https://issues.apache.org/jira/browse/ATLAS-3871 > > > Repository: atlas > > > Description > --- > > AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for > ozone & hdfs path entites. > > This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls > path entities. > > Would add more tests for ozone as well. > > > Diffs > - > > common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java > 664bfb72b > > > Diff: https://reviews.apache.org/r/72631/diff/2/ > > > Testing > --- > > PC build > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console > (Finished: SUCCESS) > > > Thanks, > > Nikhil Bonte > >
Re: Review Request 72627: ATLAS-3867 : Relationship search API should have a provision to fetch custom attributes in search results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72627/ --- (Updated July 1, 2020, 5:26 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- rebase patch Bugs: ATLAS-3867 https://issues.apache.org/jira/browse/ATLAS-3867 Repository: atlas Description --- **Issue:** Attributes in search result of Relationship api are minimal. **WorkAround:** Relationship api request should have provision to specify attributes to be present in search result. **Example Request:** /v2/search/relationship?guid=ac9e04cc-f927-4334-af08-c83bc3733f5b&relation=columns&sortBy=name&sortOrder=ASCENDING&attributes=dcProfiledData Diffs (updated) - repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java e64c31522 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 4169e14ef webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 076284ec4 Diff: https://reviews.apache.org/r/72627/diff/2/ Changes: https://reviews.apache.org/r/72627/diff/1-2/ Testing --- Manually tested Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1994 Thanks, Pinal Shah
Re: Review Request 72625: ATLAS-3866 : Relationship search API for hive storage desc throws error code 500
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72625/#review221104 --- Ship it! Ship It! - Sarath Subramanian On June 29, 2020, 3:49 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72625/ > --- > > (Updated June 29, 2020, 3:49 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3866 > https://issues.apache.org/jira/browse/ATLAS-3866 > > > Repository: atlas > > > Description > --- > > **Issue:** In relationship api v2/relationship, for hive_storagedesc it > throws exception > > **Reason:** it is because, sort attribute is assigned 'name' as default, and > 'name' attribute is not in the defination of hive_storagedesc. > > **Workaround:** Validate if attribute is present in relationship end def, if > not ignore sorting > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > dd4d1b441 > > > Diff: https://reviews.apache.org/r/72625/diff/2/ > > > Testing > --- > > Manually tested > Precommit : > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1992/console > > > Thanks, > > Pinal Shah > >
[jira] [Commented] (ATLAS-3869) Upgrade httpclient from 4.5.3 to 4.5.4
[
https://issues.apache.org/jira/browse/ATLAS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149096#comment-17149096
]
Nikhil Bonte commented on ATLAS-3869:
-
+1 for the patch.
Thanks [~nixon] .
> Upgrade httpclient from 4.5.3 to 4.5.4
> --
>
> Key: ATLAS-3869
> URL: https://issues.apache.org/jira/browse/ATLAS-3869
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: ATLAS-3869-Upgrade-httpclient-from-4.5.3-to-4.5.4.patch
>
>
> Found below exception where data was not getting added in SOLR from
> janusgraph.
>
> {noformat}
> Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String
> at
> org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)
> at
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112)
> at
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99)
> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:463)
> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
> at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
> at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
> ... 7 more
> 2020-06-29 07:31:25,893 WARN - [SIGTERM handler:] ~ Unable to close
> transaction standardjanusgraphtx[0x18d7a16a] (StandardJanusGraph:230)
> java.lang.IllegalArgumentException: The transaction has already been closed
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
> at
> org.janusgraph.graphdb.transaction.StandardJanusGraphTx.rollback(StandardJanusGraphTx.java:1401)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doRollback(JanusGraphBlueprintsTransaction.java:190)
> at
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.rollback(AbstractTransaction.java:114)
> at
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:182)
> at
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:179)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.close(JanusGraphBlueprintsGraph.java:327)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doClose(JanusGraphBlueprintsTransaction.java:205)
> at
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.close(AbstractTransaction.java:139)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction.close(JanusGraphBlueprintsTransaction.java:215)
> at
> org.janusgraph.graphdb.database.StandardJanusGraph.closeInternal(StandardJanusGraph.java:226)
> at
> org.janusgraph.graphdb.database.StandardJanusGraph.access$500(StandardJanusGraph.java:91)
> at
> org.janusgraph.graphdb.database.StandardJanusGraph$ShutdownThread.start(StandardJanusGraph.java:818)
> at
> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102)
> at
> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
> at java.lang.Shutdown.runHooks(Shutd
Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72631/ --- (Updated July 1, 2020, 4:29 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. Changes --- Updated pre-commit build status Bugs: ATLAS-3871 https://issues.apache.org/jira/browse/ATLAS-3871 Repository: atlas Description --- AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for ozone & hdfs path entites. This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Would add more tests for ozone as well. Diffs - common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 664bfb72b Diff: https://reviews.apache.org/r/72631/diff/2/ Testing (updated) --- PC build https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console (Finished: SUCCESS) Thanks, Nikhil Bonte
[jira] [Assigned] (ATLAS-3872) While creating Column entity shows error message
[ https://issues.apache.org/jira/browse/ATLAS-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-3872: -- Assignee: Mandar Ambawane > While creating Column entity shows error message > > > Key: ATLAS-3872 > URL: https://issues.apache.org/jira/browse/ATLAS-3872 > Project: Atlas > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Durga Kadam >Assignee: Mandar Ambawane >Priority: Minor > Attachments: column_entity_name_field_issue.ogv > > > Description:: While creating Column entity(whose entity-type has been > generated from running quick_start.py) if user does not provide the 'name'(by > enabling the toggle 'Required All') it shows below error message. > Error message:: > Invalid instance creation/updation parameters passed : Column.name: mandatory > attribute value missing in type Asset > Steps to reproduce issue:: > # Go to Create entity > # Select 'Column' type > # Enter 'QualifiedName(string)' and select the 'Table(Table)' from drop-down > # Click on Create buttonIt will show error message > Note- > # If user provides the 'name' which is listed inside the 'Required All' tab > it will allow user to save the entity. PFA attached video. > # Working fine for Hive_column and Hbase_column as user provides the > mandatory name field which is listed outside the 'Required All' tab -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3801) NotificationHookConsumer: Introduce Retry Logic for PermanentLockingException
[ https://issues.apache.org/jira/browse/ATLAS-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-3801: --- Attachment: ATLAS-3801-Notification-hook-processing-concurrent-e.patch > NotificationHookConsumer: Introduce Retry Logic for PermanentLockingException > - > > Key: ATLAS-3801 > URL: https://issues.apache.org/jira/browse/ATLAS-3801 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-3801-Notification-hook-processing-concurrent-e.patch > > > *Background* > _NotificationHookConsumer_ when performing graph commit can potentially > encounter _JanusGraph's PermanentLockingException_. > JanusGraph [documentation|https://docs.janusgraph.org/basics/transactions/] > details the cases in which this exception occurs. The way past this exception > is rollback, pause and replay the data being committed. > *Solution* > Introduce the rollback, pause and retry mechanism in > _NotificationHookConsumer_. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3870) UI: hive_table entity table not updating queryparams based on filter selected
[
https://issues.apache.org/jira/browse/ATLAS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-3870:
--
Component/s: atlas-webui
> UI: hive_table entity table not updating queryparams based on filter selected
> -
>
> Key: ATLAS-3870
> URL: https://issues.apache.org/jira/browse/ATLAS-3870
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3870.patch, Screenshot 2020-06-30 at 2.33.11
> PM.png, Screenshot 2020-06-30 at 2.34.06 PM.png
>
>
> * Go to hive_db entity detail page
> * Click on Table tab
> * toggle "Show historical entities" checkbox and check the network for the
> API
> API always send the same value for
> {code:java}
> excludeDeletedEntities: true
> {code}
> !Screenshot 2020-06-30 at 2.33.11 PM.png|width=374,height=219!
> !Screenshot 2020-06-30 at 2.34.06 PM.png|width=397,height=142!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3870) UI: hive_table entity table not updating queryparams based on filter selected
[
https://issues.apache.org/jira/browse/ATLAS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-3870:
--
Labels: ui (was: )
> UI: hive_table entity table not updating queryparams based on filter selected
> -
>
> Key: ATLAS-3870
> URL: https://issues.apache.org/jira/browse/ATLAS-3870
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Labels: ui
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3870.patch, Screenshot 2020-06-30 at 2.33.11
> PM.png, Screenshot 2020-06-30 at 2.34.06 PM.png
>
>
> * Go to hive_db entity detail page
> * Click on Table tab
> * toggle "Show historical entities" checkbox and check the network for the
> API
> API always send the same value for
> {code:java}
> excludeDeletedEntities: true
> {code}
> !Screenshot 2020-06-30 at 2.33.11 PM.png|width=374,height=219!
> !Screenshot 2020-06-30 at 2.34.06 PM.png|width=397,height=142!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3870) UI: hive_table entity table not updating queryparams based on filter selected
[
https://issues.apache.org/jira/browse/ATLAS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keval Bhatt updated ATLAS-3870:
---
Attachment: ATLAS-3870.patch
> UI: hive_table entity table not updating queryparams based on filter selected
> -
>
> Key: ATLAS-3870
> URL: https://issues.apache.org/jira/browse/ATLAS-3870
> Project: Atlas
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3870.patch, Screenshot 2020-06-30 at 2.33.11
> PM.png, Screenshot 2020-06-30 at 2.34.06 PM.png
>
>
> * Go to hive_db entity detail page
> * Click on Table tab
> * toggle "Show historical entities" checkbox and check the network for the
> API
> API always send the same value for
> {code:java}
> excludeDeletedEntities: true
> {code}
> !Screenshot 2020-06-30 at 2.33.11 PM.png|width=374,height=219!
> !Screenshot 2020-06-30 at 2.34.06 PM.png|width=397,height=142!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3872) While creating Column entity shows error message
Durga Kadam created ATLAS-3872: -- Summary: While creating Column entity shows error message Key: ATLAS-3872 URL: https://issues.apache.org/jira/browse/ATLAS-3872 Project: Atlas Issue Type: Bug Affects Versions: 3.0.0 Reporter: Durga Kadam Attachments: column_entity_name_field_issue.ogv Description:: While creating Column entity(whose entity-type has been generated from running quick_start.py) if user does not provide the 'name'(by enabling the toggle 'Required All') it shows below error message. Error message:: Invalid instance creation/updation parameters passed : Column.name: mandatory attribute value missing in type Asset Steps to reproduce issue:: # Go to Create entity # Select 'Column' type # Enter 'QualifiedName(string)' and select the 'Table(Table)' from drop-down # Click on Create buttonIt will show error message Note- # If user provides the 'name' which is listed inside the 'Required All' tab it will allow user to save the entity. PFA attached video. # Working fine for Hive_column and Hbase_column as user provides the mandatory name field which is listed outside the 'Required All' tab -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72615: ATLAS-3855 Bulk entity tag association and bulk api enhancement
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72615/ --- (Updated June 30, 2020, 2:56 p.m.) Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-3855 https://issues.apache.org/jira/browse/ATLAS-3855 Repository: atlas Description --- API : /api/atlas/v2/entity/bulk/classification & v2/entity/bulk : if some guids belong to entities on which user is unauthorized other guids belong to entities on which user is authorized This APIs fail with 403 error without returning the authorized entities. 1.Unauthorized guids are filtered with this patch for both the APIs. 2.Added ignoreUnauthorisedGuids flag for /bulk/classification API as it doesn't return any object 2.Also added unauthorized guids in return object of /bulk in response for user's reference. Diffs - intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 4d8c94894 repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 7b9455ef3 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java bf1629cb3 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java b9cbef1b0 webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 00b29e6c8 webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 88de8b679 webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 615bc0f1b Diff: https://reviews.apache.org/r/72615/diff/4/ Testing --- Tested with ranger policies: For /bulk api following policy were applied: 1.Added hive_column entity as entity type. 2.Included admin as user and prohibited access for read entity all this under deny policy. For /bulk/classification following policy were applied: 1.Added hive_column entity as entity type. 2.Gave all the access to admin but kwpt hive_column entity type in excluded state. Thanks, chaitali
Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72631/ --- (Updated June 30, 2020, 1:34 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. Changes --- Added Pre commit build details Bugs: ATLAS-3871 https://issues.apache.org/jira/browse/ATLAS-3871 Repository: atlas Description --- AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for ozone & hdfs path entites. This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Would add more tests for ozone as well. Diffs - common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 664bfb72b Diff: https://reviews.apache.org/r/72631/diff/2/ Testing (updated) --- PC build https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console (In Progress) Thanks, Nikhil Bonte
Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72631/ --- (Updated June 30, 2020, 1:30 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. Changes --- Refactorings Bugs: ATLAS-3871 https://issues.apache.org/jira/browse/ATLAS-3871 Repository: atlas Description --- AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for ozone & hdfs path entites. This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Would add more tests for ozone as well. Diffs (updated) - common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 664bfb72b Diff: https://reviews.apache.org/r/72631/diff/2/ Changes: https://reviews.apache.org/r/72631/diff/1-2/ Testing --- Thanks, Nikhil Bonte
Re: Review Request 72567: ATLAS-3782 : Support NOT_CONTAINS operator in basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72567/ --- (Updated June 30, 2020, 1:09 p.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-3782 https://issues.apache.org/jira/browse/ATLAS-3782 Repository: atlas Description --- **Issue:** The operator 'SearchParameters.Operator.NOT_CONTAINS' is defined and implemented in SearchProcessors. It would allow a search of entities that do not contain in given string in a specified attribute (eg exclude entities from search that contain 'temp' in the qualified name). **WorkAround:** JanusGraph doesn't allow NOT_CONTAINS operator. So we will handle this in inmemory BasicSearch generates query via three modes 1. Index query -> NOT_CONTAINS will not be supported 2. InMemeory Predicates -> NOT_CONTAINS will be supported, Already handled #123 SearchProcessor 3. Graph query -> NOT_CONTAINS will not be supported As in index and graph query wiil not support not_Contains operator, We need to apply filter(inMemoryPredicate) after either index/graph query. To support above, I have modified ClassificationSearchProcessor + For both cases index as well as graph, added typeNamePredicate and attributePredicate + Added these predicate after query - Removed gremlinQuery block **Operator Value:** It can be either "not_contains" or "NOT_CONTAINS" **Note:** As part of this jira, not_contains is also added in quick search. Diffs - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 6c06a3cbe repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 9c72cd4a2 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java c9a605355 repository/src/test/java/org/apache/atlas/BasicTestSetup.java 8b98b3990 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java 9b16e919d repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java b7ce97845 Diff: https://reviews.apache.org/r/72567/diff/4/ Testing (updated) --- Added testcases Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1999 Thanks, Pinal Shah
Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72631/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-3871 https://issues.apache.org/jira/browse/ATLAS-3871 Repository: atlas Description --- AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for ozone & hdfs path entites. This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Would add more tests for ozone as well. Diffs - common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 664bfb72b Diff: https://reviews.apache.org/r/72631/diff/1/ Testing --- Thanks, Nikhil Bonte
[jira] [Updated] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities
[ https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil Bonte updated ATLAS-3871: Description: AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for ozone & hdfs path entites. This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Would add more tests for ozone as well. was: AtlasPathExtractorUtilTest added in ATLAS-3836 which covers ozone & hdfs path entites unit tests. This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Also will add more tests for ozone path. > Add unit tests to path extractor utility for s3, s3 v2, adls path entities > -- > > Key: ATLAS-3871 > URL: https://issues.apache.org/jira/browse/ATLAS-3871 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > > AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for > ozone & hdfs path entites. > > This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & > adls path entities. > Would add more tests for ozone as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities
[ https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil Bonte updated ATLAS-3871: Description: AtlasPathExtractorUtilTest added in ATLAS-3836 which covers ozone & hdfs path entites unit tests. This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Also will add more tests for ozone path. was: AtlasPathExtractorUtilTest is added in ATLAS-3836 which covers ozone & hdfs path entites unit tests. This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Also will add more tests for ozone path. > Add unit tests to path extractor utility for s3, s3 v2, adls path entities > -- > > Key: ATLAS-3871 > URL: https://issues.apache.org/jira/browse/ATLAS-3871 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > > AtlasPathExtractorUtilTest added in ATLAS-3836 which covers ozone & hdfs path > entites unit tests. > This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & > adls path entities. > Also will add more tests for ozone path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities
[ https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil Bonte updated ATLAS-3871: Summary: Add unit tests to path extractor utility for s3, s3 v2, adls path entities (was: Add unit tests for s3, s3 v2, adls path entities in path extractor utility) > Add unit tests to path extractor utility for s3, s3 v2, adls path entities > -- > > Key: ATLAS-3871 > URL: https://issues.apache.org/jira/browse/ATLAS-3871 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > > AtlasPathExtractorUtilTest is added in ATLAS-3836 which covers ozone & hdfs > path entites unit tests. > This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & > adls path entities. > Also will add more tests for ozone path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3871) Add unit tests for s3, s3 v2, adls path entities in path extractor utility
Nikhil Bonte created ATLAS-3871: --- Summary: Add unit tests for s3, s3 v2, adls path entities in path extractor utility Key: ATLAS-3871 URL: https://issues.apache.org/jira/browse/ATLAS-3871 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.0.0 Reporter: Nikhil Bonte Assignee: Nikhil Bonte AtlasPathExtractorUtilTest is added in ATLAS-3836 which covers ozone & hdfs path entites unit tests. This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path entities. Also will add more tests for ozone path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3869) Upgrade httpclient from 4.5.3 to 4.5.4
[
https://issues.apache.org/jira/browse/ATLAS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nixon Rodrigues updated ATLAS-3869:
---
Attachment: ATLAS-3869-Upgrade-httpclient-from-4.5.3-to-4.5.4.patch
> Upgrade httpclient from 4.5.3 to 4.5.4
> --
>
> Key: ATLAS-3869
> URL: https://issues.apache.org/jira/browse/ATLAS-3869
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: ATLAS-3869-Upgrade-httpclient-from-4.5.3-to-4.5.4.patch
>
>
> Found below exception where data was not getting added in SOLR from
> janusgraph.
>
> {noformat}
> Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String
> at
> org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)
> at
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112)
> at
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99)
> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:463)
> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
> at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
> at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
> ... 7 more
> 2020-06-29 07:31:25,893 WARN - [SIGTERM handler:] ~ Unable to close
> transaction standardjanusgraphtx[0x18d7a16a] (StandardJanusGraph:230)
> java.lang.IllegalArgumentException: The transaction has already been closed
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
> at
> org.janusgraph.graphdb.transaction.StandardJanusGraphTx.rollback(StandardJanusGraphTx.java:1401)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doRollback(JanusGraphBlueprintsTransaction.java:190)
> at
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.rollback(AbstractTransaction.java:114)
> at
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:182)
> at
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:179)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.close(JanusGraphBlueprintsGraph.java:327)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doClose(JanusGraphBlueprintsTransaction.java:205)
> at
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.close(AbstractTransaction.java:139)
> at
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction.close(JanusGraphBlueprintsTransaction.java:215)
> at
> org.janusgraph.graphdb.database.StandardJanusGraph.closeInternal(StandardJanusGraph.java:226)
> at
> org.janusgraph.graphdb.database.StandardJanusGraph.access$500(StandardJanusGraph.java:91)
> at
> org.janusgraph.graphdb.database.StandardJanusGraph$ShutdownThread.start(StandardJanusGraph.java:818)
> at
> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102)
> at
> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
> at java.lang.Shutdown.runHooks(Shutdown.java:123)
>
[jira] [Commented] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[
https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148549#comment-17148549
]
yangchengkai commented on ATLAS-3488:
-
[~nixon]
Thanks for apply.
Our company use CDH 6.2.1 to management hadoop cluster, so the hive version of
CDH 6.2.1 is hive 2.1.1.
Anyway I did not change the hive version to 2.1.1 in my pom cause many Chinese
blogs tell me not to do so.
I decide to use Atlas 2.0.0 version, because many blogs use 2.0.0 to solve the
CDH compatible problems.
Again thanks for you help.
> Update with Filebased auth password with ShaPasswordEncoder with salt.
> --
>
> Key: ATLAS-3488
> URL: https://issues.apache.org/jira/browse/ATLAS-3488
> Project: Atlas
> Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: ATLAS-3488.07.patch, ATLAS-3488.6.patch
>
>
> Current password is encoded with {{sha256 method for filebased
> authentication.}}
>
> {{Update the password with }}{{sha256}}{{ with salt encoding which is more
> secured.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[
https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148508#comment-17148508
]
Nixon Rodrigues commented on ATLAS-3488:
[~18810276...@163.com]
"org.apache.hadoop.hive.metastore.api.Database.getCatalogName()"ERROR?
This is issue is due to mismatch in hive version, Atlas expects hive with
version 3.1.0.
from the logs from provided.
{noformat}
17:47:41.269 [main] DEBUG org.apache.atlas.AtlasBaseClient - Call : GET
api/atlas/v2/entity/uniqueAttribute/type/
17:47:41.269 [main] DEBUG org.apache.atlas.AtlasBaseClient - Content-type :
application/json; charset=UTF-8
17:47:41.269 [main] DEBUG org.apache.atlas.AtlasBaseClient - Accept :
application/json
17:47:41.283 [main] DEBUG org.apache.atlas.AtlasBaseClient - HTTP Status : 404
{noformat}
its giving 404 since database do not exist in Atlas and authentication has
passed. if there was authentication issue then it will give 401.
> Update with Filebased auth password with ShaPasswordEncoder with salt.
> --
>
> Key: ATLAS-3488
> URL: https://issues.apache.org/jira/browse/ATLAS-3488
> Project: Atlas
> Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: ATLAS-3488.07.patch, ATLAS-3488.6.patch
>
>
> Current password is encoded with {{sha256 method for filebased
> authentication.}}
>
> {{Update the password with }}{{sha256}}{{ with salt encoding which is more
> secured.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 72567: ATLAS-3782 : Support NOT_CONTAINS operator in basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72567/ --- (Updated June 30, 2020, 10:04 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-3782 https://issues.apache.org/jira/browse/ATLAS-3782 Repository: atlas Description (updated) --- **Issue:** The operator 'SearchParameters.Operator.NOT_CONTAINS' is defined and implemented in SearchProcessors. It would allow a search of entities that do not contain in given string in a specified attribute (eg exclude entities from search that contain 'temp' in the qualified name). **WorkAround:** JanusGraph doesn't allow NOT_CONTAINS operator. So we will handle this in inmemory BasicSearch generates query via three modes 1. Index query -> NOT_CONTAINS will not be supported 2. InMemeory Predicates -> NOT_CONTAINS will be supported, Already handled #123 SearchProcessor 3. Graph query -> NOT_CONTAINS will not be supported As in index and graph query wiil not support not_Contains operator, We need to apply filter(inMemoryPredicate) after either index/graph query. To support above, I have modified ClassificationSearchProcessor + For both cases index as well as graph, added typeNamePredicate and attributePredicate + Added these predicate after query - Removed gremlinQuery block **Operator Value:** It can be either "not_contains" or "NOT_CONTAINS" **Note:** As part of this jira, not_contains is also added in quick search. Diffs - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 6c06a3cbe repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 9c72cd4a2 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java c9a605355 repository/src/test/java/org/apache/atlas/BasicTestSetup.java 8b98b3990 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java 9b16e919d repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java b7ce97845 Diff: https://reviews.apache.org/r/72567/diff/4/ Testing --- Added testcases Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1952 Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1983 Thanks, Pinal Shah
[jira] [Commented] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[
https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148491#comment-17148491
]
yangchengkai commented on ATLAS-3488:
-
[~nixon]
PS:
Is this the "Invalid salt version" Error caused my
"org.apache.hadoop.hive.metastore.api.Database.getCatalogName()"ERROR?
I google and baidu around,and did not find the problems like this.
> Update with Filebased auth password with ShaPasswordEncoder with salt.
> --
>
> Key: ATLAS-3488
> URL: https://issues.apache.org/jira/browse/ATLAS-3488
> Project: Atlas
> Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: ATLAS-3488.07.patch, ATLAS-3488.6.patch
>
>
> Current password is encoded with {{sha256 method for filebased
> authentication.}}
>
> {{Update the password with }}{{sha256}}{{ with salt encoding which is more
> secured.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[ https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148487#comment-17148487 ] yangchengkai commented on ATLAS-3488: - [~nixon],hi: I can run quick-start.py successfully, this is my output: root@master:/usr/local/src/atlas/apache-atlas-2.1.0# bin/quick_start.py Enter username for atlas :- admin Enter password for atlas :- Creating sample types: Created type [DB] Created type [Table] Created type [StorageDesc] Created type [Column] Created type [LoadProcess] Created type [LoadProcessExecution] Created type [View] Created type [JdbcAccess] Created type [ETL] Created type [Metric] Created type [PII] Created type [Fact] Created type [Dimension] Created type [Log Data] Created type [Table_DB] Created type [View_DB] Created type [View_Tables] Created type [Table_Columns] Created type [Table_StorageDesc] Creating sample entities: Created entity of type [DB], guid: 0809e0f7-b772-4521-bc5e-46192e3c0ef1 Created entity of type [DB], guid: 7a4ba630-2c18-49d9-ba65-d127a2b12a28 Created entity of type [DB], guid: 5effe5a7-7b4c-439a-9b3a-c523bebe2dfc Created entity of type [Table], guid: 33578c65-ad00-4081-9bab-4fad7dfdc379 but when I run the import_hive.sh,it still output the same error on my console: 17:47:40.926 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:40.926 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:40.926 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:40.926 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:40.927 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:40.927 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:40.927 [main] DEBUG org.apache.hadoop.hive.ql.exec.Utilities - Hive Conf not found or Session not initiated, use thread based class loader instead 17:47:41.047 [main] DEBUG org.apache.hadoop.security.SecurityUtil - Setting hadoop.security.token.service.use_ip to true 17:47:41.058 [main] DEBUG org.apache.hadoop.security.Groups - Creating new Groups object 17:47:41.063 [main] DEBUG org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30; warningDeltaMs=5000 17:47:41.066 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login 17:47:41.066 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login commit 17:47:41.067 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - using local user:UnixPrincipal: root 17:47:41.067 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - Using user: "UnixPrincipal: root" with name root 17:47:41.067 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - User entry: "root" 17:47:41.068 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - UGI loginUser:root (auth:SIMPLE) 17:47:41.081 [main] INFO hive.metastore - HMS client filtering is enabled. 17:47:41.081 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://master:9083 17:47:41.094 [main] INFO hive.metastore - Opened a connection to metastore, current connections: 1 17:47:41.097 [main] DEBUG org.apache.hadoop.security.Groups - GroupCacheLoader - load. 17:47:41.114 [main] INFO hive.metastore - Connected to metastore. 17:47:41.255 [main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata 17:47:41.260 [main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 2 databases 17:47:41.264 [main] DEBUG org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Searching Atlas for database default 17:47:41.268 [main] DEBUG org.apache.atlas.AtlasBaseClient - -- 17:47:41.269 [main] DEBUG org.apache.atlas.AtlasBaseClient - Call : GET api/atlas/v2/entity/uniqueAttribute/type/ 17:47:41.269 [main] DEBUG org.apache.atlas.AtlasBaseClient - Content-type : application/json; charset=UTF-8 17:47:41.269 [main] DEBUG org.apache.atlas.AtlasBaseClient - Accept : application/json 17:47:41.283 [main] DEBUG org.apache.atlas.AtlasBaseClient - HTTP Status : 404 Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.Database.getCatalogName()Ljava/lang/String; at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.getDatabaseName(HiveMetaStoreBridge.java:577) at org.apache.
Re: Review Request 72567: ATLAS-3782 : Support NOT_CONTAINS operator in basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72567/ --- (Updated June 30, 2020, 9:54 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- Rebase patch and added not_contains in quicksearch aggregation metrics Bugs: ATLAS-3782 https://issues.apache.org/jira/browse/ATLAS-3782 Repository: atlas Description --- The operator 'SearchParameters.Operator.NOT_CONTAINS' is defined and implemented in SearchProcessors. It would allow a search of entities that do not contain in given string in a specified attribute (eg exclude entities from search that contain 'temp' in the qualified name). JanusGraph doesn't allow NOT_CONTAINS operator. So we will handle this in inmemory BasicSearch can generates query via three modes 1. Index query -> NOT_CONTAINS will not be supported 2. InMemeory Predicated -> NOT_CONTAINS will be supported, Already handled #123 SearchProcessor 3. Graph query -> NOT_CONTAINS will not be supported We need to apply filter(inMemoryPredicate) after either index/graph query. To support above, i have modified ClassificationSearchProcessor + For both cases index as well as graph, added typeNamePredicate and attributePredicate + Added these predicate after query - Removed gremlinQuery block Diffs (updated) - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java 6c06a3cbe repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 9c72cd4a2 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java c9a605355 repository/src/test/java/org/apache/atlas/BasicTestSetup.java 8b98b3990 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java 9b16e919d repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java b7ce97845 Diff: https://reviews.apache.org/r/72567/diff/4/ Changes: https://reviews.apache.org/r/72567/diff/3-4/ Testing --- Added testcases Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1952 Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1983 Thanks, Pinal Shah
[jira] [Commented] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[
https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148473#comment-17148473
]
Nixon Rodrigues commented on ATLAS-3488:
[~18810276...@163.com],
I verified quick-start.py which is using same authentication route and its
working correctly with admin/admin.
can you verify if you are using same users-credentials.properties file provided
in 2.1.0-rc1 bits.
admin=ADMIN::a4a88c0872bf652bb9ed803ece5fd6e82354838a9bf59ab4babb1dab322154e1
> Update with Filebased auth password with ShaPasswordEncoder with salt.
> --
>
> Key: ATLAS-3488
> URL: https://issues.apache.org/jira/browse/ATLAS-3488
> Project: Atlas
> Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: ATLAS-3488.07.patch, ATLAS-3488.6.patch
>
>
> Current password is encoded with {{sha256 method for filebased
> authentication.}}
>
> {{Update the password with }}{{sha256}}{{ with salt encoding which is more
> secured.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3836:
Description:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
was:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png,
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png,
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop -
> [https://hadoop.apache.or
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3836:
Description:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt' into
table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
was:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png,
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png,
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop -
> [https://hadoop.apache.o
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3836:
Description:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
was:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png,
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png,
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop -
> [https://hadoop.apache.org
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3836:
Description:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
was:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png,
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png,
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop -
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities.
> Hive hook should also be updated to create lineage between ozone entities and
> hive tables (for EXTERNAL TABLE)
> +*Approach :*+
> # Refactored BaseHiveEvent.getPathEntity() -> moved to
> AtlasPathExtractorUtil.java
> # Created PathExtractorContext.java to wrap most arguments.
> # AtlasPathExtractorUtil.getPathEntity() -> accept Path,
> PathExtractorContext -> return AtlasEntityWithExtInfo
> # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to
> handle Ozone path
> -> path starts with "ofs://" or "o3fs://"
> # Added UT around AtlasPathExtractorUtil.getPathEntity() ->
> AtlasPathExtractorUtilTest.java
>
> +*Examples :*+
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields
> terminated by ' ' stored as textfile location
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
>
>
> || ||Name||Qualified Name||
> |ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
> |ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
> |ozone_volume|volume1|o3fs://volume1@cm|
>
>
> {code:java}
> -> create EXTERNAL table stocks (id int) row form
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3836:
Description:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
was:
Apache Ozone is the new object store for Hadoop -
[https://hadoop.apache.org/ozone/]
Apache Atlas needs to add entity types to support creation of Ozone entities.
Hive hook should also be updated to create lineage between ozone entities and
hive tables (for EXTERNAL TABLE)
+*Approach :*+
# Refactored BaseHiveEvent.getPathEntity() -> moved to
AtlasPathExtractorUtil.java
# Created PathExtractorContext.java to wrap most arguments.
# AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext
-> return AtlasEntityWithExtInfo
# Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle
Ozone path
-> path starts with "ofs://" or "o3fs://"
# Added UT around AtlasPathExtractorUtil.getPathEntity() ->
AtlasPathExtractorUtilTest.java
+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated
by ' ' stored as textfile location
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt'
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|
> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png,
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png,
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop -
> [https://hadoop.a
[jira] [Created] (ATLAS-3870) UI: hive_table entity table not updating queryparams based on filter selected
Keval Bhatt created ATLAS-3870:
--
Summary: UI: hive_table entity table not updating queryparams
based on filter selected
Key: ATLAS-3870
URL: https://issues.apache.org/jira/browse/ATLAS-3870
Project: Atlas
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Keval Bhatt
Assignee: Keval Bhatt
Fix For: 2.1.0, 3.0.0
Attachments: Screenshot 2020-06-30 at 2.33.11 PM.png, Screenshot
2020-06-30 at 2.34.06 PM.png
* Go to hive_db entity detail page
* Click on Table tab
* toggle "Show historical entities" checkbox and check the network for the API
API always send the same value for
{code:java}
excludeDeletedEntities: true
{code}
!Screenshot 2020-06-30 at 2.33.11 PM.png|width=374,height=219!
!Screenshot 2020-06-30 at 2.34.06 PM.png|width=397,height=142!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[ https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil Bonte updated ATLAS-3836: Description: Apache Ozone is the new object store for Hadoop - [https://hadoop.apache.org/ozone/] Apache Atlas needs to add entity types to support creation of Ozone entities. Hive hook should also be updated to create lineage between ozone entities and hive tables (for EXTERNAL TABLE) +*Approach :*+ # Refactored BaseHiveEvent.getPathEntity() -> moved to AtlasPathExtractorUtil.java # Created PathExtractorContext.java to wrap most arguments. # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext -> return AtlasEntityWithExtInfo # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle Ozone path -> path starts with "ofs://" or "o3fs://" # Added UT around AtlasPathExtractorUtil.getPathEntity() -> AtlasPathExtractorUtilTest.java was: Apache Ozone is the new object store for Hadoop - [https://hadoop.apache.org/ozone/] Apache Atlas needs to add entity types to support creation of Ozone entities. Hive hook should also be updated to create lineage between ozone entities and hive tables (for EXTERNAL TABLE) > Add Apache Ozone support in hive hook > - > > Key: ATLAS-3836 > URL: https://issues.apache.org/jira/browse/ATLAS-3836 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sarath Subramanian >Assignee: Nikhil Bonte >Priority: Major > Labels: hive-hooks, ozone > Fix For: 2.1.0 > > Attachments: Hive_table_lineage.png, > Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, > Ozone_volume.png > > > Apache Ozone is the new object store for Hadoop - > [https://hadoop.apache.org/ozone/] > Apache Atlas needs to add entity types to support creation of Ozone entities. > Hive hook should also be updated to create lineage between ozone entities and > hive tables (for EXTERNAL TABLE) > +*Approach :*+ > # Refactored BaseHiveEvent.getPathEntity() -> moved to > AtlasPathExtractorUtil.java > # Created PathExtractorContext.java to wrap most arguments. > # AtlasPathExtractorUtil.getPathEntity() -> accept Path, > PathExtractorContext -> return AtlasEntityWithExtInfo > # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to > handle Ozone path > -> path starts with "ofs://" or "o3fs://" > # Added UT around AtlasPathExtractorUtil.getPathEntity() -> > AtlasPathExtractorUtilTest.java > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[ https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148414#comment-17148414 ] yangchengkai edited comment on ATLAS-3488 at 6/30/20, 8:15 AM: --- Hi,I got this error in logs/application.log when I run ./bin/import_hive.sh: 2020-06-30 15:50:24,815 DEBUG - [pool-2-thread-2:] ~ checkPasswordBCrypt() (UserDao:159) 2020-06-30 15:50:24,816 DEBUG - [pool-2-thread-2:] ~ checkPasswordBCrypt(): failed (UserDao:168) java.lang.IllegalArgumentException: Invalid salt version at org.springframework.security.crypto.bcrypt.BCrypt.hashpw(BCrypt.java:556) at org.springframework.security.crypto.bcrypt.BCrypt.checkpw(BCrypt.java:659) at org.apache.atlas.web.dao.UserDao.checkPasswordBCrypt(UserDao.java:165) at org.apache.atlas.web.dao.UserDao.checkEncrypted(UserDao.java:144) at org.apache.atlas.web.security.AtlasFileAuthenticationProvider.authenticate(AtlasFileAuthenticationProvider.java:65) at org.apache.atlas.web.security.AtlasAuthenticationProvider.authenticate(AtlasAuthenticationProvider.java:143) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:142) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillIn
[jira] [Comment Edited] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[ https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148414#comment-17148414 ] yangchengkai edited comment on ATLAS-3488 at 6/30/20, 8:15 AM: --- Hi,I got this error in logs/application.log when I run ./bin/import_hive.sh: 2020-06-30 15:50:24,815 DEBUG - [pool-2-thread-2:] ~ checkPasswordBCrypt() (UserDao:159) 2020-06-30 15:50:24,816 DEBUG - [pool-2-thread-2:] ~ checkPasswordBCrypt(): failed (UserDao:168) java.lang.IllegalArgumentException: Invalid salt version at org.springframework.security.crypto.bcrypt.BCrypt.hashpw(BCrypt.java:556) at org.springframework.security.crypto.bcrypt.BCrypt.checkpw(BCrypt.java:659) at org.apache.atlas.web.dao.UserDao.checkPasswordBCrypt(UserDao.java:165) at org.apache.atlas.web.dao.UserDao.checkEncrypted(UserDao.java:144) at org.apache.atlas.web.security.AtlasFileAuthenticationProvider.authenticate(AtlasFileAuthenticationProvider.java:65) at org.apache.atlas.web.security.AtlasAuthenticationProvider.authenticate(AtlasAuthenticationProvider.java:143) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:142) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillIn
[jira] [Commented] (ATLAS-3488) Update with Filebased auth password with ShaPasswordEncoder with salt.
[ https://issues.apache.org/jira/browse/ATLAS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148414#comment-17148414 ] yangchengkai commented on ATLAS-3488: - Hi,I got this error in logs/application.log when I run ./bin/import_hive.sh: 2020-06-30 15:50:24,815 DEBUG - [pool-2-thread-2:] ~ checkPasswordBCrypt() (UserDao:159) 2020-06-30 15:50:24,816 DEBUG - [pool-2-thread-2:] ~ checkPasswordBCrypt(): failed (UserDao:168) java.lang.IllegalArgumentException: Invalid salt version at org.springframework.security.crypto.bcrypt.BCrypt.hashpw(BCrypt.java:556) at org.springframework.security.crypto.bcrypt.BCrypt.checkpw(BCrypt.java:659) at org.apache.atlas.web.dao.UserDao.checkPasswordBCrypt(UserDao.java:165) at org.apache.atlas.web.dao.UserDao.checkEncrypted(UserDao.java:144) at org.apache.atlas.web.security.AtlasFileAuthenticationProvider.authenticate(AtlasFileAuthenticationProvider.java:65) at org.apache.atlas.web.security.AtlasAuthenticationProvider.authenticate(AtlasAuthenticationProvider.java:143) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:142) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.ecli
[jira] [Resolved] (ATLAS-3836) Add Apache Ozone support in hive hook
[ https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3836. --- Resolution: Fixed > Add Apache Ozone support in hive hook > - > > Key: ATLAS-3836 > URL: https://issues.apache.org/jira/browse/ATLAS-3836 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sarath Subramanian >Assignee: Nikhil Bonte >Priority: Major > Labels: hive-hooks, ozone > Fix For: 2.1.0 > > Attachments: Hive_table_lineage.png, > Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, > Ozone_volume.png > > > Apache Ozone is the new object store for Hadoop - > [https://hadoop.apache.org/ozone/] > Apache Atlas needs to add entity types to support creation of Ozone entities. > Hive hook should also be updated to create lineage between ozone entities and > hive tables (for EXTERNAL TABLE) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3836) Add Apache Ozone support in hive hook
[ https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148384#comment-17148384 ] ASF subversion and git services commented on ATLAS-3836: Commit f4a1e078006985c21883f7cef00e7cf94750d96e in atlas's branch refs/heads/branch-2.0 from Nikhil Bonte [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=f4a1e07 ] ATLAS-3836 Add Apache Ozone support in hive hook Signed-off-by: Sarath Subramanian (cherry picked from commit 8b50ac0c7d8eba8efa0deb97f1fee68d382f11ad) > Add Apache Ozone support in hive hook > - > > Key: ATLAS-3836 > URL: https://issues.apache.org/jira/browse/ATLAS-3836 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sarath Subramanian >Assignee: Nikhil Bonte >Priority: Major > Fix For: 2.1.0 > > Attachments: Hive_table_lineage.png, > Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, > Ozone_volume.png > > > Apache Ozone is the new object store for Hadoop - > [https://hadoop.apache.org/ozone/] > Apache Atlas needs to add entity types to support creation of Ozone entities. > Hive hook should also be updated to create lineage between ozone entities and > hive tables (for EXTERNAL TABLE) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook
[ https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3836: -- Labels: hive-hooks ozone (was: ) > Add Apache Ozone support in hive hook > - > > Key: ATLAS-3836 > URL: https://issues.apache.org/jira/browse/ATLAS-3836 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sarath Subramanian >Assignee: Nikhil Bonte >Priority: Major > Labels: hive-hooks, ozone > Fix For: 2.1.0 > > Attachments: Hive_table_lineage.png, > Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, > Ozone_volume.png > > > Apache Ozone is the new object store for Hadoop - > [https://hadoop.apache.org/ozone/] > Apache Atlas needs to add entity types to support creation of Ozone entities. > Hive hook should also be updated to create lineage between ozone entities and > hive tables (for EXTERNAL TABLE) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3836) Add Apache Ozone support in hive hook
[ https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148383#comment-17148383 ] ASF subversion and git services commented on ATLAS-3836: Commit 8b50ac0c7d8eba8efa0deb97f1fee68d382f11ad in atlas's branch refs/heads/master from Nikhil Bonte [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=8b50ac0 ] ATLAS-3836 Add Apache Ozone support in hive hook Signed-off-by: Sarath Subramanian > Add Apache Ozone support in hive hook > - > > Key: ATLAS-3836 > URL: https://issues.apache.org/jira/browse/ATLAS-3836 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sarath Subramanian >Assignee: Nikhil Bonte >Priority: Major > Fix For: 2.1.0 > > Attachments: Hive_table_lineage.png, > Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, > Ozone_volume.png > > > Apache Ozone is the new object store for Hadoop - > [https://hadoop.apache.org/ozone/] > Apache Atlas needs to add entity types to support creation of Ozone entities. > Hive hook should also be updated to create lineage between ozone entities and > hive tables (for EXTERNAL TABLE) -- This message was sent by Atlassian Jira (v8.3.4#803005)
