[jira] [Commented] (ATLAS-3827) UI: Use moment date format instead of default date format.

2020-09-17 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197438#comment-17197438
 ] 

Nikhil Bonte commented on ATLAS-3827:
-

+1 for the patch, thanks [~kevalbhatt].

> UI: Use moment date format instead of default date format.
> --
>
> Key: ATLAS-3827
> URL: https://issues.apache.org/jira/browse/ATLAS-3827
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3827-1.patch, ATLAS-3827.patch, 
> image-2020-09-16-17-11-33-386.png, image-2020-09-16-17-12-04-512.png, 
> new_date_format.png
>
>
>  
> UI configured to use the following date formats:
> {code:java}
>  /MM/DD
>  /MM/DD HH:mm:ss
>  MM/DD/ h:mm A z
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3945) UI: Entity details page, Show N/A for date if date value is 0 or null

2020-09-17 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197436#comment-17197436
 ] 

Nikhil Bonte commented on ATLAS-3945:
-

+1 for the patch, thanks [~kevalbhatt].

> UI: Entity details page,  Show N/A for date if date value is 0 or null
> --
>
> Key: ATLAS-3945
> URL: https://issues.apache.org/jira/browse/ATLAS-3945
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3945.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3932) Upgrade Spring security , libpam4j and jna lib version

2020-09-03 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190094#comment-17190094
 ] 

Nikhil Bonte commented on ATLAS-3932:
-

+1 for the patch.

Thanks [~nixon].

> Upgrade Spring security , libpam4j and jna lib version
> --
>
> Key: ATLAS-3932
> URL: https://issues.apache.org/jira/browse/ATLAS-3932
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 
> 0001-ATLAS-3932-Upgrade-jna-spring.security-and-libpam4j-.patch
>
>
> Upgrade jna lib version from 4.1.0 to 5.2.0
> Upgrade libpam4j lib version from 1.8 to 1.11
> and Upgrade Spring security  version from 4.2.16.RELEASE to  4.2.17.RELEASE



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3913) Swagger documentation for APIs

2020-08-07 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3913:

Description: 
Current packaging does not package swagger ui into Atlas war.

This change will pack swagger ui into the war file.

Also update enunciate version from 2.11.1 to 2.13.1

  was:
Current packaging does not package swagger ui into Atlas war.

This change will pack swagger ui into war.

Also update enunciate version from 2.11.1 to 2.13.1


> Swagger documentation for APIs
> --
>
> Key: ATLAS-3913
> URL: https://issues.apache.org/jira/browse/ATLAS-3913
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Current packaging does not package swagger ui into Atlas war.
> This change will pack swagger ui into the war file.
> Also update enunciate version from 2.11.1 to 2.13.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3913) Swagger documentation for APIs

2020-08-07 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3913:
---

 Summary: Swagger documentation for APIs
 Key: ATLAS-3913
 URL: https://issues.apache.org/jira/browse/ATLAS-3913
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core, atlas-webui
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


Enunciate does not package swagger ui into Atlas war.

This change will pack swagger ui into war.

Also update enunciate version from 2.11.1 to 2.13.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3913) Swagger documentation for APIs

2020-08-07 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3913:

Description: 
Current packaging does not package swagger ui into Atlas war.

This change will pack swagger ui into war.

Also update enunciate version from 2.11.1 to 2.13.1

  was:
Enunciate does not package swagger ui into Atlas war.

This change will pack swagger ui into war.

Also update enunciate version from 2.11.1 to 2.13.1


> Swagger documentation for APIs
> --
>
> Key: ATLAS-3913
> URL: https://issues.apache.org/jira/browse/ATLAS-3913
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Current packaging does not package swagger ui into Atlas war.
> This change will pack swagger ui into war.
> Also update enunciate version from 2.11.1 to 2.13.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3909) UI: Improve readability for the table view in the entity detail page

2020-08-03 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169969#comment-17169969
 ] 

Nikhil Bonte commented on ATLAS-3909:
-

+1 for the patch, thanks [~kevalbhatt].

> UI: Improve readability for the table view in the entity detail page 
> -
>
> Key: ATLAS-3909
> URL: https://issues.apache.org/jira/browse/ATLAS-3909
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3909.patch, image-2020-08-03-12-48-10-984.png, 
> image-2020-08-03-12-48-28-192.png
>
>
> Change the key style to bold for table view in the entity details page. 
>  
> !image-2020-08-03-12-48-10-984.png|width=508,height=311!
> *Expected:*
> !image-2020-08-03-12-48-28-192.png|width=541,height=341!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Issue Comment Deleted] (ATLAS-3909) UI: Improve readability for the table view in the entity detail page

2020-08-03 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3909:

Comment: was deleted

(was: +1 for the patch, thanks [~kevalbhatt] .)

> UI: Improve readability for the table view in the entity detail page 
> -
>
> Key: ATLAS-3909
> URL: https://issues.apache.org/jira/browse/ATLAS-3909
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2020-08-03-12-48-10-984.png, 
> image-2020-08-03-12-48-28-192.png
>
>
> Change the key style to bold for table view in the entity details page. 
>  
> !image-2020-08-03-12-48-10-984.png|width=508,height=311!
> *Expected:*
> !image-2020-08-03-12-48-28-192.png|width=541,height=341!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3909) UI: Improve readability for the table view in the entity detail page

2020-08-03 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169961#comment-17169961
 ] 

Nikhil Bonte commented on ATLAS-3909:
-

+1 for the patch, thanks [~kevalbhatt] .

> UI: Improve readability for the table view in the entity detail page 
> -
>
> Key: ATLAS-3909
> URL: https://issues.apache.org/jira/browse/ATLAS-3909
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3909.patch, image-2020-08-03-12-48-10-984.png, 
> image-2020-08-03-12-48-28-192.png
>
>
> Change the key style to bold for table view in the entity details page. 
>  
> !image-2020-08-03-12-48-10-984.png|width=508,height=311!
> *Expected:*
> !image-2020-08-03-12-48-28-192.png|width=541,height=341!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3879) Ozone: ozone_key entity is directly created under ozone_bucket

2020-07-22 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3879:

Description: 
The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

 

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_volume)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_volume)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.

  was:
The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

 

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.


> Ozone: ozone_key entity is directly created under ozone_bucket
> --
>
> Key: ATLAS-3879
> URL: https://issues.apache.org/jira/browse/ATLAS-3879
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
>
> The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
> entity is getting created under ozone_bucket, but there is no support to 
> create a hierarchy of parent ozone_key and child ozone_key.
>  
> e.g.
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields 
> terminated by ' ' stored as textfile location 
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
> This query will create only 1 ozone_key directly under bucket "bucket1".
> Created entities :
>  * o3fs://volume1@cm (ozone_volume)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)
>  
> This Jira will improve implementation to support creating the hierarchy of 
> ozone_keys.
> With this, above query will result into following entites
>  * o3fs://volume1@cm (ozone_volume)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
> ozone_key*
>  
> This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
> models.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3885) import-hive.sh: Hive entities with Ozone location created by import-hive.sh creates hdfs_path entity instead of ozone in Atlas

2020-07-08 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3885:

Description: 
import-hive.sh when attempts to import hive entities with Ozone location 
expected is that ozone_key should get created as path entity, but script 
creates hdfs_path entity instead of ozone_key in Atlas.

 

Steps to reproduce
 # Disable HMS Hook and Hive Hook
 # Create external table with ozone location
 # Run import-hive.sh

The location entity is created as an hdfs_path entity instead of ozone_key.

  was:
import-hive.sh when attempts to import hive entities with Ozone location 
expected is that ozone_key should get created as path entity, but script 
creates hdfs_path entity instead of ozone_key in Atlas.

 

Steps to reproduce
 # Disable HMS Hook and Hive Hook
 # Create external table with ozone location
 # Run import-hive.sh

 

The location entity is created as an hdfs_path entity instead of ozone_key.


> import-hive.sh: Hive entities with Ozone location created by import-hive.sh 
> creates hdfs_path entity instead of ozone in Atlas
> --
>
> Key: ATLAS-3885
> URL: https://issues.apache.org/jira/browse/ATLAS-3885
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: ozone
>
> import-hive.sh when attempts to import hive entities with Ozone location 
> expected is that ozone_key should get created as path entity, but script 
> creates hdfs_path entity instead of ozone_key in Atlas.
>  
> Steps to reproduce
>  # Disable HMS Hook and Hive Hook
>  # Create external table with ozone location
>  # Run import-hive.sh
> The location entity is created as an hdfs_path entity instead of ozone_key.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3885) import-hive.sh: Hive entities with Ozone location created by import-hive.sh creates hdfs_path entity instead of ozone in Atlas

2020-07-08 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3885:

Description: 
import-hive.sh when attempts to import hive entities with Ozone location 
expected is that ozone_key should get created as path entity, but script 
creates hdfs_path entity instead of ozone_key in Atlas.

 

Steps to reproduce
 # Disable HMS Hook and Hive Hook
 # Create external table with ozone location
 # Run import-hive.sh

 

The location entity is created as an hdfs_path entity instead of ozone_key.

  was:
import-hive.sh when attempts to import hive entities with Ozone location 
expected is that ozone_key will get created as path entity, but script creates 
hdfs_path entity instead of ozone_key in Atlas.

 

Steps to reproduce
 # Disable HMS Hook and Hive Hook
 # Create external table with ozone location
 # Run import-hive.sh

 

The location entity is created as an hdfs_path entity instead of ozone_key.


> import-hive.sh: Hive entities with Ozone location created by import-hive.sh 
> creates hdfs_path entity instead of ozone in Atlas
> --
>
> Key: ATLAS-3885
> URL: https://issues.apache.org/jira/browse/ATLAS-3885
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: ozone
>
> import-hive.sh when attempts to import hive entities with Ozone location 
> expected is that ozone_key should get created as path entity, but script 
> creates hdfs_path entity instead of ozone_key in Atlas.
>  
> Steps to reproduce
>  # Disable HMS Hook and Hive Hook
>  # Create external table with ozone location
>  # Run import-hive.sh
>  
> The location entity is created as an hdfs_path entity instead of ozone_key.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3885) import-hive.sh: Hive entities with Ozone location created by import-hive.sh creates hdfs_path entity instead of ozone in Atlas

2020-07-08 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3885:
---

 Summary: import-hive.sh: Hive entities with Ozone location created 
by import-hive.sh creates hdfs_path entity instead of ozone in Atlas
 Key: ATLAS-3885
 URL: https://issues.apache.org/jira/browse/ATLAS-3885
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


import-hive.sh when attempts to import hive entities with Ozone location 
expected is that ozone_key will get created as path entity, but script creates 
hdfs_path entity instead of ozone_key in Atlas.

 

Steps to reproduce
 # Disable HMS Hook and Hive Hook
 # Create external table with ozone location
 # Run import-hive.sh

 

The location entity is created as an hdfs_path entity instead of ozone_key.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-07-07 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt' into 
table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

*Note:* The approach has been updated in ATLAS-3879

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt' into 
table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new 

[jira] [Updated] (ATLAS-3879) Ozone: ozone_key entity is directly created under ozone_bucket

2020-07-07 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3879:

Description: 
The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

 

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.

  was:
Apache ozone support addded in ATLAS-3836

Currently ozone_key entity is getting created under ozone_bucket.

No support to create ozone_key under ozone_key.

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support add ozone_key under ozone_key.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.


> Ozone: ozone_key entity is directly created under ozone_bucket
> --
>
> Key: ATLAS-3879
> URL: https://issues.apache.org/jira/browse/ATLAS-3879
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
> entity is getting created under ozone_bucket, but there is no support to 
> create a hierarchy of parent ozone_key and child ozone_key.
>  
> e.g.
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields 
> terminated by ' ' stored as textfile location 
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
> This query will create only 1 ozone_key directly under bucket "bucket1".
> Created entities :
>  * o3fs://volume1@cm (ozone_key)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)
>  
> This Jira will improve implementation to support creating the hierarchy of 
> ozone_keys.
> With this, above query will result into following entites
>  * o3fs://volume1@cm (ozone_key)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
> ozone_key*
>  
> This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
> models.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3879) Ozone: ozone_key entity is directly created under ozone_bucket

2020-07-07 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3879:

Description: 
Apache ozone support addded in ATLAS-3836

Currently ozone_key entity is getting created under ozone_bucket.

No support to create ozone_key under ozone_key.

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support add ozone_key under ozone_key.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.

  was:
Apache ozone support addded in ATLAS-3836

Currently ozone_key entity is getting created under ozone_bucket.

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support add ozone_key under ozone_key.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.


> Ozone: ozone_key entity is directly created under ozone_bucket
> --
>
> Key: ATLAS-3879
> URL: https://issues.apache.org/jira/browse/ATLAS-3879
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Apache ozone support addded in ATLAS-3836
> Currently ozone_key entity is getting created under ozone_bucket.
> No support to create ozone_key under ozone_key.
> e.g.
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields 
> terminated by ' ' stored as textfile location 
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
> This query will create only 1 ozone_key directly under bucket "bucket1".
> Created entities :
>  * o3fs://volume1@cm (ozone_key)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)
>  
> This Jira will improve implementation to support add ozone_key under 
> ozone_key.
> With this, above query will result into following entites
>  * o3fs://volume1@cm (ozone_key)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
> ozone_key*
>  
> This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
> models.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3879) Ozone: ozone_key entity is directly created under ozone_bucket

2020-07-07 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3879:

Description: 
Apache ozone support addded in ATLAS-3836

Currently ozone_key entity is getting created under ozone_bucket.

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support add ozone_key under ozone_key.

With this, above query will result into following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.

  was:
Apache ozone support addded in ATLAS-3836

Currently ozone_key entity is getting created under ozone_bucket.

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Entities will be
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support add ozone_key under ozone_key.

With this above query will create following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.


> Ozone: ozone_key entity is directly created under ozone_bucket
> --
>
> Key: ATLAS-3879
> URL: https://issues.apache.org/jira/browse/ATLAS-3879
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Apache ozone support addded in ATLAS-3836
> Currently ozone_key entity is getting created under ozone_bucket.
> e.g.
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields 
> terminated by ' ' stored as textfile location 
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
> This query will create only 1 ozone_key directly under bucket "bucket1".
> Created entities :
>  * o3fs://volume1@cm (ozone_key)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)
>  
> This Jira will improve implementation to support add ozone_key under 
> ozone_key.
> With this, above query will result into following entites
>  * o3fs://volume1@cm (ozone_key)
>  * o3fs://volume1.bucket1@cm (ozone_bucket)
>  * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
>  * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
> ozone_key*
>  
> This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
> models.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3879) Ozone: ozone_key entity is directly created under ozone_bucket

2020-07-07 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3879:
---

 Summary: Ozone: ozone_key entity is directly created under 
ozone_bucket
 Key: ATLAS-3879
 URL: https://issues.apache.org/jira/browse/ATLAS-3879
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


Apache ozone support addded in ATLAS-3836

Currently ozone_key entity is getting created under ozone_bucket.

e.g.
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
This query will create only 1 ozone_key directly under bucket "bucket1".

Entities will be
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support add ozone_key under ozone_key.

With this above query will create following entites
 * o3fs://volume1@cm (ozone_key)
 * o3fs://volume1.bucket1@cm (ozone_bucket)
 * o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) *child of ozone_bucket*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) *child of ozone_key*
 * o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) *child of 
ozone_key*

 

This will bring Ozone model to be consistent with AWS S3 v2 and ADLS-Gen2 
models.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3869) Upgrade httpclient from 4.5.3 to 4.5.4

2020-06-30 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149096#comment-17149096
 ] 

Nikhil Bonte commented on ATLAS-3869:
-

+1 for the patch.

Thanks [~nixon] .

> Upgrade httpclient from 4.5.3 to 4.5.4
> --
>
> Key: ATLAS-3869
> URL: https://issues.apache.org/jira/browse/ATLAS-3869
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: ATLAS-3869-Upgrade-httpclient-from-4.5.3-to-4.5.4.patch
>
>
> Found below exception where data was not getting added in SOLR from 
> janusgraph.
>  
> {noformat}
> Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String
>   at 
> org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)
>   at 
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112)
>   at 
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99)
>   at 
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:463)
>   at 
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
>   at 
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
>   at 
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
>   at 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
>   at 
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
>   at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
>   at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>   at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>   at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
>   at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
>   at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>   ... 7 more
> 2020-06-29 07:31:25,893 WARN  - [SIGTERM handler:] ~ Unable to close 
> transaction standardjanusgraphtx[0x18d7a16a] (StandardJanusGraph:230)
> java.lang.IllegalArgumentException: The transaction has already been closed
>   at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
>   at 
> org.janusgraph.graphdb.transaction.StandardJanusGraphTx.rollback(StandardJanusGraphTx.java:1401)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doRollback(JanusGraphBlueprintsTransaction.java:190)
>   at 
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.rollback(AbstractTransaction.java:114)
>   at 
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:182)
>   at 
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:179)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.close(JanusGraphBlueprintsGraph.java:327)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doClose(JanusGraphBlueprintsTransaction.java:205)
>   at 
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.close(AbstractTransaction.java:139)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction.close(JanusGraphBlueprintsTransaction.java:215)
>   at 
> org.janusgraph.graphdb.database.StandardJanusGraph.closeInternal(StandardJanusGraph.java:226)
>   at 
> org.janusgraph.graphdb.database.StandardJanusGraph.access$500(StandardJanusGraph.java:91)
>   at 
> org.janusgraph.graphdb.database.StandardJanusGraph$ShutdownThread.start(StandardJanusGraph.java:818)
>   at 
> java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:102)
>   at 
> java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
>   at java.lang.Shutdown.runHooks(Shutdown.java:123)
>  

[jira] [Updated] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3871:

Description: 
AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
ozone & hdfs path entites.

 

This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls 
path entities.

Would add more tests for ozone as well.

  was:
AtlasPathExtractorUtilTest added in ATLAS-3836 which covers ozone & hdfs path 
entites unit tests.

This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls 
path entities.

Also will add more tests for ozone path.


> Add unit tests to path extractor utility for s3, s3 v2, adls path entities
> --
>
> Key: ATLAS-3871
> URL: https://issues.apache.org/jira/browse/ATLAS-3871
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
> ozone & hdfs path entites.
>  
> This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & 
> adls path entities.
> Would add more tests for ozone as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3871:

Description: 
AtlasPathExtractorUtilTest added in ATLAS-3836 which covers ozone & hdfs path 
entites unit tests.

This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls 
path entities.

Also will add more tests for ozone path.

  was:
AtlasPathExtractorUtilTest is added in ATLAS-3836 which covers ozone & hdfs 
path entites unit tests.

This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls 
path entities.

Also will add more tests for ozone path.


> Add unit tests to path extractor utility for s3, s3 v2, adls path entities
> --
>
> Key: ATLAS-3871
> URL: https://issues.apache.org/jira/browse/ATLAS-3871
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> AtlasPathExtractorUtilTest added in ATLAS-3836 which covers ozone & hdfs path 
> entites unit tests.
> This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & 
> adls path entities.
> Also will add more tests for ozone path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3871:

Summary: Add unit tests to path extractor utility for s3, s3 v2, adls path 
entities  (was: Add unit tests for s3, s3 v2, adls path entities in path 
extractor utility)

> Add unit tests to path extractor utility for s3, s3 v2, adls path entities
> --
>
> Key: ATLAS-3871
> URL: https://issues.apache.org/jira/browse/ATLAS-3871
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> AtlasPathExtractorUtilTest is added in ATLAS-3836 which covers ozone & hdfs 
> path entites unit tests.
> This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & 
> adls path entities.
> Also will add more tests for ozone path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3871) Add unit tests for s3, s3 v2, adls path entities in path extractor utility

2020-06-30 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3871:
---

 Summary: Add unit tests for s3, s3 v2, adls path entities in path 
extractor utility
 Key: ATLAS-3871
 URL: https://issues.apache.org/jira/browse/ATLAS-3871
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


AtlasPathExtractorUtilTest is added in ATLAS-3836 which covers ozone & hdfs 
path entites unit tests.

This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls 
path entities.

Also will add more tests for ozone path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> 

[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt' into 
table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> 

[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales|o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> 

[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
 

 
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
 
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)


+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path 
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

 

 


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)
> +*Approach :*+
>  # Refactored BaseHiveEvent.getPathEntity() -> moved to 
> AtlasPathExtractorUtil.java
>  # Created PathExtractorContext.java to wrap most arguments.
>  # AtlasPathExtractorUtil.getPathEntity() -> accept Path, 
> PathExtractorContext -> return AtlasEntityWithExtInfo
>  # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to 
> handle Ozone path
>  -> path starts with "ofs://" or "o3fs://"
>  # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
> AtlasPathExtractorUtilTest.java
>  
> +*Examples :*+
> {code:java}
> -> CREATE EXTERNAL TABLE sales (id int) row format delimited fields 
> terminated by ' ' stored as textfile location 
> 'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
>  
>  
> || ||Name||Qualified Name||
> |ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
> |ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
> |ozone_volume|volume1|o3fs://volume1@cm|
>  
>  
> {code:java}
> -> create EXTERNAL table stocks (id int) row 

[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)

+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

+*Examples :*+
{code:java}
-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';{code}
 

 
|| ||Name||Qualified Name||
|ozone_key|/sale1/q1/sales2|o3fs://bucket1.volume1.ozone1/sale1/q1/sales2@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 
{code:java}
-> create EXTERNAL table stocks (id int) row format delimited fields terminated 
by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket1.volume1.ozone1/stocks.txt' into table 
stocks;
{code}
 
|| ||Name||Qualified Name||
|ozone_key|/stocks.txt|o3fs://bucket1.volume1.ozone1/stocks.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 
{code:java}
-> create EXTERNAL table stocks_q1 (id int) row format delimited fields 
terminated by ' ' stored as textfile;
-> load data inpath 'o3fs://bucket15.volume15.ozone1/quarter1/stocks_q1.txt' 
into table stocks_q1;{code}
|| ||Name||Qualified Name||
|ozone_key|/quarter1/stocks_q1.txt|o3fs://bucket1.volume1.ozone1/quarter1/stocks_q1.txt@cm|
|ozone_bucket|bucket1|o3fs://volume1.bucket1@cm|
|ozone_volume|volume1|o3fs://volume1@cm|

 

 


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> 

[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Description: 
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)


+*Approach :*+
 # Refactored BaseHiveEvent.getPathEntity() -> moved to 
AtlasPathExtractorUtil.java
 # Created PathExtractorContext.java to wrap most arguments.
 # AtlasPathExtractorUtil.getPathEntity() -> accept Path, PathExtractorContext 
-> return AtlasEntityWithExtInfo
 # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to handle 
Ozone path 
 -> path starts with "ofs://" or "o3fs://"
 # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
AtlasPathExtractorUtilTest.java

 

 

 

  was:
Apache Ozone is the new object store for Hadoop - 
[https://hadoop.apache.org/ozone/]

Apache Atlas needs to add entity types to support creation of Ozone entities. 
Hive hook should also be updated to create lineage between ozone entities and 
hive tables (for EXTERNAL TABLE)


> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)
> +*Approach :*+
>  # Refactored BaseHiveEvent.getPathEntity() -> moved to 
> AtlasPathExtractorUtil.java
>  # Created PathExtractorContext.java to wrap most arguments.
>  # AtlasPathExtractorUtil.getPathEntity() -> accept Path, 
> PathExtractorContext -> return AtlasEntityWithExtInfo
>  # Added specific condition in AtlasPathExtractorUtil.getPathEntity() to 
> handle Ozone path 
>  -> path starts with "ofs://" or "o3fs://"
>  # Added UT around AtlasPathExtractorUtil.getPathEntity() -> 
> AtlasPathExtractorUtilTest.java
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3856) Create new entity typeDefs for Apache Ozone

2020-06-26 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3856:

Description: 
Add following ozone typeDefs to Atlas models
 * entityDefs:
 ** ozone_volume
 ** ozone_bucket
 ** ozone_key
 * enumDefs:
 ** ozone_storage_type
 ** ozone_replication_type
 * structDefs:
 ** ozone_key_location

 * relationshipDefs:
 ** ozone_volume_buckets
 ** ozone_bucket_keys

  was:
Add following ozone typeDefs to Atlas models
 * entityDefs:
 ** ozone_volume
 ** ozone_bucket
 ** ozone_key
 * enumDefs:
 ** ozone_storage_type
 ** ozone_replication_type
 * structDefs:

 ** ozone_key_location

 * relationshipDefs:

 ** ozone_volume_buckets
 ** ozone_bucket_keys


> Create new entity typeDefs for Apache Ozone
> ---
>
> Key: ATLAS-3856
> URL: https://issues.apache.org/jira/browse/ATLAS-3856
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: 
> ATLAS-3856-Create-new-entity-typeDefs-for-Apache-Ozone-v0.patch
>
>
> Add following ozone typeDefs to Atlas models
>  * entityDefs:
>  ** ozone_volume
>  ** ozone_bucket
>  ** ozone_key
>  * enumDefs:
>  ** ozone_storage_type
>  ** ozone_replication_type
>  * structDefs:
>  ** ozone_key_location
>  * relationshipDefs:
>  ** ozone_volume_buckets
>  ** ozone_bucket_keys



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3856) Create new entity typeDefs for Apache Ozone

2020-06-26 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3856:

Description: 
Add following ozone typeDefs to Atlas models
 * entityDefs:
 ** ozone_volume
 ** ozone_bucket
 ** ozone_key
 * enumDefs:
 ** ozone_storage_type
 ** ozone_replication_type
 * structDefs:

 ** ozone_key_location

 * relationshipDefs:

 ** ozone_volume_buckets
 ** ozone_bucket_keys

  was:
Add following ozone typeDefs to Atlas models

 * Ozone volume
 * Ozone bucket
 * Ozone key


> Create new entity typeDefs for Apache Ozone
> ---
>
> Key: ATLAS-3856
> URL: https://issues.apache.org/jira/browse/ATLAS-3856
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: 
> ATLAS-3856-Create-new-entity-typeDefs-for-Apache-Ozone-v0.patch
>
>
> Add following ozone typeDefs to Atlas models
>  * entityDefs:
>  ** ozone_volume
>  ** ozone_bucket
>  ** ozone_key
>  * enumDefs:
>  ** ozone_storage_type
>  ** ozone_replication_type
>  * structDefs:
>  ** ozone_key_location
>  * relationshipDefs:
>  ** ozone_volume_buckets
>  ** ozone_bucket_keys



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-26 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Attachment: Ozone_key.png
Ozone_bucket.png
Ozone_volume.png
Hive_table_lineage.png
Hive_table_lineage_load_in_path.png

> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (ATLAS-3852) Entity Bulk Create with unique reference

2020-06-25 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144830#comment-17144830
 ] 

Nikhil Bonte edited comment on ATLAS-3852 at 6/25/20, 10:51 AM:


[~vasanthkumar] - Currently Atlas does support creating entities which are 
referenced in same bulk request using negative guid approach.

[^BulkPostCreate.json]

I have attached sample payload which will create new a hive_db, a hive_table & 
a hive_column in single request.

Hope this would help.

 


was (Author: nikhilbonte):
[~vasanthkumar] - Currently Atlas does support creating entities with 
referenced in same bulk request using negative guid approach.

[^BulkPostCreate.json]

I have attached sample payload which will create new a hive_db, a hive_table & 
a hive_column in single request.

Hope this would help.

 

> Entity Bulk Create with unique reference
> 
>
> Key: ATLAS-3852
> URL: https://issues.apache.org/jira/browse/ATLAS-3852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Vasanth kumar RJ
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: ATLAS-3852.patch, BulkPostCreate.json
>
>
> Entities created in bulk and unique referenced in same request. Bulk create 
> DB, Table and Column referenced in a request itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3852) Entity Bulk Create with unique reference

2020-06-25 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144830#comment-17144830
 ] 

Nikhil Bonte commented on ATLAS-3852:
-

[~vasanthkumar] - Currently Atlas does support creating entities with 
referenced in same bulk request using negative guid approach.

[^BulkPostCreate.json]

I have attached sample payload which will create new a hive_db, a hive_table & 
a hive_column in single request.

Hope this would help.

 

> Entity Bulk Create with unique reference
> 
>
> Key: ATLAS-3852
> URL: https://issues.apache.org/jira/browse/ATLAS-3852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Vasanth kumar RJ
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: ATLAS-3852.patch, BulkPostCreate.json
>
>
> Entities created in bulk and unique referenced in same request. Bulk create 
> DB, Table and Column referenced in a request itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3852) Entity Bulk Create with unique reference

2020-06-25 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3852:

Attachment: BulkPostCreate.json

> Entity Bulk Create with unique reference
> 
>
> Key: ATLAS-3852
> URL: https://issues.apache.org/jira/browse/ATLAS-3852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Vasanth kumar RJ
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: ATLAS-3852.patch, BulkPostCreate.json
>
>
> Entities created in bulk and unique referenced in same request. Bulk create 
> DB, Table and Column referenced in a request itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3856) Create new entity typeDefs for Apache Ozone

2020-06-22 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3856:

Attachment: ATLAS-3856-Create-new-entity-typeDefs-for-Apache-Ozone-v0.patch

> Create new entity typeDefs for Apache Ozone
> ---
>
> Key: ATLAS-3856
> URL: https://issues.apache.org/jira/browse/ATLAS-3856
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3856-Create-new-entity-typeDefs-for-Apache-Ozone-v0.patch
>
>
> Add following ozone typeDefs to Atlas models
>  * Ozone volume
>  * Ozone bucket
>  * Ozone key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3856) Create new entity typeDefs for Apache Ozone

2020-06-22 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3856:

Summary: Create new entity typeDefs for Apache Ozone  (was: Create new 
entity types for Apache Ozone)

> Create new entity typeDefs for Apache Ozone
> ---
>
> Key: ATLAS-3856
> URL: https://issues.apache.org/jira/browse/ATLAS-3856
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Add following ozone typeDefs to Atlas models
>  * Ozone volume
>  * Ozone bucket
>  * Ozone key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-22 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3836:

Summary: Add Apache Ozone support in hive hook  (was: Create new entity 
types for Apache Ozone and update hive hook to handle reference to Ozone path)

> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3856) Create new entity types for Apache Ozone

2020-06-22 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3856:
---

 Summary: Create new entity types for Apache Ozone
 Key: ATLAS-3856
 URL: https://issues.apache.org/jira/browse/ATLAS-3856
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


Add following ozone typeDefs to Atlas models

 * Ozone volume
 * Ozone bucket
 * Ozone key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3849) UI: Create icons for new entity types for Apache ozone

2020-06-19 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3849:

Description: 
New typeDefs:
 * ozone_volume
 * ozone_bucket
 * ozone_key

> UI: Create icons for new entity types for Apache ozone
> --
>
> Key: ATLAS-3849
> URL: https://issues.apache.org/jira/browse/ATLAS-3849
> Project: Atlas
>  Issue Type: New Feature
>  Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Keval Bhatt
>Priority: Major
>
> New typeDefs:
>  * ozone_volume
>  * ozone_bucket
>  * ozone_key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3849) UI: Create icons for new entity types for Apache ozone

2020-06-19 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3849:

Affects Version/s: (was: 2.1.0)
   2.0.0

> UI: Create icons for new entity types for Apache ozone
> --
>
> Key: ATLAS-3849
> URL: https://issues.apache.org/jira/browse/ATLAS-3849
> Project: Atlas
>  Issue Type: New Feature
>  Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Keval Bhatt
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3849) UI: Create icons for new entity types for Apache ozone

2020-06-19 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3849:
---

 Summary: UI: Create icons for new entity types for Apache ozone
 Key: ATLAS-3849
 URL: https://issues.apache.org/jira/browse/ATLAS-3849
 Project: Atlas
  Issue Type: New Feature
  Components: atlas-webui
Affects Versions: 2.1.0
Reporter: Nikhil Bonte
Assignee: Keval Bhatt






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (ATLAS-3836) Create new entity types for Apache Ozone and update hive hook to handle reference to Ozone path

2020-06-19 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte reassigned ATLAS-3836:
---

Assignee: Nikhil Bonte  (was: Sarath Subramanian)

> Create new entity types for Apache Ozone and update hive hook to handle 
> reference to Ozone path
> ---
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3831) UI: Add entity icon for spark

2020-06-10 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130523#comment-17130523
 ] 

Nikhil Bonte commented on ATLAS-3831:
-

+1 for the patch, thanks [~kevalbhatt] for the patch.

> UI: Add entity icon for spark
> -
>
> Key: ATLAS-3831
> URL: https://issues.apache.org/jira/browse/ATLAS-3831
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3831.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3769) UI: Changing page limit in Admin audits resets any filters applied on that page

2020-06-10 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130522#comment-17130522
 ] 

Nikhil Bonte commented on ATLAS-3769:
-

+1 for the patch, thanks [~kevalbhatt] for the patch.

> UI: Changing page limit in Admin audits resets any filters applied on that 
> page
> ---
>
> Key: ATLAS-3769
> URL: https://issues.apache.org/jira/browse/ATLAS-3769
> Project: Atlas
>  Issue Type: Bug
>Reporter: Rahul Kurup
>Assignee: Keval Bhatt
>Priority: Minor
> Attachments: ATLAS-3769.patch
>
>
> If any filters are set in the Audits Page in the Administration menu, 
> changing the page limit in Admin audits resets the filters applied on that 
> page.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3829) UI: Clicking on meanings from relationship tab should redirect to terms details page.

2020-06-10 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130519#comment-17130519
 ] 

Nikhil Bonte commented on ATLAS-3829:
-

+1 for the patch, thanks [~kevalbhatt] for the patch.

> UI: Clicking on meanings from relationship tab should redirect to terms 
> details page.
> -
>
> Key: ATLAS-3829
> URL: https://issues.apache.org/jira/browse/ATLAS-3829
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3829.patch, image-2020-06-09-17-14-17-856.png, 
> image-2020-06-09-17-14-17-989.png
>
>
> * Assign term to Entity
>  * Go to relationship tab click on meanings link
>  * It should redirect to term detail page not entity detail page.
>  
> !image-2020-06-09-17-14-17-989.png|width=564,height=343!!image-2020-06-09-17-14-17-856.png|width=555,height=334!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-06-05 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3816:

Description: 
*Background:*

Atlas instance with property 
_"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

Query will be :  
*$v$*"Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

Default value for index search prefix is _"$v$"_

Can be changed using the property _"atlas.graph.index.search.vertex.prefix"_

 

  was:
*Background:*

Atlas with property 
_"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

Query will be :  
*$v$*"Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

Default value for index search prefix is "$v$"

Can be changed using the property "atlas.graph.index.search.vertex.prefix"

 


> Backport: Basic search issue with IndexSerializer
> -
>
> Key: ATLAS-3816
> URL: https://issues.apache.org/jira/browse/ATLAS-3816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3816-Backport-Basic-search-issue-with-IndexSer-v2.patch
>
>
> *Background:*
> Atlas instance with property 
> _"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_
> Direct index query which has pattern *"v."* throws exception, e.g.
>  
> v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> This will throw 
> {code:java}
> java.lang.IllegalArgumentException: Found reference to non-existant property 
> key in query at position...{code}
>  
> *Solution:*
> Backporting  ATLAS-2371.
> It is resolved with janusgraph, for titan implementation backporting required.
> Query will be :  
> *$v$*"Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> Default value for index search prefix is _"$v$"_
> Can be changed using the property _"atlas.graph.index.search.vertex.prefix"_
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-06-05 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3816:

Description: 
*Background:*

Atlas with property 
_"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

Query will be :  
*$v$*"Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

Default value for index search prefix is "$v$"

Can be changed using the property "atlas.graph.index.search.vertex.prefix"

 

  was:
*Background:*

Atlas with property 
_"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

 


> Backport: Basic search issue with IndexSerializer
> -
>
> Key: ATLAS-3816
> URL: https://issues.apache.org/jira/browse/ATLAS-3816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3816-Backport-Basic-search-issue-with-IndexSer-v2.patch
>
>
> *Background:*
> Atlas with property 
> _"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_
> Direct index query which has pattern *"v."* throws exception, e.g.
>  
> v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> This will throw 
> {code:java}
> java.lang.IllegalArgumentException: Found reference to non-existant property 
> key in query at position...{code}
>  
> *Solution:*
> Backporting  ATLAS-2371.
> It is resolved with janusgraph, for titan implementation backporting required.
> Query will be :  
> *$v$*"Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> Default value for index search prefix is "$v$"
> Can be changed using the property "atlas.graph.index.search.vertex.prefix"
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3240) Update 'http' urls to 'https' in maven settings

2020-06-04 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3240:

Attachment: ATLAS-3240-Update-http-urls-to-https-in-maven-settin-#2.patch

> Update 'http' urls to 'https' in maven settings
> ---
>
> Key: ATLAS-3240
> URL: https://issues.apache.org/jira/browse/ATLAS-3240
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Minor
> Fix For: 1.2.0, 2.1.0
>
> Attachments: 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-#2.patch, 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3240) Update 'http' urls to 'https' in maven settings

2020-06-04 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3240:

Attachment: 
ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-#2.patch

> Update 'http' urls to 'https' in maven settings
> ---
>
> Key: ATLAS-3240
> URL: https://issues.apache.org/jira/browse/ATLAS-3240
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Minor
> Fix For: 1.2.0, 2.1.0
>
> Attachments: 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-#2.patch, 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3240) Update 'http' urls to 'https' in maven settings

2020-06-04 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3240:

Attachment: (was: 
ATLAS-3240-Update-http-urls-to-https-in-maven-settin-#2.patch)

> Update 'http' urls to 'https' in maven settings
> ---
>
> Key: ATLAS-3240
> URL: https://issues.apache.org/jira/browse/ATLAS-3240
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Minor
> Fix For: 1.2.0, 2.1.0
>
> Attachments: 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-#2.patch, 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (ATLAS-3240) Update 'http' urls to 'https' in maven settings

2020-06-04 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125677#comment-17125677
 ] 

Nikhil Bonte edited comment on ATLAS-3240 at 6/4/20, 10:19 AM:
---

Attached patch to backport this to branch-0.8 
[^ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch].

 

PC build : 
[https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1950/console]

(Finished: SUCCESS)


was (Author: nikhilbonte):
Attached patch to backport this to branch-0.8 
[^ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch].

> Update 'http' urls to 'https' in maven settings
> ---
>
> Key: ATLAS-3240
> URL: https://issues.apache.org/jira/browse/ATLAS-3240
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Minor
> Fix For: 1.2.0, 2.1.0
>
> Attachments: 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3240) Update 'http' urls to 'https' in maven settings

2020-06-04 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3240:

Attachment: 
ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch

> Update 'http' urls to 'https' in maven settings
> ---
>
> Key: ATLAS-3240
> URL: https://issues.apache.org/jira/browse/ATLAS-3240
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Minor
> Fix For: 1.2.0, 2.1.0
>
> Attachments: 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3240) Update 'http' urls to 'https' in maven settings

2020-06-04 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125677#comment-17125677
 ] 

Nikhil Bonte commented on ATLAS-3240:
-

Attached patch to backport this to branch-0.8 
[^ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch].

> Update 'http' urls to 'https' in maven settings
> ---
>
> Key: ATLAS-3240
> URL: https://issues.apache.org/jira/browse/ATLAS-3240
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Minor
> Fix For: 1.2.0, 2.1.0
>
> Attachments: 
> ATLAS-3240-Update-http-urls-to-https-in-maven-branch-0.8-v0.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-05-28 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3816:

Description: 
*Background:*

Atlas with property 
_"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

 

  was:
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

 


> Backport: Basic search issue with IndexSerializer
> -
>
> Key: ATLAS-3816
> URL: https://issues.apache.org/jira/browse/ATLAS-3816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> *Background:*
> Atlas with property 
> _"atlas.use.index.query.to.find.entity.by.unique.attributes=true"_
> Direct index query which has pattern *"v."* throws exception, e.g.
>  
> v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> This will throw 
> {code:java}
> java.lang.IllegalArgumentException: Found reference to non-existant property 
> key in query at position...{code}
>  
> *Solution:*
> Backporting  ATLAS-2371.
> It is resolved with janusgraph, for titan implementation backporting required.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-05-28 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3816:

Description: 
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting  ATLAS-2371.

It is resolved with janusgraph, for titan implementation backporting required.

 

  was:
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting direct index query issue resolved in ATLAS-2371.

 


> Backport: Basic search issue with IndexSerializer
> -
>
> Key: ATLAS-3816
> URL: https://issues.apache.org/jira/browse/ATLAS-3816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> *Background:*
> Direct index query which has pattern *"v."* throws exception, e.g.
>  
> v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> This will throw 
> {code:java}
> java.lang.IllegalArgumentException: Found reference to non-existant property 
> key in query at position...{code}
>  
> *Solution:*
> Backporting  ATLAS-2371.
> It is resolved with janusgraph, for titan implementation backporting required.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-05-28 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3816:

Description: 
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting direct index query issue resolved in ATLAS-2371.

 

  was:
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in*{color:#FF}v.{color}*as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting direct index query issue resolved in ATLAS-2371.

 


> Backport: Basic search issue with IndexSerializer
> -
>
> Key: ATLAS-3816
> URL: https://issues.apache.org/jira/browse/ATLAS-3816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> *Background:*
> Direct index query which has pattern *"v."* throws exception, e.g.
>  
> v."Referenceable.qualifiedName":"atscale_agg_in{color:#ff}v.{color}as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> This will throw 
> {code:java}
> java.lang.IllegalArgumentException: Found reference to non-existant property 
> key in query at position...{code}
>  
> *Solution:*
> Backporting direct index query issue resolved in ATLAS-2371.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-05-28 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3816:

Description: 
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in*{color:#FF}v.{color}*as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting direct index query issue resolved in ATLAS-2371.

 

  was:
*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in*v.*as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting direct index query issue resolved in ATLAS-2371.

 


> Backport: Basic search issue with IndexSerializer
> -
>
> Key: ATLAS-3816
> URL: https://issues.apache.org/jira/browse/ATLAS-3816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> *Background:*
> Direct index query which has pattern *"v."* throws exception, e.g.
>  
> v."Referenceable.qualifiedName":"atscale_agg_in*{color:#FF}v.{color}*as_agg_2cfb287d_dal_fixed_inco@edldev"
>  
> This will throw 
> {code:java}
> java.lang.IllegalArgumentException: Found reference to non-existant property 
> key in query at position...{code}
>  
> *Solution:*
> Backporting direct index query issue resolved in ATLAS-2371.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3816) Backport: Basic search issue with IndexSerializer

2020-05-28 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3816:
---

 Summary: Backport: Basic search issue with IndexSerializer
 Key: ATLAS-3816
 URL: https://issues.apache.org/jira/browse/ATLAS-3816
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.8.4
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


*Background:*

Direct index query which has pattern *"v."* throws exception, e.g.

 
v."Referenceable.qualifiedName":"atscale_agg_in*v.*as_agg_2cfb287d_dal_fixed_inco@edldev"

 

This will throw 
{code:java}
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...{code}
 

*Solution:*

Backporting direct index query issue resolved in ATLAS-2371.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3805) UI:- Basic Search: System Attribute 'typeName' in classification filter, should list classification Names in drop down

2020-05-27 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117601#comment-17117601
 ] 

Nikhil Bonte commented on ATLAS-3805:
-

+1 for the patch,

Thanks Keval.

> UI:-  Basic Search: System Attribute 'typeName' in classification filter, 
> should list classification Names in drop down
> ---
>
> Key: ATLAS-3805
> URL: https://issues.apache.org/jira/browse/ATLAS-3805
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3805.patch, Screen Shot 2020-05-21 at 12.55.53 
> PM.png
>
>
> !Screen Shot 2020-05-21 at 12.55.53 PM.png|width=602,height=342!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3798) Import Service: Percentage Display Problem with Large Datasets

2020-05-15 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108334#comment-17108334
 ] 

Nikhil Bonte commented on ATLAS-3798:
-

+1 for the patch.

Thanks [~amestry]

> Import Service: Percentage Display Problem with Large Datasets
> --
>
> Key: ATLAS-3798
> URL: https://issues.apache.org/jira/browse/ATLAS-3798
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Minor
> Fix For: trunk
>
> Attachments: ATLAS-3798-Import-percentage-display-fix.patch
>
>
> *Steps to Duplicate*
>  # Start migration import with 100M entities. Observe the log for 
> _last-imported_.
> _Expected output:_ Display should proceed in increasing order of %ages.
> _Actual output_: Display starts showing -ve %ages after 37%.
> *Root Cause*
> Existing implementation uses _integer_ for storing _currentIndex_.
> *Additional Information*
>  * Start position computation is not impacted.
>  * Resume is not impacted. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3799) Migration Import: EntityConsumer Adds GUIDs to Result that Will Not Get Cleared

2020-05-15 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108331#comment-17108331
 ] 

Nikhil Bonte commented on ATLAS-3799:
-

+1 for the patch.

Thanks [~amestry]

> Migration Import: EntityConsumer Adds GUIDs to Result that Will Not Get 
> Cleared
> ---
>
> Key: ATLAS-3799
> URL: https://issues.apache.org/jira/browse/ATLAS-3799
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3799-EntityConumer-only-adds-entity-GUIDs-that.patch
>
>
> *Background*
> Zip file-based migration import introduced _EntityConsumer_. This uses 
> _StatusReporter_ that is used to determine which which of the produced items 
> are processed. This, in turn, helps determine the right position to set in 
> case of resuming the import.
> The _EntityConumer_ class is responsible for adding the right GUIDs to the 
> result that helps _StatusReporter_ determine the right GUIDs. This is done 
> correctly except that it also adds additional GUIDs that were updated or 
> deleted. This causes _StatusReporter_._processedSet_ to be filled with GUIDs 
> that will never be cleared. This will grow over time and can potentially lead 
> to memory pressure.
> *Solution*
> Add only the GUIDs that were processed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3775) Migration Import Display: Incorrect Status Display

2020-05-05 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100023#comment-17100023
 ] 

Nikhil Bonte commented on ATLAS-3775:
-

+1 for the patch, thanks [~amestry] .

> Migration Import Display: Incorrect Status Display
> --
>
> Key: ATLAS-3775
> URL: https://issues.apache.org/jira/browse/ATLAS-3775
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3775-Migration-Import-Status-display-fix.patch
>
>
> *Background*
> Zip file based migration introduced _DataMigrationStatusService_. This class 
> is responsible for displaying the status.
> *Steps to Duplicate*
> Pre-requisites:
>  * Keep the REST http:///api/atlas/admin output open in a browser 
> window.
>  * Keep zip-file based migration import in progress.
>  * Ensure that you see the current status. Once migration progress, refresh 
> the REST API output to display the status. The status should be accurate.
>  * Refresh the REST API output after 5 mins. This will expire the status 
> cache and make a request to _DataMigrationService_ for status.
> _Expected results_
>  * Status returned by API should reflect the status logged.
>  
> _Actual results_
>  * There is a mismatch in the status.
>  * Additional information: The status is never updated for that session.
> Root cause:
> - _DataMigrationService_ maintains cached status object for each requestor. 
> This is not refreshed from the database status.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3702) Entity Creation: Improve Edge Creation

2020-04-05 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076041#comment-17076041
 ] 

Nikhil Bonte commented on ATLAS-3702:
-

+1 for the part 2 patch.

Thanks Ashutosh for the patch.

> Entity Creation: Improve Edge Creation
> --
>
> Key: ATLAS-3702
> URL: https://issues.apache.org/jira/browse/ATLAS-3702
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3702-Edge-creation-performance-improvements.-P.patch
>
>
> *Background*
> When creating entities with Atlas at scale, few inefficiencies were observed.
> They were:
>  * _GraphHelper.getAdjacentEdgesByLabel_: The existing approach results in 
> high memory usage. The method could use graph api that already exists.
>  * _AtlasRelationshipStoreV2.getRelationshipEdge_: This does full vertex 
> comparison. This is expensive.
>  * Edge label is not index. Indexing this will improve speed of edge by label 
> lookup.
> *Solution*
> Profile Atlas with (using metrics) to establish baseline.
> Use the fixes mentioned to the places above and profile.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (ATLAS-3701) Performance improvements in writing audit logs - Classification dissociation

2020-04-01 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte resolved ATLAS-3701.
-
Fix Version/s: 3.0.0
   2.1.0
   Resolution: Fixed

> Performance improvements in writing audit logs - Classification dissociation
> 
>
> Key: ATLAS-3701
> URL: https://issues.apache.org/jira/browse/ATLAS-3701
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
>
> Further to ATLAS-3568, 
> this will improve audit writing in case of classfication dissociation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3701) Performance improvements in writing audit logs - Classification dissociation

2020-03-28 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3701:

Component/s:  atlas-core

> Performance improvements in writing audit logs - Classification dissociation
> 
>
> Key: ATLAS-3701
> URL: https://issues.apache.org/jira/browse/ATLAS-3701
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Further to ATLAS-3568, 
> this will improve audit writing in case of classfication dissociation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3701) Performance improvements in writing audit logs - Classification dissociation

2020-03-28 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3701:
---

 Summary: Performance improvements in writing audit logs - 
Classification dissociation
 Key: ATLAS-3701
 URL: https://issues.apache.org/jira/browse/ATLAS-3701
 Project: Atlas
  Issue Type: Improvement
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


Further to ATLAS-3568, 

this will improve audit writing in case of classfication dissociation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3358) NPE when Atlas is trying to read from graphDB

2020-03-20 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063377#comment-17063377
 ] 

Nikhil Bonte commented on ATLAS-3358:
-

Committed to branch-0.8 - 4099b072c70f1480f7c4f1f494b2b400dea7df59

> NPE when Atlas is trying to read from graphDB
> -
>
> Key: ATLAS-3358
> URL: https://issues.apache.org/jira/browse/ATLAS-3358
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3358-NPE-when-Atlas-is-trying-to-read-from-gra.patch
>
>
> Reading vertex which has '__type.category' as null & Atlas fails to startup.
>  
> {noformat}
> 2019-07-31 17:47:46,717 INFO  - [main:] ~ Restoring type 
> CLASS.fs_path.fs_path (GraphBackedTypeStore:257)
> 2019-07-31 17:47:46,756 DEBUG - [main:] ~ Reading property __type.category 
> from vertex[id=496386248 type=null guid=null] (GraphHelper:433)
> 2019-07-31 17:47:46,757 DEBUG - [main:] ~ Reading property __type.name from 
> vertex[id=496386248 type=null guid=null] (GraphHelper:433)
> 2019-07-31 17:47:46,757 DEBUG - [main:] ~ Reading property __type.description 
> from vertex[id=496386248 type=null guid=null] (GraphHelper:433)
> 2019-07-31 17:47:46,757 INFO  - [main:] ~ Restoring type null.null.null 
> (GraphBackedTypeStore:257)
> 2019-07-31 17:47:46,758 ERROR - [main:] ~ graph rollback due to exception  
> (GraphTransactionInterceptor:71)
> java.lang.NullPointerException
> at 
> org.apache.atlas.repository.typestore.GraphBackedTypeStore.getTypesFromVertices(GraphBackedTypeStore.java:258)
> at 
> org.apache.atlas.repository.typestore.GraphBackedTypeStore.restore(GraphBackedTypeStore.java:233)
> at 
> org.apache.atlas.repository.typestore.GraphBackedTypeStore$$FastClassBySpringCGLIB$$89e4b665.invoke()
> at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at 
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
> at 
> org.apache.atlas.repository.typestore.GraphBackedTypeStore$$EnhancerBySpringCGLIB$$95613fc.restore()
> at 
> org.apache.atlas.services.DefaultMetadataService.restoreTypeSystem(DefaultMetadataService.java:143)
> at 
> org.apache.atlas.services.DefaultMetadataService.(DefaultMetadataService.java:132)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
> at 
> org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
> at 
> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
> at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
> at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
> at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
> at 
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> at 
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at 
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at 
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at 
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
> at 
> 

[jira] [Commented] (ATLAS-3642) PC Framework: Refactoring: Modify WorkItemManager.getResults to return Queue

2020-03-02 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049203#comment-17049203
 ] 

Nikhil Bonte commented on ATLAS-3642:
-

+1 for the patch.

> PC Framework: Refactoring: Modify WorkItemManager.getResults to return Queue
> 
>
> Key: ATLAS-3642
> URL: https://issues.apache.org/jira/browse/ATLAS-3642
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3642-PC-fx-WorkItemManager-getResults-Modifica.patch
>
>
> Minor refactoring: Modify WorkItemManager.getResults.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3630) UI: Update node version to 12.x

2020-02-21 Thread Nikhil Bonte (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041725#comment-17041725
 ] 

Nikhil Bonte commented on ATLAS-3630:
-

+1 for the patch, Thanks [~kevalbhatt] for the patch.

> UI: Update node version to 12.x
> ---
>
> Key: ATLAS-3630
> URL: https://issues.apache.org/jira/browse/ATLAS-3630
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3630.1.patch, ATLAS-3630.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3604) Data Migration: Redirect root URI to migration status page

2020-02-16 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3604:

Fix Version/s: trunk

> Data Migration: Redirect root URI to migration status page
> --
>
> Key: ATLAS-3604
> URL: https://issues.apache.org/jira/browse/ATLAS-3604
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3604-Data-Migration-Metrics-Page-Display-with-UI.patch
>
>
>  
> Data migration service added under [ATLAS-2461|http://example.com/] do not 
> provide UI to show migration status.
> This feature includes server side changes to allow a html page to be shown in 
> migration mode.
> In migration mode request to root URI will be redirected to 
> migration-status.html.
> This page will render the response from API "api/atlas/admin/metrics".
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-16 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Fix Version/s: trunk

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: trunk
>
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-02-16 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3603:

Fix Version/s: trunk

> Data Migration: Auto-start zip Migration Import
> ---
>
> Key: ATLAS-3603
> URL: https://issues.apache.org/jira/browse/ATLAS-3603
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3595-ATLAS-3603-Auto-start-migration-import.patch
>
>
>  
> This feature will integrate with ATLAS-3595.
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> Approach:
> ZipMigrationImporter will first check if zip is available, accessible & 
> completely written.
>  * If yes, migration will start ingesting zip.
>  * if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
> It uses the file size comparison approach. Once modify event is received, 
> file size is compared in an interval. If sizes match, it means file is 
> available and not being written.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-02-16 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3603:

Attachment: ATLAS-3595-ATLAS-3603-Auto-start-migration-import.patch

> Data Migration: Auto-start zip Migration Import
> ---
>
> Key: ATLAS-3603
> URL: https://issues.apache.org/jira/browse/ATLAS-3603
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: ATLAS-3595-ATLAS-3603-Auto-start-migration-import.patch
>
>
>  
> This feature will integrate with ATLAS-3595.
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> Approach:
> ZipMigrationImporter will first check if zip is available, accessible & 
> completely written.
>  * If yes, migration will start ingesting zip.
>  * if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
> It uses the file size comparison approach. Once modify event is received, 
> file size is compared in an interval. If sizes match, it means file is 
> available and not being written.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-16 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Attachment: (was: 
ATLAS-3603-ATLAS-3595-Data-Migration-Auto-start-zip-migration.patch)

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-16 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Attachment: (was: zip-direct-2.zip)

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3603-ATLAS-3595-Data-Migration-Auto-start-zip-migration.patch
>
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Attachment: zip-direct-2.zip

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3603-ATLAS-3595-Data-Migration-Auto-start-zip-migration.patch, 
> zip-direct-2.zip
>
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Attachment: (was: 
ATLAS-3595-Migration-import-support-zip-file-for-dat-v6.patch)

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3603-ATLAS-3595-Data-Migration-Auto-start-zip-migration.patch
>
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Attachment: 
ATLAS-3603-ATLAS-3595-Data-Migration-Auto-start-zip-migration.patch

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3603-ATLAS-3595-Data-Migration-Auto-start-zip-migration.patch
>
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3603:

Attachment: (was: 
ATLAS-3603-Data-Migration-Auto-start-zip-migration-import.patch)

> Data Migration: Auto-start zip Migration Import
> ---
>
> Key: ATLAS-3603
> URL: https://issues.apache.org/jira/browse/ATLAS-3603
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
>  
> This feature will integrate with ATLAS-3595.
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> Approach:
> ZipMigrationImporter will first check if zip is available, accessible & 
> completely written.
>  * If yes, migration will start ingesting zip.
>  * if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
> It uses the file size comparison approach. Once modify event is received, 
> file size is compared in an interval. If sizes match, it means file is 
> available and not being written.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Property value for migration should contain full path of file, below is the 
example. 
 * Data migration service will first check migration property if value refers 
to a zip file.
 * If true, it will call ZipMigrationImporter to import data in zip file.
 * ZipMigrationImporter internally uses existing ImportService to ingest zip.
 * Audit service & Notification service will also be started as a part of zip 
import.

E.g. properties to enable zip data migration:
 * atlas.migration.data.filename=/path/fileName.zip
 atlas.migration.mode.workers=5
 atlas.migration.mode.batch.size=20

  was:
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Property value for migration should contain full path of file, below is the 
example. 
 * Data migration service will first check migration property if value refers 
to a zip file.
 * If true, it will call ZipMigrator to import data in zip file.
 * ZipMigrator internally uses existing ImportService to ingest zip.
 * Audit service & Notification service will also be started as a part of zip 
import.

E.g. properties to enable zip data migration:
 * atlas.migration.data.filename=/path/fileName.zip
 atlas.migration.mode.workers=5
 atlas.migration.mode.batch.size=20


> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3595-Migration-import-support-zip-file-for-dat-v6.patch
>
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrationImporter to import data in zip file.
>  * ZipMigrationImporter internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3603:

Description: 
 

This feature will integrate with ATLAS-3595.

Migration import should start when the data (zip) to be ingested is available 
at the pre-configured location.

Approach:

ZipMigrationImporter will first check if zip is available, accessible & 
completely written.
 * If yes, migration will start ingesting zip.
 * if not, will poll the location periodically. when existence is detected, 
import will proceed as usual.

It uses the file size comparison approach. Once modify event is received, file 
size is compared in an interval. If sizes match, it means file is available and 
not being written.

 

  was:
 

This feature will integrate with ATLAS-3595.

Migration import should start when the data (zip) to be ingested is available 
at the pre-configured location.

Approach:

Data migration service will first check if zip is available & accessible.
 * If yes, migration will start ingesting zip.
 * if not, will poll the location periodically. when existence is detected, 
import will proceed as usual.

 


> Data Migration: Auto-start zip Migration Import
> ---
>
> Key: ATLAS-3603
> URL: https://issues.apache.org/jira/browse/ATLAS-3603
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3603-Data-Migration-Auto-start-zip-migration-import.patch
>
>
>  
> This feature will integrate with ATLAS-3595.
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> Approach:
> ZipMigrationImporter will first check if zip is available, accessible & 
> completely written.
>  * If yes, migration will start ingesting zip.
>  * if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
> It uses the file size comparison approach. Once modify event is received, 
> file size is compared in an interval. If sizes match, it means file is 
> available and not being written.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3604) Data Migration: Redirect root URI to migration status page

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3604:

Attachment: ATLAS-3604-Data-Migration-Metrics-Page-Display-with-UI.patch

> Data Migration: Redirect root URI to migration status page
> --
>
> Key: ATLAS-3604
> URL: https://issues.apache.org/jira/browse/ATLAS-3604
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3604-Data-Migration-Metrics-Page-Display-with-UI.patch
>
>
>  
> Data migration service added under [ATLAS-2461|http://example.com/] do not 
> provide UI to show migration status.
> This feature includes server side changes to allow a html page to be shown in 
> migration mode.
> In migration mode request to root URI will be redirected to 
> migration-status.html.
> This page will render the response from API "api/atlas/admin/metrics".
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3604) Data Migration: Redirect root URI to migration status page

2020-02-13 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3604:

Attachment: (was: 
ATLAS-3604-Data-Migration-Redirect-root-URI-to-status-page.patch)

> Data Migration: Redirect root URI to migration status page
> --
>
> Key: ATLAS-3604
> URL: https://issues.apache.org/jira/browse/ATLAS-3604
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3604-Data-Migration-Metrics-Page-Display-with-UI.patch
>
>
>  
> Data migration service added under [ATLAS-2461|http://example.com/] do not 
> provide UI to show migration status.
> This feature includes server side changes to allow a html page to be shown in 
> migration mode.
> In migration mode request to root URI will be redirected to 
> migration-status.html.
> This page will render the response from API "api/atlas/admin/metrics".
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3604) Data Migration: Redirect root URI to migration status page

2020-01-31 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3604:
---

 Summary: Data Migration: Redirect root URI to migration status page
 Key: ATLAS-3604
 URL: https://issues.apache.org/jira/browse/ATLAS-3604
 Project: Atlas
  Issue Type: New Feature
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


 

Data migration service added under [ATLAS-2461|http://example.com/] do not 
provide UI to show migration status.

This feature includes server side changes to allow a html page to be shown in 
migration mode.

In migration mode request to root URI will be redirected to 
migration-status.html.

This page will render the response from API "api/atlas/admin/metrics".

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3603:

Attachment: ATLAS-3603-Data-Migration-Auto-start-zip-migration-import.patch

> Data Migration: Auto-start zip Migration Import
> ---
>
> Key: ATLAS-3603
> URL: https://issues.apache.org/jira/browse/ATLAS-3603
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Attachments: 
> ATLAS-3603-Data-Migration-Auto-start-zip-migration-import.patch
>
>
>  
> This feature will integrate with ATLAS-3595.
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> Approach:
> Data migration service will first check if zip is available & accessible.
>  * If yes, migration will start ingesting zip.
>  * if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3603:

Attachment: (was: 
ATLAS-3603-Data-Migration-Auto-start-zip-migration-import-v0.patch)

> Data Migration: Auto-start zip Migration Import
> ---
>
> Key: ATLAS-3603
> URL: https://issues.apache.org/jira/browse/ATLAS-3603
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
>  
> This feature will integrate with ATLAS-3595.
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> Approach:
> Data migration service will first check if zip is available & accessible.
>  * If yes, migration will start ingesting zip.
>  * if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3603) Data Migration: Auto-start zip Migration Import

2020-01-31 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3603:
---

 Summary: Data Migration: Auto-start zip Migration Import
 Key: ATLAS-3603
 URL: https://issues.apache.org/jira/browse/ATLAS-3603
 Project: Atlas
  Issue Type: New Feature
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte


 

This feature will integrate with ATLAS-3595.

Migration import should start when the data (zip) to be ingested is available 
at the pre-configured location.

Approach:

Data migration service will first check if zip is available & accessible.
 * If yes, migration will start ingesting zip.
 * if not, will poll the location periodically. when existence is detected, 
import will proceed as usual.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Property value for migration should contain full path of file, below is the 
example. 
 * Data migration service will first check migration property if value refers 
to a zip file.
 * If true, it will call ZipMigrator to import data in zip file.
 * ZipMigrator internally uses existing ImportService to ingest zip.
 * Audit service & Notification service will also be started as a part of zip 
import.

E.g. properties to enable zip data migration:
 * atlas.migration.data.filename=/path/fileName.zip
 atlas.migration.mode.workers=5
 atlas.migration.mode.batch.size=20

  was:
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Property value for migration should contain full path of file, below is the 
example. 
 * Data migration service will first check in input directory for a zip file.
 * If zip file found, it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.
 * Audit service & Notification service will also be started as a part of zip 
import.

E.g. properties to enable zip data migration:
 * atlas.migration.data.filename=/path/fileName.zip
atlas.migration.mode.workers=5
atlas.migration.mode.batch.size=20


> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check migration property if value refers 
> to a zip file.
>  * If true, it will call ZipMigrator to import data in zip file.
>  * ZipMigrator internally uses existing ImportService to ingest zip.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
>  atlas.migration.mode.workers=5
>  atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Property value for migration should contain full path of file, below is the 
example. 
 * Data migration service will first check in input directory for a zip file.
 * If zip file found, it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.
 * Audit service & Notification service will also be started as a part of zip 
import.

E.g. properties to enable zip data migration:
 * atlas.migration.data.filename=/path/fileName.zip
atlas.migration.mode.workers=5
atlas.migration.mode.batch.size=20

  was:
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for a zip file.
 * If zip file found, it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.
 * Audit service & Notification service will also be started as a part of zip 
import.


> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Property value for migration should contain full path of file, below is 
> the example. 
>  * Data migration service will first check in input directory for a zip file.
>  * If zip file found, it will call ZipMigrator to import data in file.
>  * ZipMigrator internally uses existing import functionality in ImportService.
>  * Audit service & Notification service will also be started as a part of zip 
> import.
> E.g. properties to enable zip data migration:
>  * atlas.migration.data.filename=/path/fileName.zip
> atlas.migration.mode.workers=5
> atlas.migration.mode.batch.size=20



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for a zip file.
 * If zip file found, it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.
 * Audit service & Notification service will also be started as a part of zip 
import.

  was:
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for a zip file.
 * If zip file found, it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.


> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Data migration service will first check in input directory for a zip file.
>  * If zip file found, it will call ZipMigrator to import data in file.
>  * ZipMigrator internally uses existing import functionality in ImportService.
>  * Audit service & Notification service will also be started as a part of zip 
> import.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for a zip file.
 * If zip file found, it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.

  was:
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for any zip file.
 * If zip file found it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.


> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Data migration service will first check in input directory for a zip file.
>  * If zip file found, it will call ZipMigrator to import data in file.
>  * ZipMigrator internally uses existing import functionality in ImportService.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added under [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for any zip file.
 * If zip file found it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.

  was:
Data migration service added in [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for any zip file.
 * If zip file found it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.


> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added under [ATLAS-2461|http://example.com/] works 
> with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Data migration service will first check in input directory for any zip 
> file.
>  * If zip file found it will call ZipMigrator to import data in file.
>  * ZipMigrator internally uses existing import functionality in ImportService.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Data migration service added in [ATLAS-2461|http://example.com/] works with 
json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:
 * Data migration service will first check in input directory for any zip file.
 * If zip file found it will call ZipMigrator to import data in file.
 * ZipMigrator internally uses existing import functionality in ImportService.

  was:
Currently Data migration service works with json files as input.
This improvement will enable Data Migration service to import data using a zip 
file.




> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Data migration service added in [ATLAS-2461|http://example.com/] works with 
> json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.
> Approach:
>  * Data migration service will first check in input directory for any zip 
> file.
>  * If zip file found it will call ZipMigrator to import data in file.
>  * ZipMigrator internally uses existing import functionality in ImportService.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: Support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Summary: Data Migration: Support zip file for data migration  (was: Data 
Migration: support zip file for data migration)

> Data Migration: Support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Currently Data migration service works with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Data Migration: support zip file for data migration

2020-01-31 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Summary: Data Migration: support zip file for data migration  (was: 
Migration import: support zip file for data migration)

> Data Migration: support zip file for data migration
> ---
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Currently Data migration service works with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Migration import: support zip file for data migration

2020-01-24 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Currently Data migration service works with json files as input.
This improvement will enable Data Migration service to import data using a zip 
file.



  was:
Currently Data migration service works with json files as input.
This feature will enable Data Migration service to import data from zip file


> Migration import: support zip file for data migration
> -
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Currently Data migration service works with json files as input.
> This improvement will enable Data Migration service to import data using a 
> zip file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Migration import: support zip file for data migration

2020-01-24 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Description: 
Currently Data migration service works with json files as input.
This feature will enable Data Migration service to import data from zip file

> Migration import: support zip file for data migration
> -
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Currently Data migration service works with json files as input.
> This feature will enable Data Migration service to import data from zip file



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3595) Migration import: support zip file for data migration

2020-01-24 Thread Nikhil Bonte (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikhil Bonte updated ATLAS-3595:

Issue Type: Improvement  (was: New Feature)

> Migration import: support zip file for data migration
> -
>
> Key: ATLAS-3595
> URL: https://issues.apache.org/jira/browse/ATLAS-3595
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> Currently Data migration service works with json files as input.
> This feature will enable Data Migration service to import data from zip file



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3595) Migration import: support zip file for data migration

2020-01-24 Thread Nikhil Bonte (Jira)
Nikhil Bonte created ATLAS-3595:
---

 Summary: Migration import: support zip file for data migration
 Key: ATLAS-3595
 URL: https://issues.apache.org/jira/browse/ATLAS-3595
 Project: Atlas
  Issue Type: New Feature
Reporter: Nikhil Bonte
Assignee: Nikhil Bonte






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


  1   2   3   >