[jira] [Commented] (ATLAS-2974) Docker build: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin...

[Chenglong Hu (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108814#comment-17108814
 ] 

Chenglong Hu commented on ATLAS-2974:
-

Hi [~jonesn], I'm curious about what you have done to build the docker image ok 
off master since I met the same error as "Cannot connect to cluster at 
localhost:2181: cluster not found/not ready" both from master and branch-2.0.

Hope for your response!

> Docker build: Failed to execute goal 
> org.apache.maven.plugins:maven-enforcer-plugin...
> --
>
> Key: ATLAS-2974
> URL: https://issues.apache.org/jira/browse/ATLAS-2974
> Project: Atlas
>  Issue Type: Bug
>Reporter: Dmitry Goryunov
>Assignee: Xinran Tinney
>Priority: Major
> Attachments: tmp
>
>
> Version: commit 945de013f5466ffcf71d16c6ec1772fdee83e2f6 (Also reproducable 
> with 1.1.0-rc2)
>  
> Runnign command:
> {code:java}
> docker build -t atlas_docker .
> {code}
> Breaks at some point with:
> {code:java}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce 
> (enforce-versions) on project apache-atlas: Some Enforcer rules have failed. 
> Look above for specific messages explaining why the rule failed. -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :apache-atlas
> The command '/bin/sh -c mvn clean install -DskipTests 
> -Pdist,embedded-hbase-solr -f ./atlas/pom.xml' returned a non-zero code: 1
> {code}
> The complete logs are attached.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72512: Edge Creation: Refactoring

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72512/#review220790
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 15, 2020, 5:19 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72512/
> ---
> 
> (Updated May 15, 2020, 5:19 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-3797
> https://issues.apache.org/jira/browse/ATLAS-3797
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Modified: _GraphHelper.getOrCreateEdge_ to use 
> _AtlasJanusGraph.getEdgeBetweenVertices_.
> Modified: _AtlasJanusGraph.getEdgeBetweenVertices_ modified fetch.
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  35274c9d1 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> b1368e5e9 
> 
> 
> Diff: https://reviews.apache.org/r/72512/diff/3/
> 
> 
> Testing
> ---
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1896/
> 
> **Volume test**
> Case edge creation involves existing entity where large number of existing 
> edges (200K), this change shows significant improvement.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Apache Atlas 2.1 release

[Robert Yokota]

Hi Madhan,

Any update on Apache Atlas 2.1?   It looks like 2.0 was released over 1
year ago (5/14/19).

Thanks,
Robert

On Fri, Mar 27, 2020 at 10:18 AM Bolke de Bruin  wrote:

> That would be great! A faster release cycle would be appreciated as well.
>
> When is the vote due?
>
> Thanks
> Bolke
>
> Sent from my iPhone
>
> > On 19 Feb 2020, at 02:10, Madhan Neethiraj  wrote:
> >
> > Atlas community,
> >
> >
> >
> > Over past months the dev community has been busy in enhancing Apache
> Atlas with new features, improvements and fixes. Here are few
> features/enhancements since last major release, Apache Atlas 2.0:
> >
> > - added quick-search feature, to provide a simpler search experience
> with type-ahead suggestions
> >
> > - introduced Namespaces feature, which allows grouping of attributes
> to be applied to multiple entity-types
> >
> > - introduced labels on entity instances, and search for entities
> using the label
> >
> > - enhancement to support entity instance specific custom attributes
> >
> > - enhanced search to find entities by more than one classification
> >
> > - introduced shell/incomplete entities to handle notifications
> referencing entities that don’t (yet) exist in Atlas
> >
> > - added REST APIs to purge deleted entities
> >
> > - performance improvements in lineage retrieval and tag-propagation
> >
> > - updated Atlas server to process notifications from multiple Kafka
> topics
> >
> > - updated Hive hook to track process executions, via
> hive_process_execution entities
> >
> > - updated Hive hook to capture DDL operations, via hive_db_ddl and
> hive_table_ddl entities
> >
> > - added models for Spark; introduced new models for AWS S3
> >
> > - updated versions of dependent libraries/components: JanusGraph,
> Jackson parser, Spring Framework,
> >
> > - updated authorization model to cover new features/APIs, like
> add/remove labels, purge entities, update namespace attributes
> >
> >
> >
> > With significant improvements in place, it is time for the next
> maintenance release of Apache Atlas!
> >
> >
> >
> > I propose to release Apache Atlas 2.1 by early next month. Please review
> and send your comments.
> >
> >
> >
> > Regards,
> >
> > Madhan
> >
>

[jira] [Created] (ATLAS-3801) NotificationHookConsumer: Introduce Retry Logic for PermanentLockingException

[Ashutosh Mestry (Jira)]

Ashutosh Mestry created ATLAS-3801:
--

 Summary: NotificationHookConsumer: Introduce Retry Logic for 
PermanentLockingException
 Key: ATLAS-3801
 URL: https://issues.apache.org/jira/browse/ATLAS-3801
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Background*

_NotificationHookConsumer_ when performing graph commit can potentially 
encounter _JanusGraph's PermanentLockingException_.

JanusGraph [documentation|https://docs.janusgraph.org/basics/transactions/] 
details the cases in which this exception occurs. The way past this exception 
is rollback, pause and replay the data being committed.

*Solution*

Introduce the rollback, pause and retry mechanism in _NotificationHookConsumer_.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3799) Migration Import: EntityConsumer Adds GUIDs to Result that Will Not Get Cleared

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108573#comment-17108573
 ] 

ASF subversion and git services commented on ATLAS-3799:


Commit 3de30f55d808df6245f60fe01fed4b6cf894a31d in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=3de30f5 ]

ATLAS-3799: EntityConumer only adds entity GUIDs that are added when they were 
produced.


> Migration Import: EntityConsumer Adds GUIDs to Result that Will Not Get 
> Cleared
> ---
>
> Key: ATLAS-3799
> URL: https://issues.apache.org/jira/browse/ATLAS-3799
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3799-EntityConumer-only-adds-entity-GUIDs-that.patch
>
>
> *Background*
> Zip file-based migration import introduced _EntityConsumer_. This uses 
> _StatusReporter_ that is used to determine which which of the produced items 
> are processed. This, in turn, helps determine the right position to set in 
> case of resuming the import.
> The _EntityConumer_ class is responsible for adding the right GUIDs to the 
> result that helps _StatusReporter_ determine the right GUIDs. This is done 
> correctly except that it also adds additional GUIDs that were updated or 
> deleted. This causes _StatusReporter_._processedSet_ to be filled with GUIDs 
> that will never be cleared. This will grow over time and can potentially lead 
> to memory pressure.
> *Solution*
> Add only the GUIDs that were processed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3796) Migration Import Display: StatusReporter Does not Remove Processed Items

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108571#comment-17108571
 ] 

ASF subversion and git services commented on ATLAS-3796:


Commit 935d0ab6465775e16db62896375d96f6a567b884 in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=935d0ab ]

ATLAS-3796: Status Reporter display fix.


> Migration Import Display: StatusReporter Does not Remove Processed Items
> 
>
> Key: ATLAS-3796
> URL: https://issues.apache.org/jira/browse/ATLAS-3796
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3796-Status-Reporter-processed-set-cleanup.patch
>
>
> *Background*
> _Migration Import_ implementation introduced _StatusReporter_. The class 
> maintains GUIDs of entities that are to be processed and entities that are 
> processed. When the entity is marked as processed, it needs to be removed 
> from the processed set.
> *Steps to Duplicate*
> _Pre-requisites_
> Large migration import dataset (over 50 M entities).
> _Steps to Duplicate_
>  * Start importing the dataset.
>  * After over 60%, the Atlas server will crash with out of memory exception.
> _Expected results_: Migration Import should continue without problems.
> _Actual results_: Atlas server crashes with out of memory exception, hprof 
> dump indicates large number of items allocated by 
> _StatusReporter.processedSet_.
> _Additional information_: This bug does not affect status reporting.
> _Work around_: Resume migration import. It will continue without problems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3798) Import Service: Percentage Display Problem with Large Datasets

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108572#comment-17108572
 ] 

ASF subversion and git services commented on ATLAS-3798:


Commit 214f2a760a1a4c8c3a83b5afaa42fdc36055ff98 in atlas's branch 
refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=214f2a7 ]

ATLAS-3798: Import percentage display fix.


> Import Service: Percentage Display Problem with Large Datasets
> --
>
> Key: ATLAS-3798
> URL: https://issues.apache.org/jira/browse/ATLAS-3798
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Minor
> Fix For: trunk
>
> Attachments: ATLAS-3798-Import-percentage-display-fix.patch
>
>
> *Steps to Duplicate*
>  # Start migration import with 100M entities. Observe the log for 
> _last-imported_.
> _Expected output:_ Display should proceed in increasing order of %ages.
> _Actual output_: Display starts showing -ve %ages after 37%.
> *Root Cause*
> Existing implementation uses _integer_ for storing _currentIndex_.
> *Additional Information*
>  * Start position computation is not impacted.
>  * Resume is not impacted. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3797) Entity Creation: Refactoring: Improve Edge Creation

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3797:
---
Attachment: (was: ATLAS-3797-Refactoring-Improve-Edge-Creation.patch)

> Entity Creation: Refactoring: Improve Edge Creation
> ---
>
> Key: ATLAS-3797
> URL: https://issues.apache.org/jira/browse/ATLAS-3797
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3797-Refactoring-Improve-Edge-Creation.patch
>
>
> *Background*
> [ATLAS-3762|https://issues.apache.org/jira/browse/ATLAS-3762] introduced edge 
> creation improvement by improving on edge existence check.
> The same change could be applied to another area within the implementation.
> *Investigation*
> This was discovered as part of volume testing done for high-volume import. 
> This included importing over 100M entities. Various parts of the entity 
> creation process had probes added to check for performance.
> During this investigation the edge creation was doing the same existence 
> check that was addressed in earlier fixes. Re-using the same implementation 
> showed marked improvement in entity creation.
> These were the parameters used for during import.
> Node configuration: 128 GB RAM, 40 cores.
> {code:java}
> atlas.migration.mode.batch.size=1500
> atlas.migration.mode.workers=40{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3797) Entity Creation: Refactoring: Improve Edge Creation

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-3797:
---
Attachment: ATLAS-3797-Refactoring-Improve-Edge-Creation.patch

> Entity Creation: Refactoring: Improve Edge Creation
> ---
>
> Key: ATLAS-3797
> URL: https://issues.apache.org/jira/browse/ATLAS-3797
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-3797-Refactoring-Improve-Edge-Creation.patch
>
>
> *Background*
> [ATLAS-3762|https://issues.apache.org/jira/browse/ATLAS-3762] introduced edge 
> creation improvement by improving on edge existence check.
> The same change could be applied to another area within the implementation.
> *Investigation*
> This was discovered as part of volume testing done for high-volume import. 
> This included importing over 100M entities. Various parts of the entity 
> creation process had probes added to check for performance.
> During this investigation the edge creation was doing the same existence 
> check that was addressed in earlier fixes. Re-using the same implementation 
> showed marked improvement in entity creation.
> These were the parameters used for during import.
> Node configuration: 128 GB RAM, 40 cores.
> {code:java}
> atlas.migration.mode.batch.size=1500
> atlas.migration.mode.workers=40{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [atlas] AmeerAssi commented on pull request #98: [ATLAS-3800] AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

AmeerAssi commented on pull request #98:
URL: https://github.com/apache/atlas/pull/98#issuecomment-629390103


   Thank you.
   Closing the PR



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [atlas] AmeerAssi closed pull request #98: [ATLAS-3800] AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

AmeerAssi closed pull request #98:
URL: https://github.com/apache/atlas/pull/98


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Review Request 72474: ATLAS-3783 : DSL query search should return results for both the relationship edge directions

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72474/#review220788
---


Ship it!




Is this change needed at all? What is implemented is correct, since 
NullPointerException is caught and added to the errorList.

- Ashutosh Mestry


On May 15, 2020, 12:14 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72474/
> ---
> 
> (Updated May 15, 2020, 12:14 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3783
> https://issues.apache.org/jira/browse/ATLAS-3783
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Unable to get results for Dsl query eg;hive_column where 
> table.name="employee".
> Due to regression only one edge direction was giving results.
> This patch gives a fix for results for both relationship edge direction 
> queries.
> Documentation added for this fiv in Advanced DSl search.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> 129c20354 
> 
> 
> Diff: https://reviews.apache.org/r/72474/diff/14/
> 
> 
> Testing
> ---
> 
> TestCases added.
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 72512: Edge Creation: Refactoring

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72512/
---

(Updated May 15, 2020, 5:19 p.m.)


Review request for atlas.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-3797
https://issues.apache.org/jira/browse/ATLAS-3797


Repository: atlas


Description
---

**Approach**
Modified: _GraphHelper.getOrCreateEdge_ to use 
_AtlasJanusGraph.getEdgeBetweenVertices_.
Modified: _AtlasJanusGraph.getEdgeBetweenVertices_ modified fetch.


Diffs (updated)
-

  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 35274c9d1 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
b1368e5e9 


Diff: https://reviews.apache.org/r/72512/diff/3/

Changes: https://reviews.apache.org/r/72512/diff/2-3/


Testing
---

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1896/

**Volume test**
Case edge creation involves existing entity where large number of existing 
edges (200K), this change shows significant improvement.


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-3788) BasicSearch: Classification with System attribute(indexed) filters has pagination issue

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3788:
--
Labels: BasicSearch  (was: )

> BasicSearch: Classification with System attribute(indexed) filters has 
> pagination issue
> ---
>
> Key: ATLAS-3788
> URL: https://issues.apache.org/jira/browse/ATLAS-3788
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>  Labels: BasicSearch
> Fix For: 2.1.0
>
>
> When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
> currentTimeStamp , it results to less or zero number of entities than actual.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3788) BasicSearch: Classification with System attribute(indexed) filters has pagination issue

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3788:
--
Component/s:  atlas-core

> BasicSearch: Classification with System attribute(indexed) filters has 
> pagination issue
> ---
>
> Key: ATLAS-3788
> URL: https://issues.apache.org/jira/browse/ATLAS-3788
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Fix For: 2.1.0
>
>
> When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
> currentTimeStamp , it results to less or zero number of entities than actual.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3788) BasicSearch: Classification with System attribute(indexed) filters has pagination issue

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3788:
--
Affects Version/s: 2.0.0

> BasicSearch: Classification with System attribute(indexed) filters has 
> pagination issue
> ---
>
> Key: ATLAS-3788
> URL: https://issues.apache.org/jira/browse/ATLAS-3788
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>
> When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
> currentTimeStamp , it results to less or zero number of entities than actual.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3788) BasicSearch: Classification with System attribute(indexed) filters has pagination issue

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3788:
--
Fix Version/s: 2.1.0

> BasicSearch: Classification with System attribute(indexed) filters has 
> pagination issue
> ---
>
> Key: ATLAS-3788
> URL: https://issues.apache.org/jira/browse/ATLAS-3788
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Fix For: 2.1.0
>
>
> When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
> currentTimeStamp , it results to less or zero number of entities than actual.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [atlas] sarathsubramanian commented on pull request #98: [ATLAS-3800] AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

sarathsubramanian commented on pull request #98:
URL: https://github.com/apache/atlas/pull/98#issuecomment-629378481


   merged the changes. Please resolve this PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (ATLAS-3800) AWS scheme is missing the aws account id that contains the S3 bucket

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108473#comment-17108473
 ] 

ASF subversion and git services commented on ATLAS-3800:


Commit cb449b61a97c46b8c3552f9e75ec7e0415d7813c in atlas's branch 
refs/heads/branch-2.0 from Ameer Assi
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=cb449b6 ]

ATLAS-3800: Add 'accountId' attribute for AWS S3 v2 bucket model 
(aws_s3_v2_bucket)

Signed-off-by: Sarath Subramanian 
(cherry picked from commit d8cb94ac65dde5e1585ffe6d547450f647b65d76)


> AWS scheme is missing the aws account id that contains the S3 bucket
> 
>
> Key: ATLAS-3800
> URL: https://issues.apache.org/jira/browse/ATLAS-3800
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Ameer Assi
>Priority: Minor
> Fix For: 2.1.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We are using Atlas models in our product.
> the product gives scan capabilities for customer's S3 in AWS.
> when scan result shown based on the model, it is missing for which account 
> this bucket belongs. so it will be easy to access the account and review the 
> scanned information by the customer.
>  
> https://github.com/apache/atlas/pull/98



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3800) AWS scheme is missing the aws account id that contains the S3 bucket

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108472#comment-17108472
 ] 

ASF subversion and git services commented on ATLAS-3800:


Commit d8cb94ac65dde5e1585ffe6d547450f647b65d76 in atlas's branch 
refs/heads/master from Ameer Assi
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=d8cb94a ]

ATLAS-3800: Add 'accountId' attribute for AWS S3 v2 bucket model 
(aws_s3_v2_bucket)

Signed-off-by: Sarath Subramanian 


> AWS scheme is missing the aws account id that contains the S3 bucket
> 
>
> Key: ATLAS-3800
> URL: https://issues.apache.org/jira/browse/ATLAS-3800
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Ameer Assi
>Priority: Minor
> Fix For: 2.1.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We are using Atlas models in our product.
> the product gives scan capabilities for customer's S3 in AWS.
> when scan result shown based on the model, it is missing for which account 
> this bucket belongs. so it will be easy to access the account and review the 
> scanned information by the customer.
>  
> https://github.com/apache/atlas/pull/98



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3800) AWS scheme is missing the aws account id that contains the S3 bucket

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3800:
--
Fix Version/s: 2.1.0

> AWS scheme is missing the aws account id that contains the S3 bucket
> 
>
> Key: ATLAS-3800
> URL: https://issues.apache.org/jira/browse/ATLAS-3800
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Ameer Assi
>Priority: Minor
> Fix For: 2.1.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We are using Atlas models in our product.
> the product gives scan capabilities for customer's S3 in AWS.
> when scan result shown based on the model, it is missing for which account 
> this bucket belongs. so it will be easy to access the account and review the 
> scanned information by the customer.
>  
> https://github.com/apache/atlas/pull/98



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3800) AWS scheme is missing the aws account id that contains the S3 bucket

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3800:
--
Affects Version/s: 2.0.0

> AWS scheme is missing the aws account id that contains the S3 bucket
> 
>
> Key: ATLAS-3800
> URL: https://issues.apache.org/jira/browse/ATLAS-3800
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Ameer Assi
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We are using Atlas models in our product.
> the product gives scan capabilities for customer's S3 in AWS.
> when scan result shown based on the model, it is missing for which account 
> this bucket belongs. so it will be easy to access the account and review the 
> scanned information by the customer.
>  
> https://github.com/apache/atlas/pull/98



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3800) AWS scheme is missing the aws account id that contains the S3 bucket

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3800:
--
Component/s:  atlas-core

> AWS scheme is missing the aws account id that contains the S3 bucket
> 
>
> Key: ATLAS-3800
> URL: https://issues.apache.org/jira/browse/ATLAS-3800
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Reporter: Ameer Assi
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We are using Atlas models in our product.
> the product gives scan capabilities for customer's S3 in AWS.
> when scan result shown based on the model, it is missing for which account 
> this bucket belongs. so it will be easy to access the account and review the 
> scanned information by the customer.
>  
> https://github.com/apache/atlas/pull/98



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72489: ATLAS-3788 : BasicSearch: Classification with System attribute(indexed) filters has pagination issue

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72489/#review220787
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 14, 2020, 8:45 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72489/
> ---
> 
> (Updated May 14, 2020, 8:45 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3788
> https://issues.apache.org/jira/browse/ATLAS-3788
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When selecting ALL_CLASSIFICATION_TYPES, with system attribute Created Time < 
> currentTimeStamp , it results to less or zero number of entities than actual.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  74c088c0e 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 958781e41 
>   
> repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72489/diff/3/
> 
> 
> Testing
> ---
> 
> Manually Tested.
> Added test cases.
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1894
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72512: Edge Creation: Refactoring

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72512/#review220785
---




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
Lines 186 (patched)


-  consider moving private methods after all public/protected methods
- it seems isPresent() can be eliminated, since !isPresent() is already 
done. Please review:
  if (gremlinEdge != null &&
  (!gremlinEdge.property(STATE_PROPERTY_KEY).isPresent() ||

AtlasEntity.Status.ACTIVE.toString().equals(gremlinEdge.property(STATE_PROPERTY_KEY).value()))
 {


- Madhan Neethiraj


On May 14, 2020, 10:41 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72512/
> ---
> 
> (Updated May 14, 2020, 10:41 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-3797
> https://issues.apache.org/jira/browse/ATLAS-3797
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Modified: _GraphHelper.getOrCreateEdge_ to use 
> _AtlasJanusGraph.getEdgeBetweenVertices_.
> Modified: _AtlasJanusGraph.getEdgeBetweenVertices_ modified fetch.
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  35274c9d1 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> b1368e5e9 
> 
> 
> Diff: https://reviews.apache.org/r/72512/diff/2/
> 
> 
> Testing
> ---
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1896/
> 
> **Volume test**
> Case edge creation involves existing entity where large number of existing 
> edges (200K), this change shows significant improvement.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

[jira] [Commented] (ATLAS-3798) Import Service: Percentage Display Problem with Large Datasets

[Nikhil Bonte (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108334#comment-17108334
 ] 

Nikhil Bonte commented on ATLAS-3798:
-

+1 for the patch.

Thanks [~amestry]

> Import Service: Percentage Display Problem with Large Datasets
> --
>
> Key: ATLAS-3798
> URL: https://issues.apache.org/jira/browse/ATLAS-3798
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Minor
> Fix For: trunk
>
> Attachments: ATLAS-3798-Import-percentage-display-fix.patch
>
>
> *Steps to Duplicate*
>  # Start migration import with 100M entities. Observe the log for 
> _last-imported_.
> _Expected output:_ Display should proceed in increasing order of %ages.
> _Actual output_: Display starts showing -ve %ages after 37%.
> *Root Cause*
> Existing implementation uses _integer_ for storing _currentIndex_.
> *Additional Information*
>  * Start position computation is not impacted.
>  * Resume is not impacted. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3799) Migration Import: EntityConsumer Adds GUIDs to Result that Will Not Get Cleared

[Nikhil Bonte (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108331#comment-17108331
 ] 

Nikhil Bonte commented on ATLAS-3799:
-

+1 for the patch.

Thanks [~amestry]

> Migration Import: EntityConsumer Adds GUIDs to Result that Will Not Get 
> Cleared
> ---
>
> Key: ATLAS-3799
> URL: https://issues.apache.org/jira/browse/ATLAS-3799
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3799-EntityConumer-only-adds-entity-GUIDs-that.patch
>
>
> *Background*
> Zip file-based migration import introduced _EntityConsumer_. This uses 
> _StatusReporter_ that is used to determine which which of the produced items 
> are processed. This, in turn, helps determine the right position to set in 
> case of resuming the import.
> The _EntityConumer_ class is responsible for adding the right GUIDs to the 
> result that helps _StatusReporter_ determine the right GUIDs. This is done 
> correctly except that it also adds additional GUIDs that were updated or 
> deleted. This causes _StatusReporter_._processedSet_ to be filled with GUIDs 
> that will never be cleared. This will grow over time and can potentially lead 
> to memory pressure.
> *Solution*
> Add only the GUIDs that were processed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72474: ATLAS-3783 : DSL query search should return results for both the relationship edge directions

[chaitali]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72474/
---

(Updated May 15, 2020, 12:14 p.m.)


Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Bugs: ATLAS-3783
https://issues.apache.org/jira/browse/ATLAS-3783


Repository: atlas


Description
---

Unable to get results for Dsl query eg;hive_column where table.name="employee".
Due to regression only one edge direction was giving results.
This patch gives a fix for results for both relationship edge direction queries.
Documentation added for this fiv in Advanced DSl search.


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
129c20354 


Diff: https://reviews.apache.org/r/72474/diff/14/

Changes: https://reviews.apache.org/r/72474/diff/13-14/


Testing
---

TestCases added.


Thanks,

chaitali

[GitHub] [atlas] AmeerAssi commented on pull request #98: [ATLAS-3800] AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

AmeerAssi commented on pull request #98:
URL: https://github.com/apache/atlas/pull/98#issuecomment-629128363


   Thank you.
   Opened this issue:
   https://issues.apache.org/jira/browse/ATLAS-3800
   
   @sarathkumarsubramanian let me know if anything else is needed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [atlas] AmeerAssi removed a comment on pull request #98: [ATLAS-3800] AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

AmeerAssi removed a comment on pull request #98:
URL: https://github.com/apache/atlas/pull/98#issuecomment-629108017


   Thank you.
   Opened this issue:
   https://issues.apache.org/jira/browse/ATLAS-3800
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [atlas] AmeerAssi commented on pull request #98: AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

AmeerAssi commented on pull request #98:
URL: https://github.com/apache/atlas/pull/98#issuecomment-629108017


   Thank you.
   Opened this issue:
   https://issues.apache.org/jira/browse/ATLAS-3800
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [atlas] AmeerAssi edited a comment on pull request #98: AWS scheme is missing the aws account id that contains the S3 bucket.

[GitBox]

AmeerAssi edited a comment on pull request #98:
URL: https://github.com/apache/atlas/pull/98#issuecomment-627370163


   @sarathkumarsubramanian kind reminder for this review.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (ATLAS-3800) AWS scheme is missing the aws account id that contains the S3 bucket

[Ameer Assi (Jira)]

Ameer Assi created ATLAS-3800:
-

 Summary: AWS scheme is missing the aws account id that contains 
the S3 bucket
 Key: ATLAS-3800
 URL: https://issues.apache.org/jira/browse/ATLAS-3800
 Project: Atlas
  Issue Type: New Feature
Reporter: Ameer Assi


We are using Atlas models in our product.
the product gives scan capabilities for customer's S3 in AWS.
when scan result shown based on the model, it is missing for which account this 
bucket belongs. so it will be easy to access the account and review the scanned 
information by the customer.

 

https://github.com/apache/atlas/pull/98



--
This message was sent by Atlassian Jira
(v8.3.4#803005)