Re: [VOTE] Release Apache Atlas version 2.1.0 - rc2

[Jayendra Parab]

+1 for Apache Atlas 2.1.0 RC2 release

Validated the following:
1. Verified the source SHA512 and MD5 checksum
2. Verified the build with -Pdist,embedded-hbase-solr
3. Started Atlas server successfully
4. Verified quick_start.py, all entities and tag get created successfully
5. Created entities, applied tags from UI
6. Checked Basic and advanced search

Regards,
Jayendra.

On Tue, Jul 7, 2020 at 9:44 AM Nikhil Bonte <
nikhil.bo...@freestoneinfotech.com> wrote:

> +1 for Apache Atlas 2.1.0 - rc2 release
>
> Validated the following items.
>
>- Verified PGP signature.
>- Verified md5 and SHA512 digests.
>- Packaged with embedded-hbase-solr
>- Verified entity creation & tag associations using quick_start.py
>- Tested basic, quick & advanced search.
>- Tested tag dissociation
>
> Thanks
> Nikhil P. Bonte
>
> On Mon, Jul 6, 2020 at 12:11 PM Ramesh Mani  wrote:
>
> > +1 for Apache Atlas 2.1.0 rc2 release.
> >
> > - Build successfully with the source
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
> > - Verified PGP signature.
> > - Verified md5 and SHA512 digests.
> >
> > Thanks.
> > Ramesh
> >
> > On Sat, Jul 4, 2020 at 11:50 PM Madhan Neethiraj 
> > wrote:
> >
> > > Atlas team,
> > >
> > > Apache Atlas 2.1.0 rc2 is now available for vote within dev community.
> > >
> > > Links to the release artifacts are given below. Please review and vote.
> > >
> > > The vote will be open for at least 72 hours or until necessary votes
> are
> > > reached.
> > >   [ ] +1 Approve
> > >   [ ] +0 No opinion
> > >   [ ] -1 Disapprove (and reason why)
> > >
> > > Thanks,
> > > Madhan
> > >
> > >
> > > List of all issues addressed in this release:
> > > https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
> > > status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
> > >
> > > Git tag for the release:
> > > https://github.com/apache/atlas/tree/release-2.1.0-rc2
> > > Sources for the release:
> > >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
> > >
> > > Source release verification:
> > >   PGP Signature:
> > >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.asc
> > >   SHA512 Hash:
> > >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.sha512
> > >   MD5 Hash:
> > >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.md5
> > >
> > > Keys to verify the signature of the release artifacts are available at:
> > > https://dist.apache.org/repos/dist/release/atlas/KEYS
> > >
> > > New features/enhancements:
> > >   - Quick Search: provides a simpler search experience with type-ahead
> > > suggestions
> > >   - Business Metadata: enables augmenting entity-types with additional
> > > attributes, search entities using these attributes
> > >   - Labels: ability to add/remove labels on entities, and search
> entities
> > > using labels
> > >   - Custom Attributes: ability to add entity instance specific custom
> > > attributes i.e. attributes not defined in entity-def or business
> metadata
> > >   - Entity Purge: added REST APIs to purge deleted entities
> > >
> > > Enhancements:
> > >   - Search: ability to find entities by more than one classification
> > >   - Performance: improvements in lineage retrieval and
> > > classification-propagation
> > >   - Notification: ability to process notifications from multiple Kafka
> > > topics
> > >   - Hive Hook: tracks process-executions via hive_process_execution
> > > entities
> > >   - Hive Hook: captures DDL operations via hive_db_ddl and
> hive_table_ddl
> > > entities
> > >   - Notification: introduced shell entities to record references to
> > > non-existing entities in notifications
> > >   - Spark: added model to capture Spark entities, processes and
> > > relationships
> > >   - AWS S3: introduced updated model to capture AWS S3 entities and
> > > relationships
> > >   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2
> > > entities and relationships
> > >   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework
> > > 4.3.20
> > >   - Authorization: updated to cover new features, like: business
> > metadata,
> > > labels, purge
> > >   - UI: multiple UI improvements, including a beta UI
> > >
> > >
> > >
> >
>
>
> --
> Regards
> Nikhil P. Bonte
>

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc2

[Nikhil Bonte]

+1 for Apache Atlas 2.1.0 - rc2 release

Validated the following items.

   - Verified PGP signature.
   - Verified md5 and SHA512 digests.
   - Packaged with embedded-hbase-solr
   - Verified entity creation & tag associations using quick_start.py
   - Tested basic, quick & advanced search.
   - Tested tag dissociation

Thanks
Nikhil P. Bonte

On Mon, Jul 6, 2020 at 12:11 PM Ramesh Mani  wrote:

> +1 for Apache Atlas 2.1.0 rc2 release.
>
> - Build successfully with the source
>
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
> - Verified PGP signature.
> - Verified md5 and SHA512 digests.
>
> Thanks.
> Ramesh
>
> On Sat, Jul 4, 2020 at 11:50 PM Madhan Neethiraj 
> wrote:
>
> > Atlas team,
> >
> > Apache Atlas 2.1.0 rc2 is now available for vote within dev community.
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes are
> > reached.
> >   [ ] +1 Approve
> >   [ ] +0 No opinion
> >   [ ] -1 Disapprove (and reason why)
> >
> > Thanks,
> > Madhan
> >
> >
> > List of all issues addressed in this release:
> > https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
> > status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/tree/release-2.1.0-rc2
> > Sources for the release:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
> >
> > Source release verification:
> >   PGP Signature:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.asc
> >   SHA512 Hash:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.sha512
> >   MD5 Hash:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.md5
> >
> > Keys to verify the signature of the release artifacts are available at:
> > https://dist.apache.org/repos/dist/release/atlas/KEYS
> >
> > New features/enhancements:
> >   - Quick Search: provides a simpler search experience with type-ahead
> > suggestions
> >   - Business Metadata: enables augmenting entity-types with additional
> > attributes, search entities using these attributes
> >   - Labels: ability to add/remove labels on entities, and search entities
> > using labels
> >   - Custom Attributes: ability to add entity instance specific custom
> > attributes i.e. attributes not defined in entity-def or business metadata
> >   - Entity Purge: added REST APIs to purge deleted entities
> >
> > Enhancements:
> >   - Search: ability to find entities by more than one classification
> >   - Performance: improvements in lineage retrieval and
> > classification-propagation
> >   - Notification: ability to process notifications from multiple Kafka
> > topics
> >   - Hive Hook: tracks process-executions via hive_process_execution
> > entities
> >   - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl
> > entities
> >   - Notification: introduced shell entities to record references to
> > non-existing entities in notifications
> >   - Spark: added model to capture Spark entities, processes and
> > relationships
> >   - AWS S3: introduced updated model to capture AWS S3 entities and
> > relationships
> >   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2
> > entities and relationships
> >   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework
> > 4.3.20
> >   - Authorization: updated to cover new features, like: business
> metadata,
> > labels, purge
> >   - UI: multiple UI improvements, including a beta UI
> >
> >
> >
>


-- 
Regards
Nikhil P. Bonte

[jira] [Created] (ATLAS-3878) Notifications: Improve Memory Usage in Scale Enviroment

[Ashutosh Mestry (Jira)]

Ashutosh Mestry created ATLAS-3878:
--

 Summary: Notifications: Improve Memory Usage in Scale Enviroment
 Key: ATLAS-3878
 URL: https://issues.apache.org/jira/browse/ATLAS-3878
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0, trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Background*

As part of entity creation, Atlas sends notifications of different types. 
Current implementation, to listeners. Listeners in turn perform specific tasks.

At a more concrete level, the _EntityAuditListenerV2_ will write audits and the 
_NotificationEntityChangeListener_ will send Kafka notifications.

Each of the listeners create notification objects. The notification objects are 
large in number and are short lived.

The transient nature of the notification objects causes memory pressure in 
scale environment.

*Solution*

Create object pool for notification objects. This way objects can be reused.and 
existing design can be kept in tact. This will also offer benefit of using 
existing test setup for verification.

*Tests Used*

_Setup_ 

Create a test rig that will spawn multiple works that will invoke Atlas' bulk 
APIs for entity creation.

Node: 40 workers, 8 GB allocated memory and 40 cores.

**_Observation_

About 40 mins into the exercise, memory pressure builds up causing GC collects 
to take longer. This causes ZK timeout and finally Atlas process crashes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72638/#review221139
---




client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Line 29 (original), 32 (patched)


Avoid * instead of individual imports.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Line 41 (original), 46 (patched)


Avoid replacing individual imports with '*'.



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 707 (patched)


Remove: TODO



webapp/pom.xml
Line 744 (original), 744 (patched)


Avoid.



webapp/src/test/java/org/apache/atlas/web/TestUtils.java
Lines 60 (patched)


Remove.



webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java
Line 95 (original), 96 (patched)


Remove comment.



webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java
Lines 145 (patched)


Remove comment.



webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java
Lines 31 (patched)


Avoid.



webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java
Lines 46 (patched)


Avoid.



webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java
Lines 31 (patched)


Avoid.



webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java
Lines 92 (patched)


Can you investigate loading JSONs instead of hand coding entities.


- Ashutosh Mestry


On July 6, 2020, 5:53 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72638/
> ---
> 
> (Updated July 6, 2020, 5:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> There are many missing API endpoints in AtlasClientV2. This solution is 
> adding functions corresponding to those missing APIs for the following Rest 
> endpoints.
> 
> 1. TypeRest
> 2. EntityRest
> 3. LineageRest
> 4. DiscoveryRest
> 5. GlossaryRest
> 6. RelationshipRest
> 
> This will enable users to test and integrate with APIs more effectively via 
> AtlasCient.
> 
> 
> Diffs
> -
> 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 7b6d1d0f3 
>   webapp/pom.xml 3c55b4dd4 
>   webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d 
>   webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
> 680028892 
>   webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
> 808f62354 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
>  cc883d615 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
>  331ea2c47 
>   webapp/src/test/resources/json/search-parameters/attribute-filters.json 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72638/diff/2/
> 
> 
> Testing
> ---
> 
> IT has been added for newly created functions in AtlasClientV2.
> Validated by running IT on local system.
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>

[jira] [Created] (ATLAS-3877) Purged entity audit throws 404 Entity not found error

[Sharmadha S (Jira)]

Sharmadha S created ATLAS-3877:
--

 Summary: Purged entity audit throws 404 Entity not found error
 Key: ATLAS-3877
 URL: https://issues.apache.org/jira/browse/ATLAS-3877
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 2.1.0
Reporter: Sharmadha S
 Fix For: 2.1.0
 Attachments: rc2.png

1. Create an hdfs_path entity.
2. Delete the entity using DELETE: /api/atlas/v2/entity/guid/. The entity 
is soft deleted
3. Purge the entity using PUT : /api/atlas/admin/purge with body [""]
4. Go to admin -> Administration -> Audits
5. Click on the GUID of purged entity. 

The purged entity details window opens but Atlas throws a 404 error "Given 
entity GUID is invalid/not found" . Attached screenshot.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3875) Adding missing APIs in AtlasClient with test cases

[Jyoti Singh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jyoti Singh updated ATLAS-3875:
---
Attachment: ATLAS-3875-1.patch

> Adding missing APIs in AtlasClient with test cases
> --
>
> Key: ATLAS-3875
> URL: https://issues.apache.org/jira/browse/ATLAS-3875
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Jyoti Singh
>Assignee: Jyoti Singh
>Priority: Major
>  Labels: api, client
> Attachments: ATLAS-3875-1.patch
>
>
> There are many new APIs added to Atlas Project but the corresponding  APIs 
> are missing from AtlasClientv2. The aim of this task is to complete the gap 
> amongst existing APIs and their endpoints in Atls client. This will also 
> include adding test cases via integration testing.
> There are functions from AtlasClient for the following REST endpoints
>  * TypeRest
>  * EntityRest
>  * LineageRest
>  * DiscoveryRest
>  * GlossaryRest
>  * RelationshipRest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints

[Jyoti Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72638/
---

(Updated July 6, 2020, 5:53 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
Subramanian, and Sidharth Mishra.


Bugs: ATLAS-3875
https://issues.apache.org/jira/browse/ATLAS-3875


Repository: atlas


Description
---

There are many missing API endpoints in AtlasClientV2. This solution is adding 
functions corresponding to those missing APIs for the following Rest endpoints.

1. TypeRest
2. EntityRest
3. LineageRest
4. DiscoveryRest
5. GlossaryRest
6. RelationshipRest

This will enable users to test and integrate with APIs more effectively via 
AtlasCient.


Diffs (updated)
-

  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 7b6d1d0f3 
  webapp/pom.xml 3c55b4dd4 
  webapp/src/test/java/org/apache/atlas/web/TestUtils.java e22a1c10d 
  webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
680028892 
  webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 
808f62354 
  
webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
 cc883d615 
  webapp/src/test/java/org/apache/atlas/web/integration/GlossaryClientV2IT.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/integration/LineageClientV2IT.java 
PRE-CREATION 
  
webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
 331ea2c47 
  webapp/src/test/resources/json/search-parameters/attribute-filters.json 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72638/diff/2/

Changes: https://reviews.apache.org/r/72638/diff/1-2/


Testing
---

IT has been added for newly created functions in AtlasClientV2.
Validated by running IT on local system.


Thanks,

Jyoti Singh

Re: Apache Atlas 2.1 release

[Robert Yokota]

Thanks for the update!

Kind regards,
Robert

On Sun, Jul 5, 2020 at 11:00 PM Madhan Neethiraj  wrote:

> Hi Robert,
>
>
>
> 2.1.0 release-candidate 2 was sent out for vote today. If all goes well, I
> expect the release to be complete by July-15-2020.
>
>
>
> Regards,
>
> Madhan
>
>
>
> From: Robert Yokota 
> Reply-To: "u...@atlas.apache.org" 
> Date: Saturday, July 4, 2020 at 8:36 AM
> To: "u...@atlas.apache.org" 
> Cc: "dev@atlas.apache.org" 
> Subject: Re: Apache Atlas 2.1 release
>
>
>
> Hi Madhan,
>
>
>
> Happy 4th of July!  Is there an ETA for the 2.1.0 release?
>
>
>
> Kind regards,
>
> Robert
>
>
>
> On Mon, Jun 1, 2020 at 10:11 PM Madhan Neethiraj 
> wrote:
>
> Bolke,
>
> As detailed in my earlier email, the release will be from branch-2.0. This
> branch is in lockdown, open for release-blockers and critical backports
> from master.
>
> I am planning to send a release-candidate for vote tomorrow (Tuesday,
> June-2), once following commits are merged into branch-2.0: ATLAS-3604,
> ATLAS-3320, ATLAS-3737, ATLAS-3797,  ATLAS-3641, ATLAS-3641.
>
> If any commits are to be cherry-picked into 2.1.0 release, please have the
> patch ready asap and let me know.
>
> Regards,
> Madhan
>
> On 5/22/20, 12:19 AM, "Bolke de Bruin"  wrote:
>
> Awesome! Do you have any idea what is in scope or out scope of this
> release? Is it a branch of master or an update of 2.0 for example with
> cherry picks from master?
>
> Sent from my iPhone
>
> > On 17 May 2020, at 16:32, Nixon Rodrigues <
> nixon.rodrig...@freestoneinfotech.com> wrote:
> >
> > Madhan,
> >
> > Thanks for volunteering this release, the release plan looks good to
> me.
> > +1 for the release
> >
> > Regards
> > Nixon
> >
> >
> >> On Sun, May 17, 2020 at 11:47 AM Madhan Neethiraj <
> mad...@apache.org> wrote:
> >>
> >> Atlas community,
> >>
> >>
> >>
> >> In the past 3 months since I sent the email on Apache Atlas 2.1
> release,
> >> the community has added 100+ commits to improve various areas like
> search,
> >> UI, performance, authorization, bulk import of glossary &
> >> business-metadata, support for ADLS-Ge2 and AWS-S3 entity types,
> updated
> >> JanusGraph version. Thanks to everyone who contributed to Apache
> Atlas -
> >> making it a feature rich, enterprise-ready, open-source governance
> tool
> >> backed by a thriving community.
> >>
> >>
> >>
> >> Let’s started on Apache Atlas 2.1 release! I volunteer to be the
> >> release-manager for this release and propose the following timeline:
> >> 05/18, Mon: branch-2.0 goes into lockdown mode to prepare for 2.1
> release.
> >> During lockdown period, only commits for release-blocker issues
> must be
> >> merged in branch-2.0. Master branch will remain open for all commits
> >> 05/18 – 05/22, Mon – Fri: community to validate branch-2.0 builds
> and
> >> address release-blocker issues
> >> 05/25, Mon: release candidate out for vote by the community
> >> 05/29, Fri: release Apache Atlas 2.1, assuming successful
> completion of
> >> votes
> >>
> >>
> >> Please review and let me know your comments/suggestions.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Madhan
> >>
> >>
> >>
> >> From: Madhan Neethiraj 
> >> Date: Tuesday, February 18, 2020 at 4:10 PM
> >> To: "dev@atlas.apache.org" , "
> u...@atlas.apache.org"
> >> 
> >> Subject: Apache Atlas 2.1 release
> >>
> >>
> >>
> >> Atlas community,
> >>
> >>
> >>
> >> Over past months the dev community has been busy in enhancing
> Apache Atlas
> >> with new features, improvements and fixes. Here are few
> >> features/enhancements since last major release, Apache Atlas 2.0:
> >>
> >>- added quick-search feature, to provide a simpler search
> experience
> >> with type-ahead suggestions
> >>
> >>- introduced Namespaces feature, which allows grouping of
> attributes
> >> to be applied to multiple entity-types
> >>
> >>- introduced labels on entity instances, and search for entities
> using
> >> the label
> >>
> >>- enhancement to support entity instance specific custom
> attributes
> >>
> >>- enhanced search to find entities by more than one
> classification
> >>
> >>- introduced shell/incomplete entities to handle notifications
> >> referencing entities that don’t (yet) exist in Atlas
> >>
> >>- added REST APIs to purge deleted entities
> >>
> >>- performance improvements in lineage retrieval and
> tag-propagation
> >>
> >>- updated Atlas server to process notifications from multiple
> Kafka
> >> topics
> >>
> >>- updated Hive hook to track process executions, via
> >> hive_process_execution entities
> >>
> >>- updated Hive hook to capture DDL operations, 

Re: Review Request 72636: NotificationHookConsumer: Concurrent Message Processing

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72636/
---

(Updated July 6, 2020, 5:23 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Updates include: Added test setup.


Bugs: ATLAS-3874
https://issues.apache.org/jira/browse/ATLAS-3874


Repository: atlas


Description
---

**Background**
Please see bug description.

**Approach**
At a high-level: Introduce a notion where the individual consumers are aware of 
the entities being processed by each other. If there are no entities being 
processed concurrently, everything proceeds as usual (the way it is before his 
change). If same entity is being procesed by multiple consumers, then one 
consumer waits for the other to finish before proceeding.

Classes:
New *UniqueKeysExtractor*: Extracts values of unique keys from 
*AtlasEntitiesWithExtInfo*. It navigates *relationshipAttributes* and 
*attributes* that has *objectRef* set.
New *UniquenessChecker*: Maintains a set of unique keys provided by 
*UniqueKeysExtractor*. It detects the presence of duplicates and waits until 
duplicates are resolved.
Modified *NoitficationHookConsumer.createOrUpdate* 
- Updates *UniquenessChecker* with output from *UniqueKeysExtractor*. Clears 
the keys at the end of entity creation.
- Handles JanusGraph's *PermanentLockingException*. See 
[ATLAS-3801](https://issues.apache.org/jira/browse/ATLAS-3801).
Modified *NotificationHookConsumer*: Accepts an shared instance of 
*UniqunessChecker*.

**Additional Logging**
Log entry is added when attempt is made to creates entities concurrently.


Diffs
-

  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 3f1ea05e1 
  webapp/src/main/java/org/apache/atlas/notification/UniqueKeysExtractor.java 
PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/notification/UniquenessChecker.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/notification/UniquenessCheckerTest.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/72636/diff/1/


Testing (updated)
---

**Unit tests**
Tests added to verify new clases.

*UniquessCheckerTest*
Performs worst case checking by adding 100s of keys that are duplicates and 
verifies output of those.

**Functional tests**
Used Spark hook to verify. These sequence of commands will create shell entity 
and first class entity. If resolution for first class entity is available, then 
no shell entity is created.

Start Spark shell using:
```
sudo -u hdfs spark-shell
```

Spark sql commands:
```
spark.sql("create table default.t1_1381104676(col1 int)")
spark.sql("create table default.t2_1381104676(col2 int)")
spark.sql("select * from t1_1381104676, t2_1381104676 where 
col1=col2").write.saveAsTable("t3_1381104676")
```

**Volume test**
Medium-size Kafka dump added.

*Setup* Add same kafka dump to multiple topics. This will result in contention 
for almost every message.

**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2003/


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-3841) Response Headers: Code refactoring

[Mandar Ambawane (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandar Ambawane updated ATLAS-3841:
---
Description: 
Currently headers setting code is scattered at different places.
In this patch, code refactoring is done to set response headers from 
AtlasHeaderFilter.java

This patch provides flexibility to select response headers to be excluded.
Also this patch provides flexibility to select file formats to be excluded from 
applying response headers.

> Response Headers: Code refactoring
> --
>
> Key: ATLAS-3841
> URL: https://issues.apache.org/jira/browse/ATLAS-3841
> Project: Atlas
>  Issue Type: Bug
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
>
> Currently headers setting code is scattered at different places.
> In this patch, code refactoring is done to set response headers from 
> AtlasHeaderFilter.java
> This patch provides flexibility to select response headers to be excluded.
> Also this patch provides flexibility to select file formats to be excluded 
> from applying response headers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72615: ATLAS-3855 Bulk entity tag association and bulk api enhancement

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72615/#review221135
---



Consider updating all methods that deal with multiple entities to ignore 
unauthorized entities. Below is the list of such methods from quick look:

  - GET/bulk - EntityREST.getByGuids()
  - DELETE /bulk - EntityREST.deleteByGuids()
  - POST   /bulk/classification  - 
EntityREST.addClassification()
  - POST   /bulk/setClassifications  - 
EntityREST.setClassifications()
  - GET/bulk/headers - EntityREST.getEntityHeaders()
  - GET/bulk/uniqueAttribute/type/{typeName} - 
EntityREST.getEntitiesByUniqueAttributes()

Instead of changing signature for all these methods, I suggest to add flag 
RequestContext.ignoreUnauthorizedEntities and populate this from 
AuditFilter.doFilter().

- Madhan Neethiraj


On July 6, 2020, 3:02 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72615/
> ---
> 
> (Updated July 6, 2020, 3:02 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3855
> https://issues.apache.org/jira/browse/ATLAS-3855
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> API : /api/atlas/v2/entity/bulk/classification & v2/entity/bulk : if some 
> guids belong to entities on which user is unauthorized other guids belong to 
> entities on which user is authorized This APIs fail with 403 error without 
> returning the authorized entities.
> 
> 1.Unauthorized guids are filtered with this patch for both the APIs.
> 2.Added ignoreUnauthorisedGuids flag for /bulk/classification API as it 
> doesn't return any object
> 2.Also added unauthorized guids in return object of /bulk in response for 
> user's reference.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  7b9455ef3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  bf1629cb3 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  b9cbef1b0 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 00b29e6c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 88de8b679 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 
> 615bc0f1b 
> 
> 
> Diff: https://reviews.apache.org/r/72615/diff/7/
> 
> 
> Testing
> ---
> 
> Tested with ranger policies:
> For /bulk api following policy were applied:
> 1.Added hive_column entity as entity type.
> 2.Included admin as user and prohibited access for read entity all this under 
> deny policy.
> 
> For /bulk/classification following policy were applied:
> 1.Added hive_column entity as entity type.
> 2.Gave all the access to admin but kwpt hive_column entity type in excluded 
> state.
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 72615: ATLAS-3855 Bulk entity tag association and bulk api enhancement

[chaitali]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72615/
---

(Updated July 6, 2020, 3:02 p.m.)


Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Bugs: ATLAS-3855
https://issues.apache.org/jira/browse/ATLAS-3855


Repository: atlas


Description
---

API : /api/atlas/v2/entity/bulk/classification & v2/entity/bulk : if some guids 
belong to entities on which user is unauthorized other guids belong to entities 
on which user is authorized This APIs fail with 403 error without returning the 
authorized entities.

1.Unauthorized guids are filtered with this patch for both the APIs.
2.Added ignoreUnauthorisedGuids flag for /bulk/classification API as it doesn't 
return any object
2.Also added unauthorized guids in return object of /bulk in response for 
user's reference.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
 7b9455ef3 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 bf1629cb3 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 b9cbef1b0 
  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
00b29e6c8 
  webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 88de8b679 
  webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 
615bc0f1b 


Diff: https://reviews.apache.org/r/72615/diff/7/

Changes: https://reviews.apache.org/r/72615/diff/6-7/


Testing
---

Tested with ranger policies:
For /bulk api following policy were applied:
1.Added hive_column entity as entity type.
2.Included admin as user and prohibited access for read entity all this under 
deny policy.

For /bulk/classification following policy were applied:
1.Added hive_column entity as entity type.
2.Gave all the access to admin but kwpt hive_column entity type in excluded 
state.


Thanks,

chaitali

[jira] [Resolved] (ATLAS-3844) Ignore relationship attributes while fetching entities in Import Hive bridge.

[Nixon Rodrigues (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues resolved ATLAS-3844.

Resolution: Fixed

> Ignore relationship attributes while fetching entities in Import Hive bridge.
> -
>
> Key: ATLAS-3844
> URL: https://issues.apache.org/jira/browse/ATLAS-3844
> Project: Atlas
>  Issue Type: Improvement
>  Components: hive-integration
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 
> 0001-ATLAS-3844-Ignore-relationship-attributes-while-fetc.patch
>
>
> Currently in Import Hive bridge, the hive_db and hive_table entities are 
> fetched along relationshipAttributes and related Attributes and this 
> attributes are cleared thereafter.
> Instead fetching relationshipAttributes and related Attributes in response, 
> it is will good to ignoreRelationship to reduce the size of response payload.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3844) Ignore relationship attributes while fetching entities in Import Hive bridge.

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152022#comment-17152022
 ] 

ASF subversion and git services commented on ATLAS-3844:


Commit cea851c5513cede054bc7704ac24c04cc498cd32 in atlas's branch 
refs/heads/master from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=cea851c ]

ATLAS-3844 - Ignore relationship attributes while fetching entities in Import 
Hive bridge.

Change-Id: I1d0eeb478286e896af733b51fe4f7fc62c594899


> Ignore relationship attributes while fetching entities in Import Hive bridge.
> -
>
> Key: ATLAS-3844
> URL: https://issues.apache.org/jira/browse/ATLAS-3844
> Project: Atlas
>  Issue Type: Improvement
>  Components: hive-integration
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 
> 0001-ATLAS-3844-Ignore-relationship-attributes-while-fetc.patch
>
>
> Currently in Import Hive bridge, the hive_db and hive_table entities are 
> fetched along relationshipAttributes and related Attributes and this 
> attributes are cleared thereafter.
> Instead fetching relationshipAttributes and related Attributes in response, 
> it is will good to ignoreRelationship to reduce the size of response payload.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 6, 2020, 9:14 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs
-

  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 


Diff: https://reviews.apache.org/r/72646/diff/1/


Testing (updated)
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

[jira] [Commented] (ATLAS-3819) Modernize client side technology stack

[Csaba Koncz (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151875#comment-17151875
 ] 

Csaba Koncz commented on ATLAS-3819:


- the Typesecript loader is not needed, it is only an option. As it has no 
runtime implications, we might discard it if it slows down the build too much. 
I would keep it until we can blame something serious on it. 
babel-loader can definitely be added for being able to use ES6 in javascript 
sources, I will configure it and add an example. 

- express is used mainly because of the API requests proxying at 
[https://github.com/csabakoncz/atlas-newui/blob/master/server.js#L33]
This way I have the Java Atlas server running somewhere and I can test 
different versions of the UI with the same REST api. 
The nodejs+express  server is used only for development, the packaging can be 
done the old way (i.e. the content of the 'dist' folder is added to atlas.war).
I do not plan changes to packaging/deployment (atlas.war is fine, although I 
have seen questions about separating the UI from the backend).

My original plan with Typescript was to have typings for API responses (maybe 
generated from the swagger.json). That way we might have Java-like content 
assist when implementing the views.

> Modernize client side technology stack
> --
>
> Key: ATLAS-3819
> URL: https://issues.apache.org/jira/browse/ATLAS-3819
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 3.0.0
>Reporter: Csaba Koncz
>Priority: Major
> Attachments: image-2020-07-04-16-42-10-643.png
>
>
> While RequireJS has its merits and advantages, it might be worth considering 
> replacing it with another technology.
> A bundle loader like Webpack would open up a number of possibilities:
> - use of ES6 modules, which immediately improves tooling support
> - use TypeScript, which improves tooling support even more
> - a manageable code base would enable refactoring of the JavaScript code so 
> that no logic happens in the module initialization code,
> which in turn would make it possible to write tests for the JavaScript logic.
> As a POC I created a NodeJS project that transforms the current dashboardv2 
> code into Webpack bundles and demonstrates
> that the UI is able to work without RequireJS.
> Right now the JavaScript sources and index.html are used unchanged,
> but the next step would be to start converting the source to ES6 modules.
> Here is the code: https://github.com/csabakoncz/atlas-newui
> The most important file there is `webpack.config.js` that takes over the role 
> of RequireJS.config.
> The other file of interest might be `src/index.ts` which demonstrates the 
> ability to use TypeScript code even now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs
-

  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 


Diff: https://reviews.apache.org/r/72646/diff/1/


Testing
---

Manually tested
Precommit :


Thanks,

Pinal Shah

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc2

[Ramesh Mani]

+1 for Apache Atlas 2.1.0 rc2 release.

- Build successfully with the source
https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
- Verified PGP signature.
- Verified md5 and SHA512 digests.

Thanks.
Ramesh

On Sat, Jul 4, 2020 at 11:50 PM Madhan Neethiraj  wrote:

> Atlas team,
>
> Apache Atlas 2.1.0 rc2 is now available for vote within dev community.
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes are
> reached.
>   [ ] +1 Approve
>   [ ] +0 No opinion
>   [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Madhan
>
>
> List of all issues addressed in this release:
> https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
> status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.1.0-rc2
> Sources for the release:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
>
> Source release verification:
>   PGP Signature:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.asc
>   SHA512 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.sha512
>   MD5 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.md5
>
> Keys to verify the signature of the release artifacts are available at:
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>
> New features/enhancements:
>   - Quick Search: provides a simpler search experience with type-ahead
> suggestions
>   - Business Metadata: enables augmenting entity-types with additional
> attributes, search entities using these attributes
>   - Labels: ability to add/remove labels on entities, and search entities
> using labels
>   - Custom Attributes: ability to add entity instance specific custom
> attributes i.e. attributes not defined in entity-def or business metadata
>   - Entity Purge: added REST APIs to purge deleted entities
>
> Enhancements:
>   - Search: ability to find entities by more than one classification
>   - Performance: improvements in lineage retrieval and
> classification-propagation
>   - Notification: ability to process notifications from multiple Kafka
> topics
>   - Hive Hook: tracks process-executions via hive_process_execution
> entities
>   - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl
> entities
>   - Notification: introduced shell entities to record references to
> non-existing entities in notifications
>   - Spark: added model to capture Spark entities, processes and
> relationships
>   - AWS S3: introduced updated model to capture AWS S3 entities and
> relationships
>   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2
> entities and relationships
>   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework
> 4.3.20
>   - Authorization: updated to cover new features, like: business metadata,
> labels, purge
>   - UI: multiple UI improvements, including a beta UI
>
>
>

[jira] [Commented] (ATLAS-3819) Modernize client side technology stack

[Keval Bhatt (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151771#comment-17151771
 ] 

Keval Bhatt commented on ATLAS-3819:


[~Koncz] I found that you are using webpack-dev-server and node server as well, 
is there any specific reason?

> Modernize client side technology stack
> --
>
> Key: ATLAS-3819
> URL: https://issues.apache.org/jira/browse/ATLAS-3819
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 3.0.0
>Reporter: Csaba Koncz
>Priority: Major
> Attachments: image-2020-07-04-16-42-10-643.png
>
>
> While RequireJS has its merits and advantages, it might be worth considering 
> replacing it with another technology.
> A bundle loader like Webpack would open up a number of possibilities:
> - use of ES6 modules, which immediately improves tooling support
> - use TypeScript, which improves tooling support even more
> - a manageable code base would enable refactoring of the JavaScript code so 
> that no logic happens in the module initialization code,
> which in turn would make it possible to write tests for the JavaScript logic.
> As a POC I created a NodeJS project that transforms the current dashboardv2 
> code into Webpack bundles and demonstrates
> that the UI is able to work without RequireJS.
> Right now the JavaScript sources and index.html are used unchanged,
> but the next step would be to start converting the source to ES6 modules.
> Here is the code: https://github.com/csabakoncz/atlas-newui
> The most important file there is `webpack.config.js` that takes over the role 
> of RequireJS.config.
> The other file of interest might be `src/index.ts` which demonstrates the 
> ability to use TypeScript code even now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3876) Relationship Search API not showing correct approximateCount

[Pinal (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pinal updated ATLAS-3876:
-
Labels: Relationships  (was: )

> Relationship Search API not showing correct approximateCount
> 
>
> Key: ATLAS-3876
> URL: https://issues.apache.org/jira/browse/ATLAS-3876
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>  Labels: Relationships
>
> Relationship Search API for hive columns showing *approximateCount* as -1 in 
> response.
>  E.g.
> {code:java}
> v2/search/relationship?limit=10=0=3e424245-0676-4d47-96a1-e17228b38367=__hive_table.columns=name=ASCENDING=true{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (ATLAS-3819) Modernize client side technology stack

[Keval Bhatt (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151768#comment-17151768
 ] 

Keval Bhatt edited comment on ATLAS-3819 at 7/6/20, 6:02 AM:
-

[~Koncz] thanks for coming up with the improvement.

I have checked the repo -> [https://github.com/csabakoncz/atlas-newui 
|https://github.com/csabakoncz/atlas-newui] and have few questions:

can we use webpack-dev-server instead of creating an express node server 
because we need a server only for development.

Do we need typescript loader for atlas code? 
We can use babel-loader for es6 which can easily be used in the current atlas 
code.


was (Author: kevalbhatt18):
[~Koncz] thanks for coming up with the improvement.

I have checked the repo -> [https://github.com/csabakoncz/atlas-newui 
|https://github.com/csabakoncz/atlas-newui] and have few questions:

can we use webpack-dev-server instead of creating an express node server 
because we need a server only for development.

Do we need typescript loader for atlas code? for backbone, we are using plain 
javascript. we can use babel-loader for es6.

> Modernize client side technology stack
> --
>
> Key: ATLAS-3819
> URL: https://issues.apache.org/jira/browse/ATLAS-3819
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 3.0.0
>Reporter: Csaba Koncz
>Priority: Major
> Attachments: image-2020-07-04-16-42-10-643.png
>
>
> While RequireJS has its merits and advantages, it might be worth considering 
> replacing it with another technology.
> A bundle loader like Webpack would open up a number of possibilities:
> - use of ES6 modules, which immediately improves tooling support
> - use TypeScript, which improves tooling support even more
> - a manageable code base would enable refactoring of the JavaScript code so 
> that no logic happens in the module initialization code,
> which in turn would make it possible to write tests for the JavaScript logic.
> As a POC I created a NodeJS project that transforms the current dashboardv2 
> code into Webpack bundles and demonstrates
> that the UI is able to work without RequireJS.
> Right now the JavaScript sources and index.html are used unchanged,
> but the next step would be to start converting the source to ES6 modules.
> Here is the code: https://github.com/csabakoncz/atlas-newui
> The most important file there is `webpack.config.js` that takes over the role 
> of RequireJS.config.
> The other file of interest might be `src/index.ts` which demonstrates the 
> ability to use TypeScript code even now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3819) Modernize client side technology stack

[Keval Bhatt (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151768#comment-17151768
 ] 

Keval Bhatt commented on ATLAS-3819:


[~Koncz] thanks for coming up with the improvement.

I have checked the repo -> [https://github.com/csabakoncz/atlas-newui 
|https://github.com/csabakoncz/atlas-newui] and have few questions:

can we use webpack-dev-server instead of creating an express node server 
because we need a server only for development.

Do we need typescript loader for atlas code? for backbone, we are using plain 
javascript. we can use babel-loader for es6.

> Modernize client side technology stack
> --
>
> Key: ATLAS-3819
> URL: https://issues.apache.org/jira/browse/ATLAS-3819
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 3.0.0
>Reporter: Csaba Koncz
>Priority: Major
> Attachments: image-2020-07-04-16-42-10-643.png
>
>
> While RequireJS has its merits and advantages, it might be worth considering 
> replacing it with another technology.
> A bundle loader like Webpack would open up a number of possibilities:
> - use of ES6 modules, which immediately improves tooling support
> - use TypeScript, which improves tooling support even more
> - a manageable code base would enable refactoring of the JavaScript code so 
> that no logic happens in the module initialization code,
> which in turn would make it possible to write tests for the JavaScript logic.
> As a POC I created a NodeJS project that transforms the current dashboardv2 
> code into Webpack bundles and demonstrates
> that the UI is able to work without RequireJS.
> Right now the JavaScript sources and index.html are used unchanged,
> but the next step would be to start converting the source to ES6 modules.
> Here is the code: https://github.com/csabakoncz/atlas-newui
> The most important file there is `webpack.config.js` that takes over the role 
> of RequireJS.config.
> The other file of interest might be `src/index.ts` which demonstrates the 
> ability to use TypeScript code even now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Atlas 2.1 release

[Madhan Neethiraj]

Hi Robert,

 

2.1.0 release-candidate 2 was sent out for vote today. If all goes well, I 
expect the release to be complete by July-15-2020.

 

Regards,

Madhan

 

From: Robert Yokota 
Reply-To: "u...@atlas.apache.org" 
Date: Saturday, July 4, 2020 at 8:36 AM
To: "u...@atlas.apache.org" 
Cc: "dev@atlas.apache.org" 
Subject: Re: Apache Atlas 2.1 release

 

Hi Madhan,

 

Happy 4th of July!  Is there an ETA for the 2.1.0 release?  

 

Kind regards,

Robert

 

On Mon, Jun 1, 2020 at 10:11 PM Madhan Neethiraj  wrote:

Bolke,

As detailed in my earlier email, the release will be from branch-2.0. This 
branch is in lockdown, open for release-blockers and critical backports from 
master.

I am planning to send a release-candidate for vote tomorrow (Tuesday, June-2), 
once following commits are merged into branch-2.0: ATLAS-3604,  ATLAS-3320, 
ATLAS-3737, ATLAS-3797,  ATLAS-3641, ATLAS-3641.

If any commits are to be cherry-picked into 2.1.0 release, please have the 
patch ready asap and let me know.

Regards,
Madhan

On 5/22/20, 12:19 AM, "Bolke de Bruin"  wrote:

Awesome! Do you have any idea what is in scope or out scope of this 
release? Is it a branch of master or an update of 2.0 for example with cherry 
picks from master?

Sent from my iPhone

> On 17 May 2020, at 16:32, Nixon Rodrigues 
 wrote:
> 
> Madhan,
> 
> Thanks for volunteering this release, the release plan looks good to me.
> +1 for the release
> 
> Regards
> Nixon
> 
> 
>> On Sun, May 17, 2020 at 11:47 AM Madhan Neethiraj  
wrote:
>> 
>> Atlas community,
>> 
>> 
>> 
>> In the past 3 months since I sent the email on Apache Atlas 2.1 release,
>> the community has added 100+ commits to improve various areas like 
search,
>> UI, performance, authorization, bulk import of glossary &
>> business-metadata, support for ADLS-Ge2 and AWS-S3 entity types, updated
>> JanusGraph version. Thanks to everyone who contributed to Apache Atlas -
>> making it a feature rich, enterprise-ready, open-source governance tool
>> backed by a thriving community.
>> 
>> 
>> 
>> Let’s started on Apache Atlas 2.1 release! I volunteer to be the
>> release-manager for this release and propose the following timeline:
>> 05/18, Mon: branch-2.0 goes into lockdown mode to prepare for 2.1 
release.
>> During lockdown period, only commits for release-blocker issues must be
>> merged in branch-2.0. Master branch will remain open for all commits
>> 05/18 – 05/22, Mon – Fri: community to validate branch-2.0 builds and
>> address release-blocker issues
>> 05/25, Mon: release candidate out for vote by the community
>> 05/29, Fri: release Apache Atlas 2.1, assuming successful completion of
>> votes
>> 
>> 
>> Please review and let me know your comments/suggestions.
>> 
>> 
>> 
>> Thanks,
>> 
>> Madhan
>> 
>> 
>> 
>> From: Madhan Neethiraj 
>> Date: Tuesday, February 18, 2020 at 4:10 PM
>> To: "dev@atlas.apache.org" , 
"u...@atlas.apache.org"
>> 
>> Subject: Apache Atlas 2.1 release
>> 
>> 
>> 
>> Atlas community,
>> 
>> 
>> 
>> Over past months the dev community has been busy in enhancing Apache 
Atlas
>> with new features, improvements and fixes. Here are few
>> features/enhancements since last major release, Apache Atlas 2.0:
>> 
>>- added quick-search feature, to provide a simpler search experience
>> with type-ahead suggestions
>> 
>>- introduced Namespaces feature, which allows grouping of attributes
>> to be applied to multiple entity-types
>> 
>>- introduced labels on entity instances, and search for entities using
>> the label
>> 
>>- enhancement to support entity instance specific custom attributes
>> 
>>- enhanced search to find entities by more than one classification
>> 
>>- introduced shell/incomplete entities to handle notifications
>> referencing entities that don’t (yet) exist in Atlas
>> 
>>- added REST APIs to purge deleted entities
>> 
>>- performance improvements in lineage retrieval and tag-propagation
>> 
>>- updated Atlas server to process notifications from multiple Kafka
>> topics
>> 
>>- updated Hive hook to track process executions, via
>> hive_process_execution entities
>> 
>>- updated Hive hook to capture DDL operations, via hive_db_ddl and
>> hive_table_ddl entities
>> 
>>- added models for Spark; introduced new models for AWS S3
>> 
>>- updated versions of dependent libraries/components: JanusGraph,
>> Jackson parser, Spring Framework,
>> 
>>- updated authorization model to cover new features/APIs, like
>> add/remove labels, purge entities, update namespace attributes
>> 
>> 
>>