[jira] [Assigned] (ATLAS-4389) Best practice or a way to bring in large number of entities on a regular basis.
[ https://issues.apache.org/jira/browse/ATLAS-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry reassigned ATLAS-4389: -- Assignee: Ashutosh Mestry > Best practice or a way to bring in large number of entities on a regular > basis. > --- > > Key: ATLAS-4389 > URL: https://issues.apache.org/jira/browse/ATLAS-4389 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, 2.1.0 >Reporter: Saad >Assignee: Ashutosh Mestry >Priority: Major > Labels: documentation, newbie, performance > Attachments: image-2021-08-05-11-22-29-259.png, > image-2021-08-05-11-23-05-440.png > > > Would you be so kind to let us know if there is any best practice or a way to > bring in large number of entities on a regular basis. > *Our use case:* > We will be bringing in around 12,000 datasets, 12,000 jobs and 70,000 > columns. We want to do this as part of our deployment pipeline for other > upstream projects. > At every deploy we want to do the following: > - Add the jobs, datasets and columns that are not in Atlas > - Update the jobs, datasets and columns that are in Atlas > - Delete the jobs from Atlas that are deleted from the upstream systems. > So far we have considered using the bulk API endpoint(/v2/entity/bulk). This > has its own issues. We found that if the payload is too big in our case > bigger than 300-500 entities this times out. The more deeper the > relationships the fewer the entities you can send through the bulk endpoint. > Inspecting some of the code we feel that both REST and streaming data through > Kafka follow the same codepath and finally yield the same performance. > Further we found that when creating entities the type registry becomes the > bottle neck. We discovered this by profiling the jvm. We found that only one > core processes the the entities and their relationships. > *Questions:* > 1- What is the best practice when bulk loading lots on entities in a > reasonable time. We are aiming to load 12k jobs, 12k datasets and 70k columns > in less than 10 mins.? > 2- Where should we start if we want to scale the API, is there any known way > to horizontally scale Atlas? > Here are some of the stats for the load testing we did, > > !image-2021-08-05-11-23-05-440.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
Apache atlas hook hive
I want to hook Hive for importing metadata automatically in Apache atlas that
is installed standalone. Instruction on apache atlas website says "untar
apache-atlas-${project.version}-hive-hook.tar.gz" and I can't find this file.
I'v tried to build apache-atlas-sources and I ran into
DependencyResolutionException error in "Apache atlas UI" step. Is there a mvn
command which build just apache-atlas-${project.version}-hive-hook.tar.gz so I
can finish the instructions? or is there a solution to
DependencyResolutionException problem?
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
+1 for Apache Atlas 2.2.0 rc1
Validated the following:
* Downloaded source and validated MD5 and SHA512 hash
* Validated signature from release manager ("Sidharth Mishra <
sidmis...@apache.org>")
* Verified source doesn't contain any binaries
* No errors in application log startup.
* Build the source successfully using embedded-hbase-solr profile
* Started Atlas and ran quick start to create sample types and entities
* Validated metrics UI works with all entity types and count
* Validated Quick search, Basic search and DSL search, Glossary
* Admin Page - TypeSystem UI, Admin audits
* Tag propagation use-cases
Thanks,
Sarath
On Mon, Aug 9, 2021 at 3:55 PM Sidharth Mishra wrote:
> Atlas team,
>
> Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> available for vote within the dev community.
>
> ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> ATLAS-4391 - NoSuchFileException for {atlas.log.dir}/atlas-metrics.out
> ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> Timestamp is < 'Entity
> Created' Timestamp
> ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> entity type on Entity detail
> page
> ATLAS-4386 - Relationship search: Sorting does not work when
> relationship label is passed
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes
> are reached.
>
> [ ] +1 Approve
>
> [ ] +0 No opinion
>
> [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Sid
>
> List of all issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.2.0-rc1
>
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
>
> Source release verification:
>
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
>
> SHA512 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
>
> MD5 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
>
>
> Keys to verify the signature of the release artifacts are available
> at: https://dist.apache.org/repos/dist/release/atlas/KEYS
>
>
> New features/enhancements:
>
> - Introduced Deferred Action - Tag propagation will be handled as
> a background process (AtlasTask)
>
> - Added Re-indexing as JAVA_PATCH
>
> - Created JavaPatch to add new super types to existing entities
>
> - Supported for Business Metadata in Atlas Export API
>
> - Supported HA for admin/task API
>
> - Enhanced export service to export Terms
>
> - Added new models and addition attributes to Atlas for Apache
> Flink, GCP and ADLS Gen 2
>
> - Added support to Configure HiveServer2 Hook to send Lineage-only
> Messages
>
> - Improved and enhanced Bulk Glossary Import supporting import
> with relations
>
> - Dependency between Atlas and Kafka Zookeeper removed
>
> - Improved performance of GraphHelper's guid and status getter methods
>
> - Added new privilege - "admin-audits" for Atlas Admin Audits
> authorization
>
> - Basic search Improvements - search text with special characters,
> optimize pagination
>
> - Updated free-text search processor to support Elasticsearch
>
> - Refactored and enhanced Atlas Python Client supporting Python 2.7
>
> - DSL Search enhancements - Support glossary terms and
> relationship, can handle null attribute values, Java Tinkerpop
> Traversals instead of GremlinScriptEngine, support search by
> classification and its attribute, Caching mechanism for translated dsl
> queries
>
> - Enhanced Export/Import Service to conditionally Support
> Simultaneous Operations
>
> - Enhanced NotificationHookConsumer for Large Message Processing
>
> - Enhanced Atlas authorization for Add/Update/Remove
> classification on entities
>
> - Provided option to add mandatory attribute to existing entity
> definition
>
> - Fixed Atlas UI to load faster in case of large number of
> classifications & entities
>
> - Updated versions of dependent libraries/components: JanusGraph,
> elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
> gremlin, Solr, groovy, netty, Kafka
>
[GitHub] [atlas] mehtaanshul edited a comment on pull request #142: ATLAS-4358 move addition of default java patches to applyAll method
mehtaanshul edited a comment on pull request #142: URL: https://github.com/apache/atlas/pull/142#issuecomment-896224325 Hey @nixonrodrigues, can you please look into this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@atlas.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [atlas] mehtaanshul commented on pull request #142: ATLAS-4358 move addition of default java patches to applyAll method
mehtaanshul commented on pull request #142: URL: https://github.com/apache/atlas/pull/142#issuecomment-896224325 @nixonrodrigues -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@atlas.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
+1 for Apache Atlas 2.2.0 rc1
This release has an impressive list of features/enhancements/fixes. In addition
to list in Sid's email below, UI updates like type-system browser, enum editor
will make it a lot easier to work with Atlas type system.
One Atlas to carry the weight of metadata and governance needs of enterprises!
Go Atlas!
Thanks,
Madhan
Performed the following to validate release candidate 1:
1. successfully built Atlas using docker, for profile external-hbase-solr
(instructions in dev-support/atlas-docker/README.txt)
2. successfully brought up Atlas in docker containers
3. successfully ran quick_start
4. search: performed basic search, advanced search, quick search
5. classification: performed add, associate, propagate, update, disassociate,
delete
6. business-metadata: performed import, add/update/remove on entities, search
7. glossary: performed import, add/remove on entities, searched
8. verified basic operations on HBase hook (create table, drop table)
9. verified basic operations on Hive hook (create database/table/view,
lineage, alter table, drop table/database)
10. verified entity audit entries for above operations
On 8/9/21, 3:55 PM, "Sidharth Mishra" wrote:
Atlas team,
Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
available for vote within the dev community.
ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
ATLAS-4391 - NoSuchFileException for {atlas.log.dir}/atlas-metrics.out
ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
Timestamp is < 'Entity
Created' Timestamp
ATLAS-4387 - UI: Propagated Classifications shows wrong count for
entity type on Entity detail
page
ATLAS-4386 - Relationship search: Sorting does not work when
relationship label is passed
Links to the release artifacts are given below. Please review and vote.
The vote will be open for at least 72 hours or until necessary votes
are reached.
[ ] +1 Approve
[ ] +0 No opinion
[ ] -1 Disapprove (and reason why)
Thanks,
Sid
List of all issues addressed in this release:
https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
Git tag for the release:
https://github.com/apache/atlas/tree/release-2.2.0-rc1
Sources for the release:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
Source release verification:
PGP Signature:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
SHA512 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
MD5 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
Keys to verify the signature of the release artifacts are available
at: https://dist.apache.org/repos/dist/release/atlas/KEYS
New features/enhancements:
- Introduced Deferred Action - Tag propagation will be handled as
a background process (AtlasTask)
- Added Re-indexing as JAVA_PATCH
- Created JavaPatch to add new super types to existing entities
- Supported for Business Metadata in Atlas Export API
- Supported HA for admin/task API
- Enhanced export service to export Terms
- Added new models and addition attributes to Atlas for Apache
Flink, GCP and ADLS Gen 2
- Added support to Configure HiveServer2 Hook to send Lineage-only
Messages
- Improved and enhanced Bulk Glossary Import supporting import
with relations
- Dependency between Atlas and Kafka Zookeeper removed
- Improved performance of GraphHelper's guid and status getter methods
- Added new privilege - "admin-audits" for Atlas Admin Audits
authorization
- Basic search Improvements - search text with special characters,
optimize pagination
- Updated free-text search processor to support Elasticsearch
- Refactored and enhanced Atlas Python Client supporting Python 2.7
- DSL Search enhancements - Support glossary terms and
relationship, can handle null attribute values, Java Tinkerpop
Traversals instead of GremlinScriptEngine, support search by
classification and its attribute, Caching mechanism for translated dsl
queries
- Enhanced Export/Import Service to conditionally Support
Simultaneous Operations
- Enhanced NotificationHookConsumer for Large Message Processing
- Enhanced Atlas authorization for Add/Update/Remove
classification on entities
- Provided option to add mandatory attribute to existing entity
definition
- Fixed Atlas UI to load faster in case of large number of
classifications &
[jira] [Resolved] (ATLAS-4393) Sample python app erroring out during discovery quick search
[ https://issues.apache.org/jira/browse/ATLAS-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Louis Christopher resolved ATLAS-4393. -- Resolution: Fixed > Sample python app erroring out during discovery quick search > > > Key: ATLAS-4393 > URL: https://issues.apache.org/jira/browse/ATLAS-4393 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Louis Christopher >Priority: Major > Fix For: 2.1.0 > > > The error occurs while running > `atlas/atlas-examples/sample-app/src/main/python/sample_client.py` > The error occurs at `DiscoveryExample.quick_search` section. > This is the URL which seems correct > `http://localhost:21000/api/atlas/v2/search/quick?query=sample_table=sample_table=False` > The basic search however works fine > `http://localhost:21000/api/atlas/v2/search/basic?query=sample_table=sample_table=False` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4393) Sample python app erroring out during discovery quick search
[ https://issues.apache.org/jira/browse/ATLAS-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Louis Christopher updated ATLAS-4393: - Fix Version/s: 2.1.0 Language: Python (was: Java Python) Affects Version/s: (was: 2.1.0) 2.0.0 > Sample python app erroring out during discovery quick search > > > Key: ATLAS-4393 > URL: https://issues.apache.org/jira/browse/ATLAS-4393 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Louis Christopher >Priority: Major > Fix For: 2.1.0 > > > The error occurs while running > `atlas/atlas-examples/sample-app/src/main/python/sample_client.py` > The error occurs at `DiscoveryExample.quick_search` section. > This is the URL which seems correct > `http://localhost:21000/api/atlas/v2/search/quick?query=sample_table=sample_table=False` > The basic search however works fine > `http://localhost:21000/api/atlas/v2/search/basic?query=sample_table=sample_table=False` -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
Thanks Sid for putting this together for vote.
+1 for 2.2.0 - rc1 release.
* Verified md5, sha512 hash, release Signature
* MVN Build was clean
* Setup Atlas server in embedded HBase and Solr.
* Ran quick start.py
* Atlas UI loading cleanly.
* Created entities using Atlas UI
* Verified metric API, statistics modal.
Regards
Nixon Rodrigues
On Tue, Aug 10, 2021 at 4:04 PM Nikhil Bonte <
nikhil.bo...@freestoneinfotech.com> wrote:
> Hi Sidharth
>
> +1 for Apache Atlas v2.2.0 rc1.
>
> Verified following:
> * Verified md5, sha512, Signature
> * Clean build embedded-hbase-solr profile.
> * Setup Atlas server in embedded HBase and Solr.
> * Verified re-indexing with JAVA_PATCH
> * Verified Import zip with REST API
> * Ran quick start
> * Created entities using Atlas UI
> * Verified metric API, statistics modal.
>
>
> Regards
> Nikhil P. Bonte
>
>
> On Tue, Aug 10, 2021 at 4:25 AM Sidharth Mishra
> wrote:
>
> > Atlas team,
> >
> > Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> > available for vote within the dev community.
> >
> > ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> > ATLAS-4391 - NoSuchFileException for
> {atlas.log.dir}/atlas-metrics.out
> > ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> > Timestamp is < 'Entity
> > Created' Timestamp
> > ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> > entity type on Entity detail
> > page
> > ATLAS-4386 - Relationship search: Sorting does not work when
> > relationship label is passed
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes
> > are reached.
> >
> > [ ] +1 Approve
> >
> > [ ] +0 No opinion
> >
> > [ ] -1 Disapprove (and reason why)
> >
> > Thanks,
> > Sid
> >
> > List of all issues addressed in this release:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/tree/release-2.2.0-rc1
> >
> > Sources for the release:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
> >
> > Source release verification:
> >
> > PGP Signature:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
> >
> > SHA512 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
> >
> > MD5 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
> >
> >
> > Keys to verify the signature of the release artifacts are available
> > at: https://dist.apache.org/repos/dist/release/atlas/KEYS
> >
> >
> > New features/enhancements:
> >
> > - Introduced Deferred Action - Tag propagation will be handled as
> > a background process (AtlasTask)
> >
> > - Added Re-indexing as JAVA_PATCH
> >
> > - Created JavaPatch to add new super types to existing entities
> >
> > - Supported for Business Metadata in Atlas Export API
> >
> > - Supported HA for admin/task API
> >
> > - Enhanced export service to export Terms
> >
> > - Added new models and addition attributes to Atlas for Apache
> > Flink, GCP and ADLS Gen 2
> >
> > - Added support to Configure HiveServer2 Hook to send Lineage-only
> > Messages
> >
> > - Improved and enhanced Bulk Glossary Import supporting import
> > with relations
> >
> > - Dependency between Atlas and Kafka Zookeeper removed
> >
> > - Improved performance of GraphHelper's guid and status getter
> methods
> >
> > - Added new privilege - "admin-audits" for Atlas Admin Audits
> > authorization
> >
> > - Basic search Improvements - search text with special characters,
> > optimize pagination
> >
> > - Updated free-text search processor to support Elasticsearch
> >
> > - Refactored and enhanced Atlas Python Client supporting Python 2.7
> >
> > - DSL Search enhancements - Support glossary terms and
> > relationship, can handle null attribute values, Java Tinkerpop
> > Traversals instead of GremlinScriptEngine, support search by
> > classification and its attribute, Caching mechanism for translated dsl
> > queries
> >
> > - Enhanced Export/Import Service to conditionally Support
> > Simultaneous Operations
> >
> > - Enhanced NotificationHookConsumer for Large Message Processing
> >
> > - Enhanced Atlas authorization for Add/Update/Remove
> > classification on entities
> >
> > - Provided option to add mandatory attribute to existing entity
> > definition
> >
> > - Fixed Atlas UI to load faster in case of large number of
> > classifications & entities
> >
> > - Updated versions of dependent libraries/components: JanusGraph,
> > elasticsearch,
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
Thanks Sid for putting this together for vote. +1 for 2.2.0 - rc1 release. * Verified md5, sha512 hash, release Signature * MVN Build was clean * Setup Atlas server in embedded HBase and Solr. * Ran quick start.py * Atlas UI loading cleanly. * Created entities using Atlas UI * Verified metric API, statistics modal. Regards Nixon Rodrigues
[jira] [Updated] (ATLAS-4393) Sample python app erroring out during discovery quick search
[ https://issues.apache.org/jira/browse/ATLAS-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Louis Christopher updated ATLAS-4393: - Component/s: (was: atlas-core) Affects Version/s: (was: trunk) > Sample python app erroring out during discovery quick search > > > Key: ATLAS-4393 > URL: https://issues.apache.org/jira/browse/ATLAS-4393 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Affects Versions: 2.1.0 > Environment: Linux >Reporter: Louis Christopher >Priority: Major > > The error occurs while running > `atlas/atlas-examples/sample-app/src/main/python/sample_client.py` > The error occurs at `DiscoveryExample.quick_search` section. > This is the URL which seems correct > `http://localhost:21000/api/atlas/v2/search/quick?query=sample_table=sample_table=False` > The basic search however works fine > `http://localhost:21000/api/atlas/v2/search/basic?query=sample_table=sample_table=False` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4393) Sample python app erroring out during discovery quick search
[ https://issues.apache.org/jira/browse/ATLAS-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Louis Christopher updated ATLAS-4393: - Summary: Sample python app erroring out during discovery quick search (was: Sample python app not erroring out during discovery quick search) > Sample python app erroring out during discovery quick search > > > Key: ATLAS-4393 > URL: https://issues.apache.org/jira/browse/ATLAS-4393 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-intg >Affects Versions: trunk, 2.1.0 > Environment: Linux >Reporter: Louis Christopher >Priority: Major > > The error occurs while running > `atlas/atlas-examples/sample-app/src/main/python/sample_client.py` > The error occurs at `DiscoveryExample.quick_search` section. > This is the URL which seems correct > `http://localhost:21000/api/atlas/v2/search/quick?query=sample_table=sample_table=False` > The basic search however works fine > `http://localhost:21000/api/atlas/v2/search/basic?query=sample_table=sample_table=False` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4393) Sample python app not erroring out during discovery quick search
Louis Christopher created ATLAS-4393: Summary: Sample python app not erroring out during discovery quick search Key: ATLAS-4393 URL: https://issues.apache.org/jira/browse/ATLAS-4393 Project: Atlas Issue Type: Bug Components: atlas-core, atlas-intg Affects Versions: 2.1.0, trunk Environment: Linux Reporter: Louis Christopher The error occurs while running `atlas/atlas-examples/sample-app/src/main/python/sample_client.py` The error occurs at `DiscoveryExample.quick_search` section. This is the URL which seems correct `http://localhost:21000/api/atlas/v2/search/quick?query=sample_table=sample_table=False` The basic search however works fine `http://localhost:21000/api/atlas/v2/search/basic?query=sample_table=sample_table=False` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3694) [Business Metadata] Max length defined for attributes of string type is not getting enforced
[
https://issues.apache.org/jira/browse/ATLAS-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mandar Ambawane resolved ATLAS-3694.
Resolution: Fixed
> [Business Metadata] Max length defined for attributes of string type is not
> getting enforced
>
>
> Key: ATLAS-3694
> URL: https://issues.apache.org/jira/browse/ATLAS-3694
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-webui
>Reporter: Umesh Padashetty
>Priority: Major
> Attachments: Screenshot 2020-03-26 at 7.49.51 PM.png, Screenshot
> 2020-03-26 at 7.51.13 PM.png
>
>
> Even when the max length applicable to an attribute is defined as 10, I was
> able to add all the alphabets of an english language i.e. a-z, 26 in length,
> as the attribute value.
> This is reproducible both from API as well as UI.
> {code:java}
> curl -X POST \
>
> 'https://:31443/api/atlas/v2/entity/guid/ab29815d-6d20-44ee-b426-8fba946b72a5/businessmetata?isOverwrite=true'
> \
> -H 'cache-control: no-cache' \
> -H 'content-type: application/json' \
> -H 'postman-token: 6c5a0476-ac64-57c9-5845-cbea945fdf99' \
> -H 'x-xsrf-header: \"\"' \
> -d '{"String Length":{"Name":"abcdefghijklmnopqrstuvwxyz"}}' {code}
> Screenshots attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ATLAS-3651) Metrics API: Count of entities for a type doesn't include entities of sub type
[ https://issues.apache.org/jira/browse/ATLAS-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Jain closed ATLAS-3651. -- > Metrics API: Count of entities for a type doesn't include entities of sub type > -- > > Key: ATLAS-3651 > URL: https://issues.apache.org/jira/browse/ATLAS-3651 > Project: Atlas > Issue Type: Bug >Reporter: Mayank Jain >Assignee: Mayank Jain >Priority: Major > Attachments: ATLAS-3651-Metrics-API-Count-of-entities-for-a-type-and > subType updation-8.patch > > > DataSet count shown in Basic/DSL Search doesn't include the count of entities > of its subtypes. If an entity is created as a DataSet then the count is > updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
Hi Sidharth
+1 for Apache Atlas v2.2.0 rc1.
Verified following:
* Verified md5, sha512, Signature
* Clean build embedded-hbase-solr profile.
* Setup Atlas server in embedded HBase and Solr.
* Verified re-indexing with JAVA_PATCH
* Verified Import zip with REST API
* Ran quick start
* Created entities using Atlas UI
* Verified metric API, statistics modal.
Regards
Nikhil P. Bonte
On Tue, Aug 10, 2021 at 4:25 AM Sidharth Mishra
wrote:
> Atlas team,
>
> Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> available for vote within the dev community.
>
> ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> ATLAS-4391 - NoSuchFileException for {atlas.log.dir}/atlas-metrics.out
> ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> Timestamp is < 'Entity
> Created' Timestamp
> ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> entity type on Entity detail
> page
> ATLAS-4386 - Relationship search: Sorting does not work when
> relationship label is passed
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes
> are reached.
>
> [ ] +1 Approve
>
> [ ] +0 No opinion
>
> [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Sid
>
> List of all issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.2.0-rc1
>
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
>
> Source release verification:
>
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
>
> SHA512 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
>
> MD5 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
>
>
> Keys to verify the signature of the release artifacts are available
> at: https://dist.apache.org/repos/dist/release/atlas/KEYS
>
>
> New features/enhancements:
>
> - Introduced Deferred Action - Tag propagation will be handled as
> a background process (AtlasTask)
>
> - Added Re-indexing as JAVA_PATCH
>
> - Created JavaPatch to add new super types to existing entities
>
> - Supported for Business Metadata in Atlas Export API
>
> - Supported HA for admin/task API
>
> - Enhanced export service to export Terms
>
> - Added new models and addition attributes to Atlas for Apache
> Flink, GCP and ADLS Gen 2
>
> - Added support to Configure HiveServer2 Hook to send Lineage-only
> Messages
>
> - Improved and enhanced Bulk Glossary Import supporting import
> with relations
>
> - Dependency between Atlas and Kafka Zookeeper removed
>
> - Improved performance of GraphHelper's guid and status getter methods
>
> - Added new privilege - "admin-audits" for Atlas Admin Audits
> authorization
>
> - Basic search Improvements - search text with special characters,
> optimize pagination
>
> - Updated free-text search processor to support Elasticsearch
>
> - Refactored and enhanced Atlas Python Client supporting Python 2.7
>
> - DSL Search enhancements - Support glossary terms and
> relationship, can handle null attribute values, Java Tinkerpop
> Traversals instead of GremlinScriptEngine, support search by
> classification and its attribute, Caching mechanism for translated dsl
> queries
>
> - Enhanced Export/Import Service to conditionally Support
> Simultaneous Operations
>
> - Enhanced NotificationHookConsumer for Large Message Processing
>
> - Enhanced Atlas authorization for Add/Update/Remove
> classification on entities
>
> - Provided option to add mandatory attribute to existing entity
> definition
>
> - Fixed Atlas UI to load faster in case of large number of
> classifications & entities
>
> - Updated versions of dependent libraries/components: JanusGraph,
> elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
> gremlin, Solr, groovy, netty, Kafka
>
[jira] [Closed] (ATLAS-4296) [Atlas: Debug Metrics] Min Time , Max Time and Average Time in UI are not matching the api response values
[
https://issues.apache.org/jira/browse/ATLAS-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Jain closed ATLAS-4296.
--
> [Atlas: Debug Metrics] Min Time , Max Time and Average Time in UI are not
> matching the api response values
> --
>
> Key: ATLAS-4296
> URL: https://issues.apache.org/jira/browse/ATLAS-4296
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Mayank Jain
>Priority: Major
> Attachments:
> 0001-ATLAS-UI-ATLAS-4296-ATLAS-4298-ATLAS-4299-Debug-Metr.patch,
> ATLAS-4296-1.patch, Screenshot 2021-05-21 at 4.18.34 PM.png, Screenshot
> 2021-05-21 at 4.18.34 PM.png
>
>
> Scenario: Enable debug Metrics
> !Screenshot 2021-05-21 at 4.18.34 PM.png|width=646,height=322!
> Eg: Consider the above highlighted example. *GlossaryREST_deleteGlossary*
> Here
> Min Time : 1.920 (seconds)
> Max Time : 5.530 (seconds) and
> Average Time : 5.530 (seconds)
> *Api response:*
> {code:java}
> "GlossaryREST_deleteGlossary": {
> "name": "GlossaryREST_deleteGlossary",
> "numops": 6,
> "minTime": 192,
> "maxTime": 553,
> "stdDevTime": 21.556128,
> "avgTime": 532
> } {code}
>
> 1.92 (seconds) has to be 1920 if the value is response is stored in
> milliseconds but it appears as 192
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ATLAS-4391) NoSuchFileException for ${atlas.log.dir}/atlas-metrics.out
[
https://issues.apache.org/jira/browse/ATLAS-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Jain closed ATLAS-4391.
--
> NoSuchFileException for ${atlas.log.dir}/atlas-metrics.out
> ---
>
> Key: ATLAS-4391
> URL: https://issues.apache.org/jira/browse/ATLAS-4391
> Project: Atlas
> Issue Type: Bug
>Reporter: Sidharth Kumar Mishra
>Assignee: Mayank Jain
>Priority: Critical
> Attachments: ATLAS-4391-1.patch
>
>
> Atlas startup logs NoSuchFileException for ${atlas.log.dir}/atlas-metrics.out:
> 2021-08-05 07:54:07,353 WARN - [main:] ~ Error creating sink
> 'atlas-debug-metrics-context' (MetricsSystemImpl:508)
> org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating
> plugin: org.apache.hadoop.metrics2.sink.FileSink
> at
> org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:210)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:531)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:503)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:479)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:62)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:58)
> at
> org.apache.atlas.web.service.DebugMetricsWrapper.init(DebugMetricsWrapper.java:43)
> ...
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
> org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:110)
> at org.apache.atlas.Atlas.main(Atlas.java:133)
> Caused by: org.apache.hadoop.metrics2.MetricsException: Error creating
> ${atlas.log.dir}/atlas-metrics.out
> at org.apache.hadoop.metrics2.sink.FileSink.init(FileSink.java:53)
> at
> org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:207)
> ... 63 more
> Caused by: java.nio.file.NoSuchFileException:
> ${atlas.log.dir}/atlas-metrics.out
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at
> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
> at java.nio.file.Files.newOutputStream(Files.java:216)
> at org.apache.hadoop.metrics2.sink.FileSink.init(FileSink.java:50)
> ... 64 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Review Request 73509: ATLAS-4392 The displayed names in a relationship should be sorted
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73509/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4392 https://issues.apache.org/jira/browse/ATLAS-4392 Repository: atlas Description --- When we have a 1:N relationship then we able to see the list of connected entities in the Atlas UI. These entities don't seem to be sorted in a meaningful way. The correct way would be if they are sorted in alphabetical order. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 794847555 Diff: https://reviews.apache.org/r/73509/diff/1/ Testing --- PreCommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/796/consoleFull Thanks, Mandar Ambawane
Re: Review Request 73508: ATLAS-4233 [Atlas: Audits] When a term is added or removed, v1 api displays action as null instead of TERM_ADD or TERM_DELETE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73508/ --- (Updated Aug. 10, 2021, 6:09 a.m.) Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and Pinal Shah. Bugs: ATLAS-4233 https://issues.apache.org/jira/browse/ATLAS-4233 Repository: atlas Description --- When we add or remove a term to an entity,in audits the action should appear as TERM_ADD or TERM_DELETE But instead the value is giving null. Diffs - repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 5c08874ff Diff: https://reviews.apache.org/r/73508/diff/1/ Testing --- PC - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/792/console Thanks, Shraddha Pardeshi
