[jira] [Updated] (ATLAS-2977) Documentation does not tell you about scaling the atlas
[ https://issues.apache.org/jira/browse/ATLAS-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] abhay kumar updated ATLAS-2977: --- Priority: Critical (was: Major) > Documentation does not tell you about scaling the atlas > > > Key: ATLAS-2977 > URL: https://issues.apache.org/jira/browse/ATLAS-2977 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.1.0 >Reporter: abhay kumar >Priority: Critical > > There is no info provided in atlas document on how to scale it. > Apache atlas is connected to cassandra or hbase in the backend which can > scale out ,but I dont know how apache atlas engine ( rest web-service and > request processor ) can scale out. > I can install multiple instances of it on different machine and have load > balancer in front of it to fan out the request. But would this model help ? > Does it do any kind of locking and do db transaction, so that this model > would not work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2926) ZipSink: Very Large Entities Cause Out Of Memory Exception
[
https://issues.apache.org/jira/browse/ATLAS-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated ATLAS-2926:
Fix Version/s: (was: trunk)
2.0.0
1.2.0
> ZipSink: Very Large Entities Cause Out Of Memory Exception
> --
>
> Key: ATLAS-2926
> URL: https://issues.apache.org/jira/browse/ATLAS-2926
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 0.8.2, 0.8.3, trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: 0.8.3, 1.2.0, 2.0.0
>
> Attachments: ATLAS-2926-ZipSink-OOM.patch
>
>
> *Steps to Duplicate*
> # Setup Atlas with very large data. Atlas one entity should be about 300 MB
> in size.
> # Perform export with parameters that will include the large entity.
> Following error will be encountered:
> {code:java}
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit at
> java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) at
> java.lang.StringCoding.encode(StringCoding.java:344) at
> java.lang.StringCoding.encode(StringCoding.java:387) at
> java.lang.String.getBytes(String.java:958) at
> org.apache.atlas.repository.impexp.ZipSink.addToZipStream(ZipSink.java:106)
> at org.apache.atlas.repository.impexp.ZipSink.saveToZip(ZipSink.java:95)
> at org.apache.atlas.repository.impexp.ZipSink.add(ZipSink.java:55) at
> org.apache.atlas.repository.impexp.ExportService.addEntity(ExportService.java:467)
> {code}
> *Additional Information*
> During conversion of entity string to bytes, a known JDK error is encountered.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ATLAS-2948) Cache implementations should be consistent and configurable
[ https://issues.apache.org/jira/browse/ATLAS-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698387#comment-16698387 ] Franco commented on ATLAS-2948: --- *Advantages:* 1) Having a pluggable implementation means you can have a NoopCache for performance and other types of testing. 2) Ensures that you always have consistent, best practice cache behaviour e.g. concurrency. 3) Allows you to swap out a cache implementation with something faster or better e.g. an off heap cache. *Use cases:* 1) Horizontal scalability. It would be useful to be able to deploy Atlas in an auto scaling group but we can't because it is stateful. Being able to point the caches at say Redis would allow it to horizontally scale with no consistency issues. 2) Recovery. During recovery scenarios we want the cache to be warm when we switch between instances. Having something like Redis, Memcache etc would allow for that. 3) Large Number of Typedefs. We want to have a very large number of typedefs which would cause instability due to GC as everything would be on-heap. I am more than happy to submit a patch for review. Just wanted to make sure the core team wasn't against it. > Cache implementations should be consistent and configurable > --- > > Key: ATLAS-2948 > URL: https://issues.apache.org/jira/browse/ATLAS-2948 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Franco >Priority: Major > > Throughout Apache Atlas caches are hardcoded. For example: > (1) AtlasTypeRegistry has a TypeDef cache which is a ConcurrentHashMap > (2) RequestContext has an entity cache which is a Map > > Instead there should be a standardised Cache implementation across the board > that is pluggable so that users can try a different approach e.g. use Redis > to help with Atlas outages. Or maybe something like Ignite to allow for Atlas > to be clustered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2948) Cache implementations should be consistent and configurable
[ https://issues.apache.org/jira/browse/ATLAS-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698380#comment-16698380 ] Madhan Neethiraj commented on ATLAS-2948: - [~nadenf] - its not clear what the advantages (or use cases) are for a pluggable caching of TypeDef cache and RequestContext cache. Can you please add details? Also, if you have a patch to address this, please create a review request at https://reviews.apache.org. > Cache implementations should be consistent and configurable > --- > > Key: ATLAS-2948 > URL: https://issues.apache.org/jira/browse/ATLAS-2948 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Franco >Priority: Major > > Throughout Apache Atlas caches are hardcoded. For example: > (1) AtlasTypeRegistry has a TypeDef cache which is a ConcurrentHashMap > (2) RequestContext has an entity cache which is a Map > > Instead there should be a standardised Cache implementation across the board > that is pluggable so that users can try a different approach e.g. use Redis > to help with Atlas outages. Or maybe something like Ignite to allow for Atlas > to be clustered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2980) fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string
[ https://issues.apache.org/jira/browse/ATLAS-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698374#comment-16698374 ] ASF subversion and git services commented on ATLAS-2980: Commit d88771e10a95654c896b837dddb2bdc1a9ac5cee in atlas's branch refs/heads/branch-1.0 from Sizhong Du [ https://git-wip-us.apache.org/repos/asf?p=atlas.git;h=d88771e ] ATLAS-2980: fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string Signed-off-by: Madhan Neethiraj (cherry picked from commit 618cf361201828faba3b7cf1f47dad2e348db634) > fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string > > > Key: ATLAS-2980 > URL: https://issues.apache.org/jira/browse/ATLAS-2980 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.0.0, 1.1.0 >Reporter: Madhan Neethiraj >Priority: Major > Fix For: 1.2.0, 2.0.0 > > Attachments: ATLAS-2980.patch > > > HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() > value in audit record field 'definition'. This should be fixed to store JSON > string of the entity, since the listEventsV2() API interprets the > 'definition' field value as JSON to convert into AtlasEntity. > Issue was reported and patch provided via pull request > https://github.com/apache/atlas/pull/19/commits, by TomDu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2980) fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string
[ https://issues.apache.org/jira/browse/ATLAS-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698373#comment-16698373 ] ASF subversion and git services commented on ATLAS-2980: Commit 618cf361201828faba3b7cf1f47dad2e348db634 in atlas's branch refs/heads/master from Sizhong Du [ https://git-wip-us.apache.org/repos/asf?p=atlas.git;h=618cf36 ] ATLAS-2980: fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string Signed-off-by: Madhan Neethiraj > fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string > > > Key: ATLAS-2980 > URL: https://issues.apache.org/jira/browse/ATLAS-2980 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.0.0, 1.1.0 >Reporter: Madhan Neethiraj >Priority: Major > Fix For: 1.2.0, 2.0.0 > > Attachments: ATLAS-2980.patch > > > HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() > value in audit record field 'definition'. This should be fixed to store JSON > string of the entity, since the listEventsV2() API interprets the > 'definition' field value as JSON to convert into AtlasEntity. > Issue was reported and patch provided via pull request > https://github.com/apache/atlas/pull/19/commits, by TomDu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2980) fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string
[ https://issues.apache.org/jira/browse/ATLAS-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-2980: Attachment: ATLAS-2980.patch > fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string > > > Key: ATLAS-2980 > URL: https://issues.apache.org/jira/browse/ATLAS-2980 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.0.0, 1.1.0 >Reporter: Madhan Neethiraj >Priority: Major > Attachments: ATLAS-2980.patch > > > HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() > value in audit record field 'definition'. This should be fixed to store JSON > string of the entity, since the listEventsV2() API interprets the > 'definition' field value as JSON to convert into AtlasEntity. > Issue was reported and patch provided via pull request > https://github.com/apache/atlas/pull/19/commits, by TomDu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2980) fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string
[ https://issues.apache.org/jira/browse/ATLAS-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-2980: Description: HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() value in audit record field 'definition'. This should be fixed to store JSON string of the entity, since the listEventsV2() API interprets the 'definition' field value as JSON to convert into AtlasEntity. Issue was reported and patch provided via pull request https://github.com/apache/atlas/pull/19/commits, by TomDu. was:HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() value in audit record field 'definition'. This should be fixed to store JSON string of the entity, since the listEventsV2() API interprets the 'definition' field value as JSON to convert into AtlasEntity. > fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string > > > Key: ATLAS-2980 > URL: https://issues.apache.org/jira/browse/ATLAS-2980 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.0.0, 1.1.0 >Reporter: Madhan Neethiraj >Priority: Major > Attachments: ATLAS-2980.patch > > > HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() > value in audit record field 'definition'. This should be fixed to store JSON > string of the entity, since the listEventsV2() API interprets the > 'definition' field value as JSON to convert into AtlasEntity. > Issue was reported and patch provided via pull request > https://github.com/apache/atlas/pull/19/commits, by TomDu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-2980) fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string
Madhan Neethiraj created ATLAS-2980: --- Summary: fix HBaseBasedAuditRepository.putEventsV2() to store entity-json string Key: ATLAS-2980 URL: https://issues.apache.org/jira/browse/ATLAS-2980 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 1.1.0, 1.0.0 Reporter: Madhan Neethiraj HBaseBasedAuditRepository.putEventsV2() currently stores entity.toString() value in audit record field 'definition'. This should be fixed to store JSON string of the entity, since the listEventsV2() API interprets the 'definition' field value as JSON to convert into AtlasEntity. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
