Hey Guys,

I was wondering if you have any best practices / limits for entity that is 
being created in atlas?
I am asking since we noticed some issues related to creating (or processing) 
entities.

For instance, we use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json.
It has following property:

"name":        "s3Objects",
"typeName":    "array<aws_s3_object>"

Now in AWS buckets you can have thousands of objects. This causes that 
s3Objects array grows quite quickly,
causing aws_s3_pseudo_dir entity Json to rich easly few MBs.

Then we start seeing problems like:

-          UI is dying on displaying entity properties or lineage

-          audit record too long: entityType=aws_s3_pseudo_dir, 
guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
entity attribute values not stored in audit (EntityAuditListenerV2:234)

-          some errors with write to HBase (java.lang.IllegalArgumentException: 
KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
param to 0)

-          kafka consumer errors (we can of course set some parameters on 
consumer, but I think it is just workaround)

…

Exception in NotificationHookConsumer (NotificationHookConsumer:332)

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
completed since the group has already rebalanced and assigned the partitions to 
another member. This means that the time between subsequen

t calls to poll() was longer than the configured max.poll.interval.ms, which 
typically implies that the poll loop is spending too much time message 
processing. You can address this either by increasing the sessio

n timeout or by reducing the maximum size of batches returned in poll() with 
max.poll.records.
…


We can of course skip creating of s3objects, but on the other hand it is quite 
usefull to have such info in pseudodirectory…


Thanks in advance for suggestions how to handle this…


Adam Rempter

Big Data Squad
[https://ispintranet.pl.ing.net/sites/default/files/content/logo_stopka2.png]
ul. Konduktorska 35, 40-155 Katowice
Poland


ING Business Shared Services B.V. z siedzibą w Amsterdamie, Holandia, VAT PL 
526-319-58-54, działająca w Polsce w formie oddziału, pod firmą ING Business 
Shared Services B.V. spółka z ograniczoną odpowiedzialnością Oddział w Polsce z 
siedzibą w Katowicach, ul. Konduktorska 35, 40-155 Katowice, NIP: 2050005130, 
wpisana do rejestru przedsiębiorców Krajowego Rejestru Sądowego prowadzonego 
przez Sąd Rejonowy Katowice-Wschód w Katowicach, VIII Wydział Gospodarczy 
Krajowego Rejestru Sądowego pod numerem KRS 0000702305.

Reply via email to