Bolke de Bruin created ATLAS-3547:
-------------------------------------
Summary: Don’t evaluate propagating classifications for non
relevant update
Key: ATLAS-3547
URL: https://issues.apache.org/jira/browse/ATLAS-3547
Project: Atlas
Issue Type: Bug
Affects Versions: trunk
Reporter: Bolke de Bruin
In case of an aws_s3_bucket with a propagating classification and with several
thousands (200K+) objects we are observing that every addition of an object to
this aws_s3_bucket triggers a re-evaluation of all entities with the
propagating tag. This exponentially increases the time that it takes a message
to be consumed if the bucket is growing in size.
We send a create or update message for an aws_s3_object by Kafka which has all
relevant information inside the message (ie. Aws_s3_bucket, aws_s3_pseudodir,
aws_s3_object) as the producer system is unaware if the relevant metadata for
the bucket and pseudodir are already available. This triggers an evaluation of
the classifications and all propagating classifications. This seems over eager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
