Bolke de Bruin created ATLAS-3547: ------------------------------------- Summary: Don’t evaluate propagating classifications for non relevant update Key: ATLAS-3547 URL: https://issues.apache.org/jira/browse/ATLAS-3547 Project: Atlas Issue Type: Bug Affects Versions: trunk Reporter: Bolke de Bruin
In case of an aws_s3_bucket with a propagating classification and with several thousands (200K+) objects we are observing that every addition of an object to this aws_s3_bucket triggers a re-evaluation of all entities with the propagating tag. This exponentially increases the time that it takes a message to be consumed if the bucket is growing in size. We send a create or update message for an aws_s3_object by Kafka which has all relevant information inside the message (ie. Aws_s3_bucket, aws_s3_pseudodir, aws_s3_object) as the producer system is unaware if the relevant metadata for the bucket and pseudodir are already available. This triggers an evaluation of the classifications and all propagating classifications. This seems over eager. -- This message was sent by Atlassian Jira (v8.3.4#803005)