Madhan Neethiraj created ATLAS-4984:
---------------------------------------
Summary: Option to ignore spark_process attributes details and
sparkPlanDescription
Key: ATLAS-4984
URL: https://issues.apache.org/jira/browse/ATLAS-4984
Project: Atlas
Issue Type: Improvement
Components: spark-integration
Reporter: Madhan Neethiraj
{{spark_process}} entity attributes {{details}} and {{sparkPlanDescription}}
are populated with query plan details, which can contain a large amount of
text, often in megabytes. In environments that don't require these attributes,
it will help to ignore these attributes while processing create/update of
{{spark_process}} entities. I suggest Atlas server notification processor to
support following configurations:
-
{{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.sparkPlanDescription}}
-
{{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.details}}
A deployment can choose to ignore these attributes using above configurations
in Atlas server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
