vladhlinsky commented on issue #91: ATLAS-3655: Create 'spark_application' type to avoid 'spark_process' from being updated for multiple operations URL: https://github.com/apache/atlas/pull/91#issuecomment-595413075 Attaching screenshots. - Installed and started Atlas. - Executed the next statements using spark-shell: ``` spark.sql("create table table_1_17(col1 int,col2 string)"); spark.sql("create table table_2_17 as select * from table_1_17"); spark.sql("create table table_3_17(col1 int,col2 string)"); spark.sql("create table table_4_17 as select * from table_3_17"); ``` - Verified that all 4 entites are connected in Atlas lineage.  - `1100-spark_model.json` is updated with proposed changes. - Once again executed similar statements: ``` spark.sql("create table table_1_37(col1 int,col2 string)"); spark.sql("create table table_2_37 as select * from table_1_37"); spark.sql("create table table_3_37(col1 int,col2 string)"); spark.sql("create table table_4_37 as select * from table_3_37"); ``` - Verified that two `spark_process` entities are created, that have a single `spark_application` entity as `application`. Each of these processes has it's own lineage.     
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
