Yi Pan (Data Infrastructure) created SAMZA-1893: ---------------------------------------------------
Summary: JobNodeConfigurationGenerator should only generate configurations for operators, streams, stores, and tables that are reachable by a JobNode Key: SAMZA-1893 URL: https://issues.apache.org/jira/browse/SAMZA-1893 Project: Samza Issue Type: Improvement Reporter: Yi Pan (Data Infrastructure) Currently, the planner does not generate multi-job ExecutionPlan yet. And hence, the current implementation of JobNodeConfigurationGenerator does not strictly follow the rule to only generate the configuration for input/output/intermediate streams, operators, stores, and tables reachable by the current JobNode yet (i.e. for single JobNode plan, everything is reachable by the only JobNode). When we extend it to multi-job plan, we need to generate the configurations only for streams, operators, stores, and tables that are reachable by a JobNode. If two JobNodes collide on the configuration for those configuration, it will result in the following problems: 1) input streams are consumed multiple times, unnecessarily 2) stores' changelog will be written by multiple jobs, creating consistency issue in recovery 3) tables will be accessed (read or write) by multiple jobs, also creating consistency issues We need to make sure that JobNodes in multi-job plans don't create collision in configuration for input/output/intermediate streams, state stores, and tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)