AnishaKondaTR opened a new issue, #57113: URL: https://github.com/apache/spark/issues/57113
Title Spark Streaming (DStream) UI / receiver diagnostics not reliably available after EMR cluster termination Description We are running a long-lived Apache Spark Streaming application using the DStream API on AWS EMR. The application consumes records from multiple Amazon Kinesis Data Streams using a custom Receiver-based consumer. While the EMR cluster is alive, we can inspect the Spark UI through the YARN ResourceManager proxy to debug: Streaming batch processing delays Scheduling delays / backlog Receiver status and errors Executor failures Stage and task failures However, our EMR clusters are ephemeral and are created/terminated by Step Functions. During incidents such as executor OOMs, YARN preemption, driver failure, or an automated restart Lambda terminating the cluster, the Spark UI disappears with the cluster. This makes it difficult to reconstruct the state of the Spark Streaming application at the time of failure. We would like guidance on the recommended pattern for preserving and debugging Spark Streaming UI/state after the hosting cluster has been terminated. Environment Apache Spark version: 3.2.3 API: Spark Streaming / DStream API Not Structured Streaming Cluster manager: YARN Platform: AWS EMR Cluster type: Ephemeral, per-job EMR clusters Application type: Long-running Spark Streaming application Source type: Custom Receiver-based Kinesis consumer Checkpointing / lease tracking: DynamoDB-backed checkpoint/lease table Sink/storage: Apache Hudi Merge-on-Read tables on S3 Application Flow The application consumes change events from four Kinesis Data Streams: Main stream Norm stream Nims stream Doc stream Each stream is provisioned per environment and region. The high-level flow is: Kinesis Data Streams ├─ novustni ├─ novustni-norm ├─ novustni-nims └─ novustni-doc │ ▼TNIStreamingApp ├─ KinesisCustomConsumer │ └─ Custom Spark Streaming Receiver │ └─ DynamoDB-backed checkpoint / lease tracking │ ├─ KinesisInputNovusMessage │ └─ Parses and validates incoming messages │ └─ Routes by collectionType and operationType │ - BULK_INGEST │ - PROMOTE │ - DEMOTE │ - REINIT │ - PROMOTE_WITH_RESET │ ├─ Workflow filtering │ └─ Further routes DOC traffic into: │ - DOCKET │ - PUBLICRECORD │ - plain DOC │ └─ Routing is based on S3 object name in the message │ └─ NovusHudiOperations � � └─ Performs Hudi Merge-on-Read upserts └─ Performs bulk inserts └─ Performs reinitializations └─ Emits Datadog metrics Problem Statement When the EMR cluster is running, we can SSH tunnel to the Spark UI and inspect the live Streaming tab, executor state, stages, tasks, and receiver information. However, once the EMR cluster is terminated, the Spark UI and YARN ResourceManager proxy are no longer available. This often happens during or after incidents, for example: Driver crash Executor OOM YARN preemption Receiver-thread failure Application restart Auto-remediation Lambda terminating the cluster Step Functions terminating the EMR cluster As a result, we lose the ability to inspect the Spark Streaming UI state for the failed application, especially the Streaming tab details such as: Last processed batches Batch processing time Scheduling delay Total delay Receiver status Receiver errors Input rate / processing rate Backlog symptoms Stage/task failures associated with the failed batch We are trying to determine whether Spark event logs plus a History Server are expected to provide complete post-mortem visibility for DStream-based Spark Streaming applications. Questions 1. Spark event log / History Server support for DStream Streaming UI For a Spark Streaming application using the DStream API with a custom Receiver, is the following expected to fully reconstruct the Streaming UI after the cluster is gone? spark.eventLog.enabled=truespark.eventLog.dir=s3://... with either: an external Spark History Server, or EMR Persistent Application UI Specifically, should the History Server be expected to reconstruct the classic Spark Streaming tab, including: Completed batches Batch processing time Scheduling delay Total delay Receiver information Receiver errors Input records Processing records Associated jobs/stages/tasks per batch Or are there known gaps where some DStream/receiver-specific UI state is only available in the live driver UI and is not fully persisted to the Spark event log? We are particularly interested in whether there are differences between: Spark Streaming / DStream event-log persistence Structured Streaming event-log / query-progress persistence 2. Ensuring the final events are flushed before ungraceful shutdown Are there recommended Spark settings or shutdown practices to improve the chance that the final batch and failure-related events are written to the event log before the cluster disappears? For example: spark.eventLog.enabled=truespark.eventLog.dir=s3://...spark.eventLog.compress=truespark.eventLog.rolling.enabled=truespark.eventLog.rolling.maxFileSize=... Are there additional settings or practices recommended for: Driver crash Executor OOM YARN application kill EMR cluster termination Spark Streaming graceful shutdown Receiver shutdown We would like to know whether Spark provides any guarantees around event-log flushing for the final batch in failure scenarios, or whether some loss of the most recent UI events is expected during abrupt termination. 3. Receiver-based custom source diagnostics Because the application uses a custom Receiver-based source rather than Structured Streaming's Source API, we would like guidance on best practices for surfacing receiver-thread failures in a way that survives cluster termination. For example, should custom Receiver implementations explicitly: log receiver-thread exceptions through the driver/executor logger call restart(...) with the failure cause call reportError(...) emit custom Spark metrics emit external metrics/logs to systems such as CloudWatch or Datadog persist receiver offsets/checkpoints externally expose receiver health through a custom listener Are receiver failures captured in the Spark event log / History Server if they are reported using Spark Streaming Receiver APIs, or should we assume that external logging/metrics are required for reliable post-mortem debugging? Expected Guidance We are looking for guidance on the recommended Spark pattern for post-mortem debugging of DStream-based Spark Streaming applications using custom Receivers, especially after the hosting cluster is terminated. In particular, we would like to understand: Whether Spark event logs and the History Server are sufficient for reconstructing the Streaming UI for DStream applications. What event-log settings or shutdown practices are recommended to minimize loss of final events. How custom Receiver implementations should report errors so that receiver failures remain diagnosable after the live Spark UI is gone. Whether there are known limitations or behavioral differences compared with Structured Streaming. Whether there are any Spark-side improvements or documentation gaps around this behavior. Additional Context This issue is especially impactful because the application runs on ephemeral EMR clusters. Once the cluster is terminated, we lose the live Spark UI and the YARN proxy endpoint. Without reliable History Server reconstruction or external diagnostics, it is difficult to determine whether a failure was caused by: Backpressure / backlog Receiver failure Kinesis read issue Executor OOM YARN preemption Long-running Hudi write Skewed batch Driver instability Cluster termination during active processing Any clarification on the expected behavior of Spark event logs and History Server for Spark Streaming DStream applications would be very helpful. @koochiswathiTR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
