dongjoon-hyun commented on PR #43743: URL: https://github.com/apache/spark/pull/43743#issuecomment-1809845273
To @mridulm . The above was the exact answer for your following question. What I meant is that Apache Spark already provide more patterns including `the prefix customazable`. > Why not continue to use the existing scheme (iirc prefix + time: I am afk, so can't confirm), but make the prefix customizable instead ? For the following question, that is the exact design goal of this PR where a upper service system uses `Apache Spark Standalone cluster as an execution subsystem`. The upper service layer already provides many features like the unique appName and security layers. So, there is no such things like `hard to debug` or `minimizing the issue`. > why we are allowing explicitly setting an application id, and introducing hard to debug modes; instead of relying on minimizing the issue. In addition, I already answered with the examples. Let me reiterate that. You can generate a SHS link like `https://shs/history/app-202311140014-0204/` with `appId` while you cannot with `appName`. `appId` is used as a foreign key across Apache Spark systems (Storage like S3 event log and Spark driver logs, SHS link, Spark Master link). It opens a wide range of opportunity to us. - https://github.com/apache/spark/pull/43743#issuecomment-1806033968 Lastly, this configuration is introduced as `Experimental` as you see because we are already aware of the risk. We don't recommend this configuration for novice users. This is more like automated subsystems in the industrial production usages. https://github.com/apache/spark/blob/2cac768fabaa7cad40390b2205dd9c5000011e4c/core/src/main/scala/org/apache/spark/internal/config/package.scala#L1852 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
