[ https://issues.apache.org/jira/browse/SPARK-52336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shruti Singhania updated SPARK-52336: ------------------------------------- Summary: Set a default and prepend Spark identifier to GCS user-agent (was: Set a default for fs.gs.application.name.suffix when GCS is used and not user-defined) > Set a default and prepend Spark identifier to GCS user-agent > ------------------------------------------------------------ > > Key: SPARK-52336 > URL: https://issues.apache.org/jira/browse/SPARK-52336 > Project: Spark > Issue Type: Task > Components: Spark Core > Affects Versions: 3.5.6, 4.0.1 > Reporter: Shruti Singhania > Priority: Minor > Labels: Google, configuration, gcs, hadoop-conf > > *1. Current Behavior:* > Apache Spark does not currently set a default value for the GCS Hadoop > connector configuration {{{}fs.gs.application.name.suffix{}}}. Users who want > to leverage this GCS connector feature for better traceability of Spark > applications in GCS logs and metrics must set it explicitly either in Hadoop > configuration files ({{{}core-site.xml{}}}), via {{{}spark-submit --conf{}}}, > or programmatically in their Spark application. > *2. Problem / Motivation:* > The {{fs.gs.application.name.suffix}} property is very useful for identifying > which application is performing GCS operations, especially in environments > where multiple Spark applications (or other Hadoop applications) interact > with GCS concurrently. > Without a default set by Spark when GCS is used: > * Many users might be unaware of this beneficial GCS connector feature. > * GCS logs and metrics is getting harder to correlate with specific Spark > applications, increasing debugging time and operational overhead. > * It introduces an extra configuration step for users who would benefit from > this tagging. > Setting a sensible default when GCS is detected would improve the experience > for Spark users on GCS, providing better traceability with no extra > configuration effort for the common case. > *3. Proposed Change:* > We propose that Spark should automatically set a default value for > {{fs.gs.application.name.suffix}} if: > # The application is interacting with Google Cloud Storage (i.e., paths with > the {{gs://}} scheme are used). > # The user has *not* already provided a value for > {{fs.gs.application.name.suffix}} in their Hadoop configuration or Spark > configuration. User-defined values should always take precedence. > *4. Implementation Considerations* {*}(Open for Discussion){*}{*}:{*} > * *Detection of GCS Usage:* Spark would need to detect when a {{FileSystem}} > for the {{gs://}} scheme is being initialized or used. This might be done in > {{HadoopFSUtils}} or a similar place where the Hadoop {{Configuration}} > object is prepared for file system interactions. > * *Precedence:* The logic must ensure that this default is only applied if > {{fs.gs.application.name.suffix}} (and potentially {{fs.gs.application.name}} > if the suffix is intended to be appended to it by the connector) is not > already present in the Hadoop {{Configuration}} being used.{*}{*} > *5. Benefits:* > * *Improved Traceability:* Easier to identify Spark application interactions > in GCS request logs and metrics provided by the GCS connector. > * *Enhanced Debugging:* Simplifies pinpointing GCS operations related to > specific Spark jobs. > * *Better User Experience:* Provides a useful GCS integration feature by > default, reducing boilerplate configuration for users. > * *Consistency:* Encourages a good practice for applications interacting > with GCS. > > *Impact:* This change is expected to be low-impact and beneficial. It adds a > configuration property that the GCS connector already understands. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org