[
https://issues.apache.org/jira/browse/SPARK-60340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60340:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid System.getenv on every JavaUtils.isTesting call
> -----------------------------------------------------
>
> Key: SPARK-60340
> URL: https://issues.apache.org/jira/browse/SPARK-60340
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.3
> Reporter: Sebastian Hillig
> Priority: Major
> Labels: pull-request-available
>
> On Linux, {{System.getenv(String)}} goes through
> {{{}java.lang.ProcessEnvironment{}}}, which wraps the name in a new
> byte[]-backed {{Variable}} on every lookup, so each {{isTesting()}} call
> allocates (56 bytes per call in the measurement below). {{isTesting}} is on
> hot paths. For example, {{AnalysisHelper.assertNotAnalysisRule}} calls it
> from the {{LogicalPlan}} overrides of {{transformDownWithPruning}} /
> {{transformUpWithPruning}} / {{{}transformAllExpressionsWithPruning{}}},
> which run once per plan node visited by every transform. In a JFR allocation
> profile of driver-side query planning, these call sites accounted for roughly
> 1.8% of driver allocation and about 1-2% of CPU.
> A JVM's environment cannot change after startup, so caching the env lookup is
> safe. The system property cannot be cached: tests set and clear it at runtime
> ({{{}SparkTestSuite.beforeAll{}}}, {{{}BenchmarkBase{}}},
> {{{}KryoSerializerBenchmark{}}}). The only code that mutates the process
> environment reflectively ({{{}SparkConnectClientSuite.withEnvs{}}},
> {{{}JavaXmlSuite.setEnv{}}}) touches {{SPARK_REMOTE}} /
> {{{}SPARK_LOCAL_IP{}}}, not {{{}SPARK_TESTING{}}}. Code that sets
> {{SPARK_TESTING}} for a child process via {{ProcessBuilder.environment()}} is
> unaffected, because the child is a new JVM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
