vinodkc commented on code in PR #53261:
URL: https://github.com/apache/spark/pull/53261#discussion_r2579619880
##########
sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/test/RemoteSparkSession.scala:
##########
@@ -93,6 +93,11 @@ object SparkConnectServerUtils {
if (isDebug) {
builder.redirectError(Redirect.INHERIT)
builder.redirectOutput(Redirect.INHERIT)
+ } else {
+ // If output is not consumed, the stdout/stderr pipe buffers will fill
up,
+ // causing the server process to block on write() calls
Review Comment:
> I think the original intention here is to redirect sub-process logs (via
Redirect.INHERIT because the parent process respects log4j config) only when
debug is on.
@pan3793
Looking at `IntegrationTestUtils.scala `
```
private[sql] def debugConfigs: Seq[String] = {
val log4j2 =
s"$connectClientHomeDir/src/test/resources/log4j2.properties"
if (isDebug) {
Seq(
// Enable to see the server plan change log
// "--conf",
// "spark.sql.planChangeLog.level=WARN",
// Enable to see the server grpc received
// "--conf",
// "spark.connect.grpc.interceptor.classes=" +
// "org.apache.spark.sql.connect.service.LoggingInterceptor",
// Redirect server log into console
"--conf",
s"spark.driver.extraJavaOptions=-Dlog4j.configurationFile=$log4j2")
} else Seq.empty
}
```
Note: When `isDebug=false`, it returns `Seq.empty`
That Means:
When `SPARK_DEBUG_SC_JVM_CLIENT=false:`
- No log4j configuration is passed to the server
- Server uses default logging (doesn't create unit-tests.log)
- Only stdout/stderr files are created
When `SPARK_DEBUG_SC_JVM_CLIENT=true`:
- Log4j configuration IS passed (line 77)
- Server uses log4j2.properties which creates unit-tests.log
- Both log4j logs AND stdout/stderr are available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
