Emil Kleszcz created KNOX-3231:
----------------------------------
Summary: Spark 4 History Server cannot be protected by Knox JWT
after migration to jakarta.servlet
Key: KNOX-3231
URL: https://issues.apache.org/jira/browse/KNOX-3231
Project: Apache Knox
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Emil Kleszcz
After upgrading to Apache Spark 4, the Spark History Server can no longer be
secured using the Knox-based JWT authentication mechanism that worked with
Spark 3.
Root cause:
- Spark 4 migrated from javax.servlet to jakarta.servlet
- The Hadoop AuthenticationFilter used previously via Knox depends on
javax.servlet
- This makes the filter incompatible and prevents the History Server from
starting
- Spark 4 introduces org.apache.spark.ui.JWSFilter, but it requires HS256 JWT
tokens
- Knox currently injects Spark 3–style JWTs that are not compatible with
JWSFilter
Impact:
- Spark 4 HS cannot currently be protected via Knox using JWT
- Only two insecure or partial workarounds exist:
- Run the HS without any UI authentication filter
- Rely solely on network/firewall protection + Knox proxying
This means:
- Spark 4 HS is functionally working
- Secure UI authentication via Knox is currently broken
Environment:
- Apache Spark: 4.0.1
- Java: 17
- Knox-proxied Spark HS
- Previously working with Spark 3 using Hadoop AuthenticationFilter + Knox JWT
Expected behaviour:
- Knox should be able to protect the Spark 4 HSUI using a supported
authentication mechanism (either via HS256-compatible tokens or an
alternative)
Open questions:
- Is this a known incompatibility between Knox and Spark 4?
- Is HS256 token generation for Spark 4 JWSFilter supported or planned in Knox?
- Is there a recommended authentication pattern for securing the Spark 4 HS
behind Knox?
My workaround in production:
- Knox proxying + strict firewall rules blocking direct access to the HS UI
ports
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
