[
https://issues.apache.org/jira/browse/KNOX-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Kumar M updated KNOX-3507:
-------------------------------
Description:
Summary
-------
Add baseline Livy HA dispatch wiring for LIVYSERVER, LIVY_FOR_SPARK3, and
LIVY_FOR_SPARK4.
This is a follow-up to KNOX-3472 / apache/knox#1411 and KNOX-3473 /
apache/knox#1412, which introduced the Livy for Spark 3 and Livy for Spark 4
service definitions. Those changes left the Livy services on LivyDispatch-only
wiring and did not connect the Knox HA dispatch path for Livy roles.
Problem
-------
The LIVYSERVER, LIVY_FOR_SPARK3, and LIVY_FOR_SPARK4 service descriptors
reference org.apache.knox.gateway.livy.LivyDispatch without ha-classname wiring
to org.apache.knox.gateway.ha.dispatch.LivyHaDispatch.
For deployments that enable Knox HA for Livy, that gap means Livy-specific HA
behavior is not consistently applied (including Livy batch/session POST
handling via LivyHttpServletRequest and Livy HA failover handling such as HTTP
307), compared to the established HA dispatch pattern used for Livy elsewhere
in Knox.
Scope / proposed fix
--------------------
- Introduce baseline LivyHaDispatch (HA dispatch + Livy-specific request
handling hooks).
- Extract LivyHttpServletRequest for reuse from both LivyDispatch and
LivyHaDispatch.
- Add gateway-service-livy as a dependency of gateway-provider-ha where
required for compilation/wiring.
- Set ha-classname on:
- services/livy/0.4.0/service.xml (LIVYSERVER)
- services/livy_for_spark3/0.6.0/service.xml (LIVY_FOR_SPARK3)
- services/livy_for_spark4/1.0.0/service.xml (LIVY_FOR_SPARK4)
Relationship to KNOX-3479
-------------------------
This work is prerequisite baseline wiring for per-instance HA routing tracked
under KNOX-3479 / apache/knox#1420. Keep per-instance routing / homepage
“?host=” behavior out of this change.
Out of scope
------------
- Per-instance HA routing, sticky cookies, Referer-based routing (KNOX-3479)
- Knox Homepage tile JavaScript changes (KNOX-3479)
- New service definition versions beyond what is required for ha-classname
wiring
Acceptance criteria
-----------------
- LivyHaDispatch exists with baseline Livy HA behavior expected from
ConfigurableHADispatch + Livy-specific overrides (including HTTP 307 handling
path consistent with existing Knox HA patterns).
- LivyHttpServletRequest is shared by LivyDispatch and LivyHaDispatch.
- ha-classname is set for all three Livy roles listed above.
- mvn -pl gateway-service-livy,gateway-provider-ha,gateway-service-definitions
-am test -DskipITs passes.
Implementation
--------------
GitHub: apache/knox#1443
was:
*Summary*
Wire baseline Livy HA dispatch for LIVYSERVER, LIVY_FOR_SPARK3, and
LIVY_FOR_SPARK4 in Apache Knox, as a follow-up to KNOX-3472 / apache/knox#1411
and KNOX-3473 / apache/knox#1412. Aligns upstream with internal Gerrit change
214416 (CDPD-45165).
*Motivation*
KNOX-3472 and KNOX-3473 added livy_for_spark3 and livy_for_spark4 service
definitions using org.apache.knox.gateway.livy.LivyDispatch only. They did not
include ha-classname="org.apache.knox.gateway.ha.dispatch.LivyHaDispatch" or
the related baseline HA wiring that downstream already uses from Gerrit 214416.
*Deployments configured for Livy HA need:*
* LivyHaDispatch for Livy-specific HA failover (including HTTP 307 handling)
* Shared LivyHttpServletRequest for Livy POST / proxyUser handling on HA
dispatch
* ha-classname wired on livy (LIVYSERVER), livy_for_spark3, and livy_for_spark4
*Relationship to KNOX-3472 / KNOX-3473*
KNOX-3472 / #1411 and KNOX-3473 / #1412 introduced service definitions, CM
discovery, and homepage assets. They explicitly did not include full
LivyHaDispatch baseline wiring from Gerrit 214416.
This ticket adds that missing HA dispatch wiring only. Per-instance homepage
routing (?host=, sticky cookies, tile JS) is KNOX-3479 and must not be merged
in the same change.
Relationship to KNOX-3479
This ticket blocks KNOX-3479. Merge upstream PR for KNOX-3507 before the
per-instance HA PR (KNOX-3479 / #1420).
*Scope*
gateway-provider-ha
* LivyHaDispatch.java — baseline Livy HA (HTTP 307 / failover; Livy POST via
LivyHttpServletRequest)
* pom.xml — dependency on gateway-service-livy
gateway-service-livy
* LivyHttpServletRequest.java — new public class (extracted from LivyDispatch)
* LivyDispatch.java — uses shared LivyHttpServletRequest
*gateway-service-definitions*
* livy/0.4.0/service.xml — ha-classname:
org.apache.knox.gateway.ha.dispatch.LivyHaDispatch (LIVYSERVER)
* livy_for_spark3/0.6.0/service.xml — same ha-classname (LIVY_FOR_SPARK3)
* livy_for_spark4/1.0.0/service.xml — same ha-classname (LIVY_FOR_SPARK4)
*Out of scope*
* Per-instance HA routing (?host=, Referer, KNOX_LIVY_BACKEND cookie)
* Knox Homepage tile JavaScript / hostname label patching
* Homepage metadata context /livy_for_spark3/ui?host=\{{BACKEND_HOST}} (and
Spark 4 equivalent)
* LivyHaDispatchTest for per-instance routing
* CM discovery or new service definitions (KNOX-3472 / KNOX-3473)
*Acceptance criteria*
* LivyHaDispatch exists with baseline Livy HA behavior (307 handling,
failover, Livy POST wrapper)
* LivyHttpServletRequest is shared by LivyDispatch and LivyHaDispatch
* ha-classname is set on livy, livy_for_spark3, and livy_for_spark4 service
descriptors
* mvn -pl gateway-service-livy,gateway-provider-ha,gateway-service-definitions
-am test -DskipITs succeeds
* No vendor-specific hostnames or internal-only references in ASF-licensed
sources
* KNOX-3479 can build on this change without duplicating baseline
LivyHaDispatch wiring
*Implementation / PR*
GitHub: [#1443|https://github.com/apache/knox/pull/1443] Branch:
samuhasavak5:KNOX-livy-ha-dispatch-wiring JIRA: KNOX-3507
> Wire LivyHaDispatch for LIVYSERVER, LIVY_FOR_SPARK3, and LIVY_FOR_SPARK4
> ------------------------------------------------------------------------
>
> Key: KNOX-3507
> URL: https://issues.apache.org/jira/browse/KNOX-3507
> Project: Apache Knox
> Issue Type: Task
> Reporter: Arun Kumar M
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Summary
> -------
> Add baseline Livy HA dispatch wiring for LIVYSERVER, LIVY_FOR_SPARK3, and
> LIVY_FOR_SPARK4.
> This is a follow-up to KNOX-3472 / apache/knox#1411 and KNOX-3473 /
> apache/knox#1412, which introduced the Livy for Spark 3 and Livy for Spark 4
> service definitions. Those changes left the Livy services on
> LivyDispatch-only wiring and did not connect the Knox HA dispatch path for
> Livy roles.
> Problem
> -------
> The LIVYSERVER, LIVY_FOR_SPARK3, and LIVY_FOR_SPARK4 service descriptors
> reference org.apache.knox.gateway.livy.LivyDispatch without ha-classname
> wiring to org.apache.knox.gateway.ha.dispatch.LivyHaDispatch.
> For deployments that enable Knox HA for Livy, that gap means Livy-specific HA
> behavior is not consistently applied (including Livy batch/session POST
> handling via LivyHttpServletRequest and Livy HA failover handling such as
> HTTP 307), compared to the established HA dispatch pattern used for Livy
> elsewhere in Knox.
> Scope / proposed fix
> --------------------
> - Introduce baseline LivyHaDispatch (HA dispatch + Livy-specific request
> handling hooks).
> - Extract LivyHttpServletRequest for reuse from both LivyDispatch and
> LivyHaDispatch.
> - Add gateway-service-livy as a dependency of gateway-provider-ha where
> required for compilation/wiring.
> - Set ha-classname on:
> - services/livy/0.4.0/service.xml (LIVYSERVER)
> - services/livy_for_spark3/0.6.0/service.xml (LIVY_FOR_SPARK3)
> - services/livy_for_spark4/1.0.0/service.xml (LIVY_FOR_SPARK4)
> Relationship to KNOX-3479
> -------------------------
> This work is prerequisite baseline wiring for per-instance HA routing tracked
> under KNOX-3479 / apache/knox#1420. Keep per-instance routing / homepage
> “?host=” behavior out of this change.
> Out of scope
> ------------
> - Per-instance HA routing, sticky cookies, Referer-based routing (KNOX-3479)
> - Knox Homepage tile JavaScript changes (KNOX-3479)
> - New service definition versions beyond what is required for ha-classname
> wiring
> Acceptance criteria
> -----------------
> - LivyHaDispatch exists with baseline Livy HA behavior expected from
> ConfigurableHADispatch + Livy-specific overrides (including HTTP 307 handling
> path consistent with existing Knox HA patterns).
> - LivyHttpServletRequest is shared by LivyDispatch and LivyHaDispatch.
> - ha-classname is set for all three Livy roles listed above.
> - mvn -pl
> gateway-service-livy,gateway-provider-ha,gateway-service-definitions -am test
> -DskipITs passes.
> Implementation
> --------------
> GitHub: apache/knox#1443
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
