[
https://issues.apache.org/jira/browse/KNOX-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Kumar M updated KNOX-3507:
-------------------------------
Description:
*Summary*
Wire baseline Livy HA dispatch for LIVYSERVER, LIVY_FOR_SPARK3, and
LIVY_FOR_SPARK4 in Apache Knox, as a follow-up to KNOX-3472 / apache/knox#1411
and KNOX-3473 / apache/knox#1412. Aligns upstream with internal Gerrit change
214416 (CDPD-45165).
*Motivation*
KNOX-3472 and KNOX-3473 added livy_for_spark3 and livy_for_spark4 service
definitions using org.apache.knox.gateway.livy.LivyDispatch only. They did not
include ha-classname="org.apache.knox.gateway.ha.dispatch.LivyHaDispatch" or
the related baseline HA wiring that downstream already uses from Gerrit 214416.
*Deployments configured for Livy HA need:*
* LivyHaDispatch for Livy-specific HA failover (including HTTP 307 handling)
* Shared LivyHttpServletRequest for Livy POST / proxyUser handling on HA
dispatch
* ha-classname wired on livy (LIVYSERVER), livy_for_spark3, and livy_for_spark4
*Relationship to KNOX-3472 / KNOX-3473*
KNOX-3472 / #1411 and KNOX-3473 / #1412 introduced service definitions, CM
discovery, and homepage assets. They explicitly did not include full
LivyHaDispatch baseline wiring from Gerrit 214416.
This ticket adds that missing HA dispatch wiring only. Per-instance homepage
routing (?host=, sticky cookies, tile JS) is KNOX-3479 and must not be merged
in the same change.
Relationship to KNOX-3479
This ticket blocks KNOX-3479. Merge upstream PR for KNOX-3507 before the
per-instance HA PR (KNOX-3479 / #1420).
*Scope*
gateway-provider-ha
* LivyHaDispatch.java — baseline Livy HA (HTTP 307 / failover; Livy POST via
LivyHttpServletRequest)
* pom.xml — dependency on gateway-service-livy
gateway-service-livy
* LivyHttpServletRequest.java — new public class (extracted from LivyDispatch)
* LivyDispatch.java — uses shared LivyHttpServletRequest
*gateway-service-definitions*
* livy/0.4.0/service.xml — ha-classname:
org.apache.knox.gateway.ha.dispatch.LivyHaDispatch (LIVYSERVER)
* livy_for_spark3/0.6.0/service.xml — same ha-classname (LIVY_FOR_SPARK3)
* livy_for_spark4/1.0.0/service.xml — same ha-classname (LIVY_FOR_SPARK4)
*Out of scope*
* Per-instance HA routing (?host=, Referer, KNOX_LIVY_BACKEND cookie)
* Knox Homepage tile JavaScript / hostname label patching
* Homepage metadata context /livy_for_spark3/ui?host=\{{BACKEND_HOST}} (and
Spark 4 equivalent)
* LivyHaDispatchTest for per-instance routing
* CM discovery or new service definitions (KNOX-3472 / KNOX-3473)
*Acceptance criteria*
* LivyHaDispatch exists with baseline Livy HA behavior (307 handling,
failover, Livy POST wrapper)
* LivyHttpServletRequest is shared by LivyDispatch and LivyHaDispatch
* ha-classname is set on livy, livy_for_spark3, and livy_for_spark4 service
descriptors
* mvn -pl gateway-service-livy,gateway-provider-ha,gateway-service-definitions
-am test -DskipITs succeeds
* No vendor-specific hostnames or internal-only references in ASF-licensed
sources
* KNOX-3479 can build on this change without duplicating baseline
LivyHaDispatch wiring
*Implementation / PR*
GitHub: [#1443|https://github.com/apache/knox/pull/1443] Branch:
samuhasavak5:KNOX-livy-ha-dispatch-wiring JIRA: KNOX-3507
was:
h2. Summary
Restore baseline Livy HA dispatch wiring that was omitted when KNOX-3472 and
KNOX-3473 landed the Livy for Spark 3/4 Knox service definitions. This aligns
upstream Apache Knox with internal Gerrit change 214416 (CDPD-45165): baseline
\{{LivyHaDispatch}}, shared \{{LivyHttpServletRequest}}, and \{{ha-classname}}
on the Livy-related service descriptors.
h2. Background
* [KNOX-3472|https://issues.apache.org/jira/browse/KNOX-3472] /
[#1411|https://github.com/apache/knox/pull/1411] added \{{livy_for_spark3}}
(service definition, CM discovery, rewrites).
* [KNOX-3473|https://issues.apache.org/jira/browse/KNOX-3473] /
[#1412|https://github.com/apache/knox/pull/1412] added \{{livy_for_spark4}}.
* Those PRs used \{{LivyDispatch}} only and did not wire
\{{org.apache.knox.gateway.ha.dispatch.LivyHaDispatch}}.
* Per-instance homepage routing (\{{?host=}}, sticky cookie, multi-tile UI) is
tracked separately in
[KNOX-3479|https://issues.apache.org/jira/browse/KNOX-3479] /
[#1420|https://github.com/apache/knox/pull/1420].
h2. Proposed changes
# Add \{{gateway-provider-ha/.../LivyHaDispatch.java}} — Livy-specific HA (HTTP
307 handling, failover) and Livy POST handling via \{{LivyHttpServletRequest}}.
# Extract \{{LivyHttpServletRequest}} to \{{gateway-service-livy}} (shared by
\{{LivyDispatch}} and \{{LivyHaDispatch}}).
# Add \{{gateway-service-livy}} dependency in \{{gateway-provider-ha/pom.xml}}.
# Set \{{ha-classname="org.apache.knox.gateway.ha.dispatch.LivyHaDispatch"}} on:
#* \{{services/livy/0.4.0/service.xml}} (LIVYSERVER)
#* \{{services/livy_for_spark3/0.6.0/service.xml}}
#* \{{services/livy_for_spark4/1.0.0/service.xml}}
h2. Out of scope (KNOX-3479)
* Per-instance backend selection (\{{?host=}}, Referer, \{{KNOX_LIVY_BACKEND}}
cookie)
* Knox Homepage tile hostname disambiguation
* Changes to \{{livy_for_spark3}} / \{{livy_for_spark4}} homepage \{{context}}
with \{{?host={{BACKEND_HOST}}}}
h2. Upstream PR
[apache/knox#1443|https://github.com/apache/knox/pull/1443] — branch
\{{KNOX-livy-ha-dispatch-wiring}}
h2. Testing
{code}
mvn -pl gateway-service-livy,gateway-provider-ha,gateway-service-definitions
-am test -DskipITs
{code}
Manual (optional): topology with HA-enabled Livy / Livy for Spark 3/4 URLs;
confirm default HA failover behavior for batch/session POST and 307 handling.
h2. Downstream
After merge, backport to CDH \{{knox}} (coordinate with CDPD-99323 / #758 and
avoid duplicating Gabor’s #796 if full backport covers Spark 4 wiring).
> Wire LivyHaDispatch for LIVYSERVER, LIVY_FOR_SPARK3, and LIVY_FOR_SPARK4
> ------------------------------------------------------------------------
>
> Key: KNOX-3507
> URL: https://issues.apache.org/jira/browse/KNOX-3507
> Project: Apache Knox
> Issue Type: Task
> Reporter: Arun Kumar M
> Priority: Major
>
> *Summary*
> Wire baseline Livy HA dispatch for LIVYSERVER, LIVY_FOR_SPARK3, and
> LIVY_FOR_SPARK4 in Apache Knox, as a follow-up to KNOX-3472 /
> apache/knox#1411 and KNOX-3473 / apache/knox#1412. Aligns upstream with
> internal Gerrit change 214416 (CDPD-45165).
> *Motivation*
> KNOX-3472 and KNOX-3473 added livy_for_spark3 and livy_for_spark4 service
> definitions using org.apache.knox.gateway.livy.LivyDispatch only. They did
> not include ha-classname="org.apache.knox.gateway.ha.dispatch.LivyHaDispatch"
> or the related baseline HA wiring that downstream already uses from Gerrit
> 214416.
> *Deployments configured for Livy HA need:*
> * LivyHaDispatch for Livy-specific HA failover (including HTTP 307 handling)
> * Shared LivyHttpServletRequest for Livy POST / proxyUser handling on HA
> dispatch
> * ha-classname wired on livy (LIVYSERVER), livy_for_spark3, and
> livy_for_spark4
> *Relationship to KNOX-3472 / KNOX-3473*
> KNOX-3472 / #1411 and KNOX-3473 / #1412 introduced service definitions, CM
> discovery, and homepage assets. They explicitly did not include full
> LivyHaDispatch baseline wiring from Gerrit 214416.
> This ticket adds that missing HA dispatch wiring only. Per-instance homepage
> routing (?host=, sticky cookies, tile JS) is KNOX-3479 and must not be merged
> in the same change.
> Relationship to KNOX-3479
> This ticket blocks KNOX-3479. Merge upstream PR for KNOX-3507 before the
> per-instance HA PR (KNOX-3479 / #1420).
> *Scope*
> gateway-provider-ha
> * LivyHaDispatch.java — baseline Livy HA (HTTP 307 / failover; Livy POST via
> LivyHttpServletRequest)
> * pom.xml — dependency on gateway-service-livy
> gateway-service-livy
> * LivyHttpServletRequest.java — new public class (extracted from
> LivyDispatch)
> * LivyDispatch.java — uses shared LivyHttpServletRequest
> *gateway-service-definitions*
> * livy/0.4.0/service.xml — ha-classname:
> org.apache.knox.gateway.ha.dispatch.LivyHaDispatch (LIVYSERVER)
> * livy_for_spark3/0.6.0/service.xml — same ha-classname (LIVY_FOR_SPARK3)
> * livy_for_spark4/1.0.0/service.xml — same ha-classname (LIVY_FOR_SPARK4)
> *Out of scope*
> * Per-instance HA routing (?host=, Referer, KNOX_LIVY_BACKEND cookie)
> * Knox Homepage tile JavaScript / hostname label patching
> * Homepage metadata context /livy_for_spark3/ui?host=\{{BACKEND_HOST}} (and
> Spark 4 equivalent)
> * LivyHaDispatchTest for per-instance routing
> * CM discovery or new service definitions (KNOX-3472 / KNOX-3473)
> *Acceptance criteria*
> * LivyHaDispatch exists with baseline Livy HA behavior (307 handling,
> failover, Livy POST wrapper)
> * LivyHttpServletRequest is shared by LivyDispatch and LivyHaDispatch
> * ha-classname is set on livy, livy_for_spark3, and livy_for_spark4 service
> descriptors
> * mvn -pl
> gateway-service-livy,gateway-provider-ha,gateway-service-definitions -am test
> -DskipITs succeeds
> * No vendor-specific hostnames or internal-only references in ASF-licensed
> sources
> * KNOX-3479 can build on this change without duplicating baseline
> LivyHaDispatch wiring
> *Implementation / PR*
> GitHub: [#1443|https://github.com/apache/knox/pull/1443] Branch:
> samuhasavak5:KNOX-livy-ha-dispatch-wiring JIRA: KNOX-3507
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
