[
https://issues.apache.org/jira/browse/PHOENIX-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-8039:
-----------------------------------------
Description:
The vector harness is a client framework responsible for ingesting large scale
vector corpora into Phoenix tables, constructing secondary vector indexes,
computing brute force ground truth, executing multidimensional query and
mutation sweeps, introducing operational disruptions, and recording execution
manifests alongside metric time series. The harness exercises the two index
architectures implemented as PHOENIX-7998 and PHOENIX-8009.
For developer convenience in testing these features in a distributed topology
(albeit single host), this work provides a Docker-based distributed cluster
topology:
- A coordination and storage tier consisting of a standalone ZooKeeper daemon,
an HDFS NameNode, a SecondaryNameNode, and three independent DataNodes.
- A distributed compute tier featuring a YARN ResourceManager, three YARN
NodeManagers, and a dedicated MapReduce JobHistoryServer.
- A database tier consisting of an HBase Master and three RegionServers
hosting Phoenix coprocessors.
- Deterministic initialization, readiness, and health check orchestration.
- Client containers providing environments for harness workload phases and
administrative maintenance utilities, including IndexFsckTool, IndexTool, the
HBase shell, and Hadoop command line interfaces.
—
Design doc: [https://gist.github.com/apurtell/5b038f7e8a89b6d6c6dbc4b73c564e07]
was:
The vector harness is a client framework responsible for ingesting large scale
vector corpora into Phoenix tables, constructing secondary vector indexes,
computing brute force ground truth, executing multidimensional query and
mutation sweeps, introducing operational disruptions, and recording execution
manifests alongside metric time series. The harness exercises the two index
architectures implemented as PHOENIX-7998 and PHOENIX-8009.
For developer convenience in testing these features in a distributed topology
(albeit single host), this work provides a Docker-based distributed cluster
topology:
- A coordination and storage tier consisting of a standalone ZooKeeper daemon,
an HDFS NameNode, a SecondaryNameNode, and three independent DataNodes.
- A distributed compute tier featuring a YARN ResourceManager, three YARN
NodeManagers, and a dedicated MapReduce JobHistoryServer.
- A database tier consisting of an HBase Master and three RegionServers hosting
Phoenix coprocessors.
- Deterministic initialization, readiness, and health check orchestration.
- Client containers providing environments for harness workload phases and
administrative maintenance utilities, including IndexFsckTool, IndexTool, the
HBase shell, and Hadoop command line interfaces.
All containers execute within the Docker virtual machine network namespace,
mapping all network listeners to localhost while isolating daemon instances via
deterministic port allocations.
---
Design doc: https://gist.github.com/apurtell/5b038f7e8a89b6d6c6dbc4b73c564e07
> Vector index test harness docker cluster
> ----------------------------------------
>
> Key: PHOENIX-8039
> URL: https://issues.apache.org/jira/browse/PHOENIX-8039
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
>
> The vector harness is a client framework responsible for ingesting large
> scale vector corpora into Phoenix tables, constructing secondary vector
> indexes, computing brute force ground truth, executing multidimensional query
> and mutation sweeps, introducing operational disruptions, and recording
> execution manifests alongside metric time series. The harness exercises the
> two index architectures implemented as PHOENIX-7998 and PHOENIX-8009.
> For developer convenience in testing these features in a distributed topology
> (albeit single host), this work provides a Docker-based distributed cluster
> topology:
> - A coordination and storage tier consisting of a standalone ZooKeeper
> daemon, an HDFS NameNode, a SecondaryNameNode, and three independent
> DataNodes.
> - A distributed compute tier featuring a YARN ResourceManager, three YARN
> NodeManagers, and a dedicated MapReduce JobHistoryServer.
> - A database tier consisting of an HBase Master and three RegionServers
> hosting Phoenix coprocessors.
> - Deterministic initialization, readiness, and health check orchestration.
> - Client containers providing environments for harness workload phases and
> administrative maintenance utilities, including IndexFsckTool, IndexTool, the
> HBase shell, and Hadoop command line interfaces.
> —
> Design doc:
> [https://gist.github.com/apurtell/5b038f7e8a89b6d6c6dbc4b73c564e07]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
