daniellavoie edited a comment on issue #5580:
URL:
https://github.com/apache/incubator-pinot/issues/5580#issuecomment-648904372
# Refactor start/stop commands
## Table of Content
* [Motivation](#motivation)
* [Goals](#goals)
* [Anti-Goals](#anti-goals)
* [Current State](#current-state)
* [Proposal](#proposal)
* [Step 1 - A unified startup command for
operator](#step-1-a-unified-startup-command-for-operators)
* [Step 2 - Implement a developer focused
CLI](#step-2-implement-a-developer-focused-cli)
## Motivation
Users typically go through 3 phases.
* **quickstart:** comes with preloaded data (quickstart-batch.sh) (start's
zookeeper, controller, server, broker in a single process)
* **standalone (local):** start Pinot on a single node and ingest data
(bin/pinot-admin.sh Start(Controller|Server|Broker))
* **cluster:** Deploy and operate Pinot on multiple nodes.
Today, each phase has a different set of scripts which makes it hard to
write the user documentation.
We should refactor some of the commands to make this process simpler.
## Goals
* Make it easier for developers to work locally with Pinot.
* Improve operation on container-based environments.
* Offer no concession to operator and developer experience. Consider
distinct tools if needed.
## Anti-Goals
* Introduce breaking changes to the configuration and operation
* Decrease the quality and the user experience of operators by introducing
non-cloud native friendly behavior to the way Pinot operates.
## Current state
The current set of scripts are not articulated around personas. You
currently need to pick a script based on how you want to run Pinot. How you
intended to run Pinot (quickstart, standalone and cluster) does not change how
you expect the service behaves.
A good example of an opinionated behavior based on your persona is if Pinot
should run in background or foreground. Developers would prefer a "background
service oriented" mode where best practice for operations would be to run as
foreground. When running in foreground, you can wrap the process with any
service manager or container orchestrator that is standard to the operator's
organization.
## Proposal
## Step 1 - A unified startup command for operators
This new `pinot` command is basically a rename of the existing
`pinot-admin.sh` only that it stripped for the notion of start mode
(quickstart, standalone). It is focused on running cluster node in foreground
with maximum configuration options. This is the command that would be used by
the Pinot Docker image.
### Command
```
pinot --help
Usage:
pinot SERVICE [options]
Starts a Pinot service
Available Service:
zookeeper
broker
server
controller
Available Options:
--help, -h
--config.paths=<Pinot-config-file>
All Pinot properties may be passed as CLI arguments,environment
variables or property files.
Refer to the Pinot reference documentation for a
complete list of all available properties.
```
### Parallel initiative - Configuration management refactoring
This Start/Stop command refactoring is also an opportunity to support all
Pinot properties to be as CLI arguments and Environment Variables rather than
only config files. See PR #5608 which introduce the ability for Pinot to load
property from independent configuration sources. Additional phase will be
required to plug cli arguments and environments variable to Pinot configuration
management.
* Step 1.1 - #5608 - Introduce the ability for Pinot to load properties from
dependent configuration sources.
* Step 1.2 - TBD - Enable properties loaded from environment variables
* Step 1.3 - TBD - Refactor startup command to support generic properties
passed as CLI arguments.
## Step 2 - Implement a developer focused CLI
This new `pinot-local` is a developer focused wrapper around the `pinot`
CLI. It main difference is that it acts as a service manager that can run Pinot
components in background. It is intended to be cross platform, and not tied to
any OS specific service manager. By default, state would be persistent in a
`.pinot` folder in the home directory of the executing user.
### Features
* Start and stop Pinot components in background
* Provides opinionated defaults for quickstart, standalone modes.
* Provides log access.
* Control and run multiple instances by switching state between state
directories
### Optional Features
* Download / Update Pinot
* Download / Update Java
### Commands
### Help
```
$ pinot-local --help
Usage:
pinot-local [generic-args] COMMAND
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing directory
Available commands:
start
stop
logs
add-table
add-ingest-job
```
#### Start
```
$ pinot-local start --help
Starts local Zookeeper and Pinot instances. State of these services will be
tracked in the provided state-dir
Usage:
pinot-local [generic-args] start START_MODE
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing directory
Required arguments:
START_MODE 'quickstart' or 'standalone'
$ Pinot quickstart batch
```
##### Quickstart
```
$ pinot-local start quickstart --help
Starts local Zookeeper and Pinot instances with preloaded sample data. State
of these services will be tracked in the provided state-dir
Usage:
pinot-local [generic-args] start quickstart MODE
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing directory
Available Modes:
batch
stream
hybrid
$ pinot-local start quickstart batch
```
#### Standalone
```
$ pinot-local start standalone --help
Usage:
pinot-local [generic-args] start standalone [options]
Starts local Zookeeper and Pinot instances. State of these services will be
tracked in the provided state-dir
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing
directory
Available options:
--zookeeper-config-file Path to a zookeeper configuration file
--config-file Path to a Pinot configuration file for all
services.
$ pinot-local standalone --zookeeper-config-file=zoo.cfg
--config-file=pinot.properties
```
#### Stop
```
$ pinot-local stop --help
Usage:
pinot-local [generic-args] stop [options]
Stops the local Zookeeper and Pinot instances found in the provided
state-dir.
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing
directory
Available options:
--zookeeper-config-file Path to a zookeeper configuration file
--config-file Path to a Pinot configuration file for all
services.
$ pinot-local standalone --zookeeper-config-file=zoo.cfg
--config-file=pinot.properties
```
#### Logs
```
$ pinot-local logs --help
Usage:
pinot-local [generic-args] logs SERVICE [options]
Prints logs of the specified service
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing
directory
Available Service:
zookeeper
broker
server
controller
Available options:
--follow, -f Follow log output
```
#### Add table
```
$ pinot-local add-table --help
Usage:
pinot-local [generic-args] add-table TABLE_CONFIG TABLE_SCHEMA
Creates a new Pinot table out of a table config and a table schema
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing directory
Required Arguments:
TABLE_CONFIG: Path to the table config json
TABLE_SCHEMA: Path to the table schema json
```
#### Add an ingestion Job
```
$ pinot-local add-ingest-job --help
Usage:
pinot-local [generic-args] add-ingest-job JOB_SPEC
Creates a new batch ingestion job
Generic arguments:
--state-dir State directory of the Pinot instance
Defaults to `~/.pinot`
Switch between Pinot instances by changing directory
Required Arguments:
JOB_SPEC: Path to the ingestion job json spec json
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
