This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch aws-quickstart in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 84c76dafb4a34dd07ed952ee90d72075c9502a4f Author: Xiang Fu <[email protected]> AuthorDate: Mon Nov 4 21:02:16 2019 -0800 Adding example for aws --- kubernetes/examples/helm/README.md | 81 ++++++++++++++++++++-- .../examples/helm/pinot-realtime-quickstart.yml | 36 ++++++++-- 2 files changed, 108 insertions(+), 9 deletions(-) diff --git a/kubernetes/examples/helm/README.md b/kubernetes/examples/helm/README.md index 2c8c280..f1648cf 100644 --- a/kubernetes/examples/helm/README.md +++ b/kubernetes/examples/helm/README.md @@ -28,6 +28,69 @@ - Configure kubectl to connect to the Kubernetes cluster. - Skip to [Section: How to setup a Pinot cluster for demo](#How to setup a Pinot cluster for demo) if a k8s cluster is already setup. + +## (Optional) Setup a Kubernetes cluster on Amazon Elastic Kubernetes Service (Amazon EKS) + +### (Optional) Create a new k8s cluster on AWS EKS + +- Install AWS CLI (<https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html#install-tool-bundled>) +- Install AWS-IAM-AUTHENTICATOR (<https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html>) +- Install eksctl (<https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html#installing-eksctl>) + +- Login to your AWS account. + +```bash +aws configure +``` + +Note that environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` will override the aws configuration in file `~/.aws/credentials`. + +- Amazon EKS Prerequisites + +It requires Role ARN, VPC, Subnets, Security Groups to create an EKS cluster. + +Please follow wiki [Amazon EKS Prerequisites](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#eks-prereqs) to +create Amazon Resource Name (ARN) and Amazon EKS Cluster VPC + +- Create an EKS cluster + +Please modify the parameters in the example command below: + +```bash +eksctl create cluster \ +--name pinot-quickstart \ +--version 1.14 \ +--region us-west-2 \ +--nodegroup-name standard-workers \ +--node-type t3.small \ +--nodes 3 \ +--nodes-min 3 \ +--nodes-max 4 \ +--node-ami auto +``` +You can monitor cluster status by command: + +```bash +EKS_CLUSTER_NAME=pinot-quickstart +aws eks describe-cluster --name ${EKS_CLUSTER_NAME} +``` + +Once the cluster is in `ACTIVE` status, it's ready to be used. + +### (Optional) How to connect to an existing cluster + +Simply run below command to get the credential for the cluster you just created or your existing cluster. + +```bash +EKS_CLUSTER_NAME=pinot-quickstart +aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} +``` + +To verify the connection, you can run +```bash +kubectl get nodes +``` + ## (Optional) Setup a Kubernetes cluster on Google Kubernetes Engine(GKE) ### (Optional) Create a new k8s cluster on GKE @@ -129,18 +192,26 @@ helm dependency update ### Start Pinot with Helm +If cluster is just initialized, ensure helm is initialized by running: + +```bash +helm init --service-account tiller +``` + +Then deploy pinot cluster by: + ```bash helm install --namespace "pinot-quickstart" --name "pinot" . ``` #### Troubleshooting -Error: Please run below command if encountering issue: +- Error: Please run below command if encountering issue: ``` Error: could not find tiller". ``` -Resolution: +- Resolution: ```bash kubectl -n kube-system delete deployment tiller-deploy @@ -148,11 +219,11 @@ kubectl -n kube-system delete service/tiller-deploy helm init --service-account tiller ``` -Error: Please run below command if encountering permission issue: +- Error: Please run below command if encountering permission issue: ```Error: release pinot failed: namespaces "pinot-quickstart" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "pinot-quickstart"``` -Resolution: +- Resolution: ```bash kubectl apply -f helm-rbac.yaml @@ -196,7 +267,7 @@ Please use below script to do local port-forwarding and open Pinot query console ### How to clean up Pinot deployment ```bash -kubectl delete -f pinot-example-loader.yml +kubectl delete -f pinot-realtime-quickstart.yml helm del --purge kafka helm del --purge pinot ``` diff --git a/kubernetes/examples/helm/pinot-realtime-quickstart.yml b/kubernetes/examples/helm/pinot-realtime-quickstart.yml index 7b47bb2..a121b11 100644 --- a/kubernetes/examples/helm/pinot-realtime-quickstart.yml +++ b/kubernetes/examples/helm/pinot-realtime-quickstart.yml @@ -414,21 +414,34 @@ data: apiVersion: batch/v1 kind: Job metadata: - name: pinot-realtime-quickstart + name: pinot-realtime-quickstart-pinot-schema-creation namespace: pinot-quickstart spec: template: spec: containers: - - name: loading-data-to-kafka - image: winedepot/pinot:0.1.13-SNAPSHOT - args: [ "StreamAvroIntoKafka", "-avroFile", "sample_data/airlineStats_data.avro", "-kafkaTopic", "flights-realtime", "-kafkaBrokerList", "kafka:9092", "-zkAddress", "kafka-zookeeper:2181" ] - name: pinot-add-example-schema image: winedepot/pinot:0.1.13-SNAPSHOT args: [ "AddSchema", "-schemaFile", "/var/pinot/examples/airlineStats_schema.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ] volumeMounts: - name: examples mountPath: /var/pinot/examples + restartPolicy: OnFailure + volumes: + - name: examples + configMap: + name: examples + backoffLimit: 100 +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: pinot-realtime-quickstart-pinot-table-creation + namespace: pinot-quickstart +spec: + template: + spec: + containers: - name: pinot-add-example-realtime-table image: winedepot/pinot:0.1.13-SNAPSHOT args: [ "AddTable", "-filePath", "/var/pinot/examples/airlineStats_realtime_table_config.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ] @@ -440,6 +453,21 @@ spec: - name: examples configMap: name: examples + backoffLimit: 100 +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: pinot-realtime-quickstart-load-data-into-kafka + namespace: pinot-quickstart +spec: + template: + spec: + containers: + - name: loading-data-to-kafka + image: winedepot/pinot:0.1.13-SNAPSHOT + args: [ "StreamAvroIntoKafka", "-avroFile", "sample_data/airlineStats_data.avro", "-kafkaTopic", "flights-realtime", "-kafkaBrokerList", "kafka:9092", "-zkAddress", "kafka-zookeeper:2181" ] + restartPolicy: OnFailure backoffLimit: 3 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
