damondouglas commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991451157
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
Review Comment:
Does this step refer to creating the terraform service account? Otherwise,
aren't we using terraform to provision needed service accounts for the project?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
Review Comment:
Is this bucket needed for terraform state management? I see the "Create GCS
bucket for state" deleted so it isn't clear now the purpose.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
Review Comment:
Where are service account keys needed?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
Review Comment:
Google Cloud APIs can be enabled via infrastructure-as-code. See
https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_service.
Typically, I use a list and the `for-each`. Due to the need for explicit
dependency declaration in terraform with Google Cloud resources, I will
provision the enabling of APIs in a separate pre-requesite terraform module
prior to applying subsequent modules. Alternatively, one would need to
remember to explicitly declare the dependency on the `google_project_service`
enable block via `depends_on`.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
Review Comment:
Are these the IAM roles needed for a particular service account? If so,
1. It's recommended to bind IAM roles using infrastructure-as-code where
feasible. What I've done in the past is create a setup module that provisions
the terraform service account. I then bind a "developer email" which is the
account authenticated via the gcloud commandline utility with the role
https://cloud.google.com/iam/docs/service-accounts#token-creator-role. The
https://cloud.google.com/iam/docs/service-accounts#user-role may be needed as
well but I would try first with just the token creator role. Then I would
provision the minimally permissive IAM roles to the terraform service account.
2. A note about the selected IAM roles above. There are times when admin
roles are needed. However, for most applications, I would incrementally
observe the minimally least permissive role that enables the service account to
perform its needed job.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
Review Comment:
May we consider removing this and not delete the original docker
pre-requisite? The link refers the developer to read more about Docker if it
is not known to the developer.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
Review Comment:
I would provision these APIs using
https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_service
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
Review Comment:
Perhaps, we list helm as one of the pre-requisites above?
```markdown
[helm](https://helm.sh/docs/intro/install/)
```
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output",
"playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output",
"playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
Review Comment:
Now that we see this type of need for extra code in a gradle context, should
we consider not using gradle to wrap our terraform? I've had concerns
initially about the gradle terraform wrapper. If I were to develop this
automation, I would do two things to satisfy provisioning from one's local
development environment manually as well as via automation. First, I would
name the folders in order of their designed application. For example, I
typically have a `00.setup` folder, followed by a `01.network` folder. These
respective folders would hold the infrastructure-as-code needed as explicitly
named by the folder. When automating infrastructure-as-code, in Google Cloud,
we can make use of Cloud Build.
https://github.com/GoogleCloudPlatform/solutions-terraform-cloudbuild-gitops
provides an end-to-end example.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows
operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux
operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for
Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions
for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need
to download ***BEAM Playground*** from the repository. Please open the terminal
and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name"
-Pdocker-tag="tag"
+Once downloading is completed you will find the "beam" folder on your disk
(you can execute the `ls -la` command for Linux, or the `dir` command for
Windows). Open that folder
+
+It is possible to configure BEAM to work with the different versions of the
environment (like staging, production, etc.), you will need to create a special
folder in `playground/terraform/environment` and put all configuration files
inside:
+* File name - `terraform.tfvars`, items inside:
+```
+network_name = "network_name" #Enter the network name - you can
choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of
created project
+gke_name = "playground-backend" #Set the GKE name - you can
choose any name for Google Kubernetes Engine according to the naming policy
above
+region = "us-east1" #Set the region - preferred
region according to your needs
+pg_location = "us-east1-b" #Select the location - location
should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud
bucket where BEAM Playground will put temp files, [terraform state file]
(https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name -
bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
```
+bucket = "bucket_name" #input bucket name - will be used
for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Deploy frontend service
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key
location` (absolute path)
+```
+* Using the following command, we will activate the newly created service
account:
+```
+ gcloud auth activate-service-account `full principal service account`
--key-file=`your service account JSON key location` (absolute path)
+```
Review Comment:
Binding a developer's account (the account authenticated via `gcloud auth
application-default login`) with the role Service Account Token Creator
eliminates the need for the older and less secure practice of using JSON keys.
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
Review Comment:
Can we just have:
```terraform
location_id = var.location_id
```
##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -167,5 +165,8 @@ variable "gke_name" {
variable "gke_location" {
description = "Location of GKE cluster"
- default = "us-central1-a"
+}
+
+variable "pg_location" {
Review Comment:
Personally I found the `pg` in this `pg_location` name confusing initially.
May we consider variable names in the language of the cloud environment? For
example, `gke_location` is clear in that it refers to the location of the GKE
node. In some infrastructure-as-code modules, I've even seen just the use of
`region` as it is conventionally understood that region refers to the Compute
Engine region. Then, when applying to the terraform GKE module, the location
property is simply assigned to the `var.region` variable.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
Review Comment:
May we replace with:
```markdown
- [Google Cloud Project with billing
enabled](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
_(Note: New temporary projects recommended for development and testing.)_
```
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
Review Comment:
Wouldn't a developer on their local machine require these?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
Review Comment:
This link already existed as part of the original listed pre-requisites.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
Review Comment:
Usually the terraform output provides naming-convention errors and/or the
documentation provides the relevant naming convention.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
Review Comment:
Wouldn't we achieve this step in the infrastructure as code?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
Review Comment:
Should we just list this above in the pre-requisites:
```markdown
[Java 11](https://openjdk.org/projects/jdk/11/)
```
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
Review Comment:
What about Mac? Instead, shouldn't we just refer to the kubectl already
bundled as an optional component of gcloud?
https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.
Refering to this covers the supported operating systems without needed to
install via kubernetes.io directly.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows
operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux
operation systems
Review Comment:
May we consider removing this?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows
operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux
operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for
Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions
for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
Review Comment:
May we just list the required technologies as a simple pre-requisite list
with links for further information? An example is:
```markdown
- [docker](https://docs.docker.com)
- [gcloud](https://cloud.google.com/sdk/docs/install)
- [terraform](https://www.terraform.io/downloads)
```
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows
operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux
operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for
Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions
for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need
to download ***BEAM Playground*** from the repository. Please open the terminal
and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
Review Comment:
May we consider removing this?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows
operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux
operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for
Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions
for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need
to download ***BEAM Playground*** from the repository. Please open the terminal
and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name"
-Pdocker-tag="tag"
+Once downloading is completed you will find the "beam" folder on your disk
(you can execute the `ls -la` command for Linux, or the `dir` command for
Windows). Open that folder
+
+It is possible to configure BEAM to work with the different versions of the
environment (like staging, production, etc.), you will need to create a special
folder in `playground/terraform/environment` and put all configuration files
inside:
+* File name - `terraform.tfvars`, items inside:
+```
+network_name = "network_name" #Enter the network name - you can
choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of
created project
+gke_name = "playground-backend" #Set the GKE name - you can
choose any name for Google Kubernetes Engine according to the naming policy
above
+region = "us-east1" #Set the region - preferred
region according to your needs
+pg_location = "us-east1-b" #Select the location - location
should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud
bucket where BEAM Playground will put temp files, [terraform state file]
(https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name -
bucket for some build-in examples for BEAM Playground
Review Comment:
Thank you for listing the required variables. However, I would like to
propose that we leverage the capability of terraform plugins to recommend the
variables within a developer's IDE. For example, if one is using visual studio
code's terraform plugin and edit a terraform.tfvars file. The IDE plugin is
smart to detect the contextual variables in a terraform module. It even shows
what is required and optional. The value of this is that we can keep the
documentation and the infrastructure-as-code orthogonal. What I mean by
orthogonal is that if we have to list all the required variables here in the
README, one needs to maintain the alignment of this list with the actual
infrastructure-as-code. Typically as time moves along, this is not maintained.
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
+ database_type = "CLOUD_DATASTORE_COMPATIBILITY"
+}
+
+resource "google_project_service" "firestore" {
+ project = var.project_id
+ service = "firestore.googleapis.com"
+ disable_dependent_services = true
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+}
+
+resource "google_app_engine_flexible_app_version" "default_app" {
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+ count = var.create_default_service ? 1 : 0
+ service = "default"
+ version_id = "mlflow-default"
+ runtime = "custom"
+ project = var.project_id
+
+ deployment {
+ container {
+ image = "gcr.io/cloudrun/hello"
Review Comment:
Why are we deploying this image?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on
GCP:
+### Before you will start with the installation steps, please read this
carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's
strongly recommended to use the new Google Cloud Project for your new projects*
*
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and
required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service
account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON
key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at
least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your
service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow
you to interact with the Google cloud platform thru the command line
(https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine
resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or
select a service account with a JSON key, and add or select state_bucket for
storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' >
/path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access
control for Google Cloud Platform resources, including the creation of service
accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for
Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch
/path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java,
HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary
instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
instructions for Linux operation systems
+* [Windows
link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows
and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure
-Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows
operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux
operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for
Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions
for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need
to download ***BEAM Playground*** from the repository. Please open the terminal
and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name"
-Pdocker-tag="tag"
+Once downloading is completed you will find the "beam" folder on your disk
(you can execute the `ls -la` command for Linux, or the `dir` command for
Windows). Open that folder
+
+It is possible to configure BEAM to work with the different versions of the
environment (like staging, production, etc.), you will need to create a special
folder in `playground/terraform/environment` and put all configuration files
inside:
+* File name - `terraform.tfvars`, items inside:
+```
+network_name = "network_name" #Enter the network name - you can
choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of
created project
+gke_name = "playground-backend" #Set the GKE name - you can
choose any name for Google Kubernetes Engine according to the naming policy
above
+region = "us-east1" #Set the region - preferred
region according to your needs
+pg_location = "us-east1-b" #Select the location - location
should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud
bucket where BEAM Playground will put temp files, [terraform state file]
(https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name -
bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
```
+bucket = "bucket_name" #input bucket name - will be used
for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Deploy frontend service
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key
location` (absolute path)
+```
+* Using the following command, we will activate the newly created service
account:
+```
+ gcloud auth activate-service-account `full principal service account`
--key-file=`your service account JSON key location` (absolute path)
+```
-```bash
-./gradlew playground:terraform:deployFrontend -Pproject_environment="env-name"
-Pdocker-tag="tag" ```
+# Infrastructure deployment:
+* To deploy the Infrastructure, use the following command (please be sure that
you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env"
(env - folder name which you created for configuration files)
+```
+# Backend deployment:
+Once the script was executed successfully, you will need to authenticate on
Docker and Google Kubernetes Engine
+The following command will authorize us in the Docker registry
+```
+ cat `your service account json key locaton` | docker login -u _json_key
--password-stdin https://`chosen_region`-docker.pkg.dev
Review Comment:
Since we are using Artifact Registry, wouldn't the command instead be:
```
gcloud auth configure-docker \
<REGION>-docker.pkg.dev
```
For example, if the region of the Artifact Registry repository is
`us-central1`, the command would be:
```
gcloud auth configure-docker \
us-central1-docker.pkg.dev
```
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output",
"playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output",
"playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ val configFileName = "values.yaml"
+ val modulePath = project(":playground").projectDir.absolutePath
+ val file = File("$modulePath/infrastructure/helm-backend/$configFileName")
+ val lines = file.readLines()
+ val endOfSlice = lines.indexOfFirst { it.contains("static_ip") }
+ if (endOfSlice != -1) {
+ val oldContent = lines.slice(0 until endOfSlice)
+ val flagDelete = file.delete()
+ if (!flagDelete) {
+ throw kotlin.RuntimeException("Deleting file failed")
+ }
+ val sb = kotlin.text.StringBuilder()
+ val lastLine = oldContent[oldContent.size - 1]
+ oldContent.forEach {
+ if (it == lastLine) {
+ sb.append(it)
+ } else {
+ sb.appendLine(it)
+ }
+ }
+ file.writeText(sb.toString())
+ }
+ file.appendText("""
+static_ip: ${ipaddr}
+redis_ip: ${redis}:6379
+project_id: ${proj}
+registry: ${registry}
+static_ip_name: ${ipaddrname}
+tag: $d_tag
+ """)
+ }
+}
+helm {
+ val backend by charts.creating {
+ chartName.set("backend")
+ sourceDir.set(file("../infrastructure/helm-backend"))
+ }
+ releases {
+ create("backend") {
+ from(backend)
+ }
+ }
+}
+task ("gkebackend") {
+ group = "deploy"
+ val init = tasks.getByName("terraformInit")
+ val apply = tasks.getByName("terraformApplyInf")
+ val takeConfig = tasks.getByName("takeConfig")
+ val push = tasks.getByName("pushBack")
+ val helm = tasks.getByName("helmInstallBackend")
+ val prepare = tasks.getByName("prepareConfig")
+ dependsOn(init)
+ dependsOn(apply)
+ dependsOn(takeConfig)
+ dependsOn(push)
+ dependsOn(helm)
+ dependsOn(prepare)
+ apply.mustRunAfter(init)
+ takeConfig.mustRunAfter(apply)
+ push.mustRunAfter(takeConfig)
+ helm.mustRunAfter(push)
Review Comment:
Can we consider
https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/gke-deploy
instead of executing within a gradle context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
