This is an automated email from the ASF dual-hosted git repository. djwang pushed a commit to branch merge-with-upstream in repository https://gitbox.apache.org/repos/asf/cloudberry-pxf.git
commit 24a69b64d3f9185445aa88fe38be0b0d54369cba Author: Dianjin Wang <[email protected]> AuthorDate: Wed Nov 29 15:16:09 2023 +0800 Add basic community files --- .github/ISSUE_TEMPLATE/bug-report.yml | 63 ++++++++++ .github/ISSUE_TEMPLATE/config.yml | 14 +++ .github/pull_request_template.md | 20 +++ CODE-OF-CONDUCT.md | 149 ++++++++++++++-------- CONTRIBUTING.md | 91 +++++++++----- NOTICE | 12 ++ README.md | 226 +++++++++++++++++++--------------- SECURITY.md | 189 ++++++++++------------------ 8 files changed, 457 insertions(+), 307 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml new file mode 100644 index 00000000..ea736c32 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -0,0 +1,63 @@ +name: "\U0001F41B Bug Report" +description: Problems and issues with code in PXF for CloudberryDB. +title: "[Bug] " +labels: ["type: Bug"] +body: + - type: markdown + attributes: + value: "Thank you for reporting the problem! We really appreciate your efforts to improve Cloudberry Database. Before reporting it, please (ideally) test on the latest release or main to see if the issue is fixed." + - type: textarea + attributes: + label: Cloudberry Database and PXF version + description: What version are you using? + - type: textarea + attributes: + label: What happened + description: Describe what happened. + placeholder: > + Please provide the context in which the problem occurred and explain what happened + validations: + required: true + - type: textarea + attributes: + label: What you think should happen instead + description: What do you think went wrong? + placeholder: > + Please explain why you think the behavior is erroneous. It is extremely helpful if you copy&paste the fragment of logs showing the exact error messages or wrong behavior. If you can provide some screenshots or videos, you can include files by dragging and dropping them here. + - type: textarea + attributes: + label: How to reproduce + description: > + What should we do to reproduce the problem? Any SQL commands or operations? + placeholder: > + Please make sure you provide a reproducible step-by-step case of how to reproduce the problem as minimally and precisely as possible. Remember that non-reproducible issues will be closed! Opening a discussion is recommended as a first step. + - type: input + attributes: + label: Operating System + description: What Operating System are you using? + placeholder: "You can get it via `cat /etc/os-release` for example" + - type: textarea + attributes: + label: Anything else + description: Anything else we need to know? + placeholder: > + How often does this problem occur? (Once? Every time? Only when certain conditions are met?) Any relevant logs to include? Put them here. + - type: checkboxes + attributes: + label: Are you willing to submit PR? + description: > + This is absolutely not required, but we are happy to guide you in the contribution process, especially if you already understand how to implement the fix. + options: + - label: Yes, I am willing to submit a PR! + - type: checkboxes + attributes: + label: Code of Conduct + description: The Code of Conduct helps create a safe space for everyone. We require that everyone agrees to it. + options: + - label: > + I agree to follow this project's + [Code of Conduct](https://cloudberrydb.org/community/coc). + required: true + - type: markdown + attributes: + value: "Thanks for completing our form!" diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..a3a57eb1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,14 @@ +blank_issues_enabled: false +contact_links: + - name: ππ» Q&A + url: https://github.com/orgs/cloudberrydb/discussions/categories/q-a + about: Ask for help when running/developing Cloudberry Database. + - name: π‘οΈ New Ideas / Feature Requests + url: https://github.com/orgs/cloudberrydb/discussions/categories/ideas-feature-requests + about: Share ideas for new features. + - name: ποΈ Submit Your Proposal + url: https://github.com/orgs/cloudberrydb/discussions/categories/proposal + about: As a developer, proposing any major change or enhancement to Cloudberry Database. + - name: π¬οΈ Join Slack for Real-time Chat + url: https://communityinviter.com/apps/cloudberrydb/welcome + about: Real-time chat including QA, Dev, Events and more. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..0dfb7e84 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,20 @@ +<!--Thank you for contributing! --> +<!--In case of an existing issue or discussions, please reference it--> +fix #ISSUE_Number +<!--Remove this section if no corresponding issue.--> + +--- + +## Change logs + +> Describe your change clearly, including what problem is being solved or what document is being added or updated. + +## Contributor's checklist + +Here are some reminders before you submit your pull request: + +* Make sure that your Pull Request has a clear title and commit message. You can take the [Git commit template](https://github.com/cloudberrydb/cloudberrydb/blob/main/.gitmessage) as a reference. +* Sign the Contributor License Agreement as prompted for your first-time contribution (*One-time setup*). +* Learn the [code contribution](https://cloudberrydb.org/contribute/code) and [doc contribution](https://cloudberrydb.org/contribute/doc) guides for better collaboration. +* List your communications in the [GitHub Issues](https://github.com/cloudberrydb/cloudberrydb-site/issues) or [Discussions](https://github.com/orgs/cloudberrydb/discussions) (if has or needed). +* Feel free to ask for the cloudberrydb team to help review and approve. diff --git a/CODE-OF-CONDUCT.md b/CODE-OF-CONDUCT.md index 3352843c..922d7f50 100644 --- a/CODE-OF-CONDUCT.md +++ b/CODE-OF-CONDUCT.md @@ -2,79 +2,130 @@ ## Our Pledge -In the interest of fostering an open and welcoming environment, we as -contributors and maintainers pledge to make participation in the Greenplum -Platform Extension Framework project and our community a harassment-free -experience for everyone, regardless of age, body size, disability, ethnicity, -sex characteristics, gender identity and expression, level of experience, -education, socio-economic status, nationality, personal appearance, race, -religion, or sexual identity and orientation. +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, caste, color, religion, or sexual +identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. ## Our Standards -Examples of behavior that contributes to creating a positive environment -include: +Examples of behavior that contributes to a positive environment for our +community include: -* Using welcoming and inclusive language -* Being respectful of differing viewpoints and experiences -* Gracefully accepting constructive criticism -* Focusing on what is best for the community -* Showing empathy towards other community members +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the overall + community -Examples of unacceptable behavior by participants include: +Examples of unacceptable behavior include: -* The use of sexualized language or imagery and unwelcome sexual attention or - advances -* Trolling, insulting/derogatory comments, and personal or political attacks +* The use of sexualized language or imagery, and sexual attention or advances of + any kind +* Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment -* Publishing others' private information, such as a physical or electronic - address, without explicit permission +* Publishing others' private information, such as a physical or email address, + without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting -## Our Responsibilities +## Enforcement Responsibilities -Project maintainers are responsible for clarifying the standards of acceptable -behavior and are expected to take appropriate and fair corrective action in -response to any instances of unacceptable behavior. +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. -Project maintainers have the right and responsibility to remove, edit, or -reject comments, commits, code, wiki edits, issues, and other contributions -that are not aligned to this Code of Conduct, or to ban temporarily or -permanently any contributor for other behaviors that they deem inappropriate, -threatening, offensive, or harmful. +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. ## Scope -This Code of Conduct applies both within project spaces and in public spaces -when an individual is representing the project or its community. Examples of -representing a project or community include using an official project e-mail -address, posting via an official social media account, or acting as an -appointed representative at an online or offline event. Representation of a -project may be further defined and clarified by project maintainers. +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported by contacting the project team at [email protected]. All -complaints will be reviewed and investigated and will result in a response that -is deemed necessary and appropriate to the circumstances. The project team is -obligated to maintain confidentiality with regard to the reporter of an -incident. Further details of specific enforcement policies may be posted -separately. +reported to the community leaders responsible for enforcement at [email protected]. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of +actions. -Project maintainers who do not follow or enforce the Code of Conduct in good -faith may face temporary or permanent repercussions as determined by other -members of the project's leadership. +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or permanent +ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the +community. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], -version 1.4, available at -https://www.contributor-covenant.org/version/1/4/code-of-conduct.html +version 2.1, available at +[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. -[homepage]: https://www.contributor-covenant.org +Community Impact Guidelines were inspired by +[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. -For answers to common questions about this code of conduct, see -https://www.contributor-covenant.org/faq +For answers to common questions about this code of conduct, see the FAQ at +[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at +[https://www.contributor-covenant.org/translations][translations]. +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3c406b6f..518942f4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,49 +1,78 @@ -# Contributing +Cloudberry Database community welcomes contributions from anyone, new and +experienced! We appreciate your interest in contributing. This guide will help +you get started with the contribution. -We warmly welcome and appreciate contributions from the community! -By participating you agree to the [code of conduct](https://github.com/greenplum-db/pxf/blob/main/CODE-OF-CONDUCT.md). -To contribute: +## Code of Conduct -- Sign our [Contributor License Agreement](https://cla.vmware.com/cla/1/preview). +Everyone who participates in Cloudberry Database, either as a user or a +contributor, is obliged to follow our community [Code of +Conduct](./CODE_OF_CONDUCT.md). Every violation against it will be reviewed +and investigated and will result in a response that is deemed necessary and +appropriate to the circumstances. The moderator team is obligated to maintain +confidentiality regarding the reporter of an incident. -- Fork the PXF repository on GitHub. +Some behaviors that contribute to creating a positive environment include: -- Clone the repository. +* Use welcoming and inclusive language. +* Respect differing viewpoints and experiences. +* Accept constructive criticism gracefully. +* Foster what's best for the community. +* Show empathy for community members. -- Follow the README.md to set up your environment and run the tests. +## GitHub Contribution Workflow -- Create a change +1. Fork this repo to your own GitHub account. +2. Clone down the repo to your local system. - - Create a topic branch. +``` +git clone https://github.com/your-user-name/pxf.git +``` - - Make commits as logical units for ease of reviewing. +3. Add the upstream repo. (You only have to do this once, not every time.) - - Follow similar coding styles as found throughout the code base. +``` +git remote add upstream https://github.com/cloudberrydb/pxf.git +``` - - Rebase with main often to stay in sync with upstream. +4. Create a new branch to hold your work. - - Add appropriate unit and automation tests. +``` +git checkout -b new-branch-name +``` - - Ensure a well written commit message as explained [here](https://chris.beams.io/posts/git-commit/) and [here](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). +5. Work on your new code. -- Submit a pull request (PR). +6. Commit your changes. - - Create a [pull request from your fork](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/.creating-a-pull-request-from-a-fork). +``` +git add <the change files> +git commit +``` - - Address PR feedback with fixup and/or squash commits. - ``` - git add . - git commit --fixup <commit SHA> - Or - git commit --squash <commit SHA> - ``` +7. Push your changes to your GitHub repo. - - Once the PR is approved, project committers will merge it to main - branch according to the product release schedule. They might further - squash the commits in the PR if they deem necessary. +``` +git push origin new-branch-name +``` -# Community +8. Open a PR(Pull Request). -Connect with PXF and Greenplum on: -- [Slack](https://greenplum.slack.com/) -- [Dev Google Group mailing list](https://groups.google.com/a/greenplum.org/forum/#!forum/gpdb-dev/join) +Go to the repo on GitHub. There will be a message about your recently pushed +branch, asking if you would like to open a pull request. Follow the prompts, +compare across repositories, and submit the PR. + +9. Get your code reviewed. +10. Congratulations! Once your PR is approved, and passes the CI/CD without +errors, then the code will be merged. Your code will be shipped in the recent +future releases. + +## Sync your branch with the upstream + +Before working on your next contribution, make sure your local repository is +up to date: + +``` +git checkout main +git fetch upstream +git rebase upstream/main +``` diff --git a/NOTICE b/NOTICE index e26d62b0..816927f0 100644 --- a/NOTICE +++ b/NOTICE @@ -1,3 +1,15 @@ +Platform Extension Framework for Cloudberry Database +Copyright 2023 HashData Technology Limited. or its affiliates. All Rights Reserved. + +This product is licensed to you under the Apache 2.0 license (the "License"). +You may not use this product except in compliance with the Apache 2.0 License. + +This product may include a number of subcomponents with separate copyright +notices and license terms. Your use of these subcomponents is subject to the +terms and conditions of the subcomponent's license, as noted in the LICENSE file. + +================================================================= + Greenplum Platform Extension Framework Copyright 2018-Present VMware, Inc. or its affiliates. All Rights Reserved. diff --git a/README.md b/README.md index 82f1510c..6e863f00 100755 --- a/README.md +++ b/README.md @@ -1,73 +1,64 @@ -**PXF Build** [](https://ci.ud.gpdb.pivotal.io/teams/main/pipelines/pxf-build) | -**PXF Certification** [](https://ci.ud.gpdb.pivotal.io/teams/main/pipelines/pxf-certification) +# Platform Extension Framework (PXF) for Cloudberry Database ----------------------------------------------------------------------- +[](https://communityinviter.com/apps/cloudberrydb/welcome) +[](https://twitter.com/cloudberrydb) +[](https://cloudberrydb.org) +[](https://github.com/orgs/cloudberrydb/discussions) -Introduction -============ +--- -PXF is an extensible framework that allows a distributed database like Greenplum to query external data files, whose metadata is not managed by the database. +## Introduction + +PXF is an extensible framework that allows a distributed database like Greenplum and Cloudberry Database to query external data files, whose metadata is not managed by the database. PXF includes built-in connectors for accessing data that exists inside HDFS files, Hive tables, HBase tables, JDBC-accessible databases and more. Users can also create their own connectors to other data storage or processing engines. -Repository Contents -================ -## external-table/ -Contains the Greenplum extension implementing an External Table protocol handler - -## fdw/ -Contains the Greenplum extension implementing a Foreign Data Wrapper (FDW) for PXF - -## server/ -Contains the server side code of PXF along with the PXF Service and all the Plugins - -## cli/ -Contains command line interface code for PXF - -## automation/ -Contains the automation and integration tests for PXF against the various datasources - -## singlecluster/ -Hadoop testing environment to exercise the pxf automation tests - -## concourse/ -Resources for PXF's Continuous Integration pipelines - -## regression/ -Contains the end-to-end (integration) tests for PXF against the various datasources, utilizing the PostgreSQL testing framework `pg_regress` - -## downloads/ -An empty directory that serves as a staging location for Greenplum RPMs for the development Docker image - -PXF Development -================= -Below are the steps to build and install PXF along with its dependencies including Greenplum and Hadoop. - -To start, ensure you have a `~/workspace` directory and have cloned the `pxf` and its prerequisites (shown below) under it. +## Repository Contents + +* `external-table/` + Contains the CloudberryDB extension implementing an External Table protocol handler +* `fdw/` + Contains the CloudberryDB extension implementing a Foreign Data Wrapper (FDW) for PXF +* `server/` + Contains the server side code of PXF along with the PXF Service and all the Plugins +* `cli/` + Contains command line interface code for PXF +* `automation/` + Contains the automation and integration tests for PXF against the various datasources +* `singlecluster/` + Hadoop testing environment to exercise the pxf automation tests +* `regression/` + Contains the end-to-end (integration) tests for PXF against the various datasources, utilizing the PostgreSQL testing framework `pg_regress` +* `downloads/` + An empty directory that serves as a staging location for CloudberryDB RPMs for the development Docker image + +## PXF Development + +Below are the steps to build and install PXF along with its dependencies including CloudberryDB and Hadoop. + +> [!Note] +> To start, ensure you have a `~/workspace` directory and have cloned the `pxf` and its prerequisites (shown below) under it. (The name `workspace` is not strictly required but will be used throughout this guide.) + ```bash mkdir -p ~/workspace cd ~/workspace -git clone https://github.com/greenplum-db/pxf.git -``` -Alternatively, you may create a symlink to your existing repo folder. -```bash -ln -s ~/<git_repos_root> ~/workspace +git clone https://github.com/cloudberrydb/pxf.git ``` -## Install Dependencies +### Install Dependencies To build PXF, you must have: 1. GCC compiler, `make` system, `unzip` package, `maven` for running integration tests -2. Installed Greenplum DB +2. Installed Cloudberry Database - Either download and install Greenplum RPM or build Greenplum from the source by following instructions in the [GPDB README](https://github.com/greenplum-db/gpdb). + Either download and install CloudberryDB RPM or build CloudberryDB from the source by following instructions in the [CloudberryDB](https://github.com/cloudberrydb/cloudberrydb). - Assuming you have installed Greenplum into `/usr/local/greenplum-db` directory, run its environment script: + Assuming you have installed CloudberryDB into `/usr/local/cloudberry-db` directory, run its environment script: ``` - source /usr/local/greenplum-db/greenplum_path.sh + source /usr/local/cloudberry-db/greenplum_path.sh ``` 3. JDK 1.8 or JDK 11 to compile/run @@ -98,8 +89,10 @@ To build PXF, you must have: Note that CentOS 6 provides an older, unsupported version of cURL (7.19). You should install a newer version from source if you are on CentOS 6. -## How to Build PXF +### How to Build PXF + PXF uses Makefiles to build its components. PXF server component uses Gradle that is wrapped into the Makefile for convenience. + ```bash cd ~/workspace/pxf @@ -110,12 +103,12 @@ make make test ``` -## How to Install PXF +### How to Install PXF -To install PXF, first make sure that the user has sufficient permissions in the `$GPHOME` and `$PXF_HOME` directories to perform the installation. It's recommended to change ownership to match the installing user. For example, when installing PXF as user `gpadmin` under `/usr/local/greenplum-db`: +To install PXF, first make sure that the user has sufficient permissions in the `$GPHOME` and `$PXF_HOME` directories to perform the installation. It's recommended to change ownership to match the installing user. For example, when installing PXF as user `gpadmin` under `/usr/local/cloudberry-db`: ```bash -export GPHOME=/usr/local/greenplum-db +export GPHOME=/usr/local/cloudberry-db export PXF_HOME=/usr/local/pxf export PXF_BASE=${HOME}/pxf-base chown -R gpadmin:gpadmin "${GPHOME}" "${PXF_HOME}" @@ -124,22 +117,26 @@ make -C ~/workspace/pxf install NOTE: if `PXF_BASE` is not set, it will default to `PXF_HOME`, and server configurations, libraries or other configurations, might get deleted after a PXF re-install. -## How to Run PXF +### How to Run PXF Ensure that PXF is in your path. This command can be added to your .bashrc + ```bash export PATH=/usr/local/pxf/bin:$PATH ``` Then you can prepare and start up PXF by doing the following. + ```bash pxf prepare pxf start ``` + If `${HOME}/pxf-base` does not exist, `pxf prepare` will create the directory for you. This command should only need to be run once. -## Re-installing PXF after making changes -Note: Local development with PXF requires a running Greenplum cluster. +### Re-installing PXF after making changes + +Note: Local development with PXF requires a running CloudberryDB cluster. Once the desired changes have been made, there are 2 options to re-install PXF: @@ -151,7 +148,7 @@ After PXF has been re-installed, you can restart the PXF instance using: pxf restart ``` -## How to demonstrate Hadoop Integration +### How to demonstrate Hadoop Integration In order to demonstrate end to end functionality you will need Hadoop installed. We have all the related hadoop components (hdfs, hive, hbase, zookeeper, etc) mapped into simple artifact named singlecluster. You can [download from here](https://storage.googleapis.com/pxf-public/singlecluster-HDP.tar.gz) and untar the `singlecluster-HDP.tar.gz` file, which contains everything needed to run Hadoop. @@ -161,7 +158,7 @@ cd ~/workspace tar xzf singlecluster-HDP.tar.gz ``` -Create a symlink using `ln -s ~/workspace/singlecluster-HDP ~/workspace/singlecluster` and then follow the steps in [Setup Hadoop](#Setup-Hadoop). +Create a symlink using `ln -s ~/workspace/singlecluster-HDP ~/workspace/singlecluster` and then follow the steps in [Setup Hadoop](####Setup-Hadoop). While PXF can run on either Java 8 or Java 11, please ensure that you are running Java 8 for hdfs, hadoop, etc. Please set your java version by seting your `JAVA_HOME` to the appropriate location. @@ -175,41 +172,17 @@ Initialize the default server configurations: cp ${PXF_HOME}/templates/*-site.xml ${PXF_BASE}/servers/default ``` -# Development With Docker -NOTE: Since the docker container will house all Single cluster Hadoop, Greenplum and PXF, we recommend that you have at least 4 cpus and 6GB memory allocated to Docker. These settings are available under docker preferences. +### Development With Docker -<!-- TODO: Understand why this only works for 6.6 RPM and not latest GPDB6 --> -The quick and easy is to download the GPDB 6.6 RPM from Github and move it into the `/downloads` folder. Then run `./dev/start.bash` to get a docker image with a running GPDB6, Hadoop cluster and an installed PXF. +> [!Note] +> Since the docker container will house all Single cluster Hadoop, CloudberryDB and PXF, we recommend that you have at least 4 cpus and 6GB memory allocated to Docker. These settings are available under docker preferences. -If you would like more control over the GPDB installation, you can use the steps below. +The quick and easy is to download the CloudberryDB RPM from GitHub and move it into the `/downloads` folder. Then run `./dev/start.bash` to get a docker image with a running CloudberryDB, Hadoop cluster and an installed PXF. -```bash -# Get the latest centos7 image for GPDB6 -docker pull gcr.io/$PROJECT_ID/gpdb-pxf-dev/gpdb6-centos7-test-pxf:latest - -# If you want to use gdb to debug gpdb you need the --privileged flag in the command below -docker run --rm -it \ - -p 5432:5432 \ - -p 5888:5888 \ - -p 8000:8000 \ - -p 5005:5005 \ - -p 8020:8020 \ - -p 9000:9000 \ - -p 9090:9090 \ - -p 50070:50070 \ - -w /home/gpadmin/workspace \ - -v ~/workspace/gpdb:/home/gpadmin/workspace/gpdb \ - -v ~/workspace/pxf:/home/gpadmin/workspace/pxf \ - -v ~/workspace/singlecluster-HDP:/home/gpadmin/workspace/singlecluster \ - gcr.io/$PROJECT_ID/gpdb-pxf-dev/gpdb6-centos7-test-pxf:latest /bin/bash -c \ - "/home/gpadmin/workspace/pxf/dev/set_up_gpadmin_user.bash && /usr/sbin/sshd && su - gpadmin" -``` - -### Setup GPDB in the Docker image +#### Setup GPDB in the Docker image Configure, build and install GPDB. This will be needed only when you use the container for the first time with GPDB source. -<!-- TODO: This may be because we no longer use greenplum-db-devel?--> ```bash ~/workspace/pxf/dev/build_gpdb.bash sudo mkdir /usr/local/greenplum-db-devel @@ -217,7 +190,7 @@ sudo chown gpadmin:gpadmin /usr/local/greenplum-db-devel ~/workspace/pxf/dev/install_gpdb.bash ``` -For subsequent minor changes to GPDB source you can simply do the following: +For subsequent minor changes to CloudberryDB source you can simply do the following: ```bash ~/workspace/pxf/dev/install_gpdb.bash ``` @@ -227,17 +200,18 @@ Run all the instructions below and run GROUP=smoke (in one script): ~/workspace/pxf/dev/smoke_shortcut.sh ``` -Create Greenplum Cluster +Create CloudberryDB Cluster ```bash -source /usr/local/greenplum-db-devel/greenplum_path.sh -make -C ~/workspace/gpdb create-demo-cluster -source ~/workspace/gpdb/gpAux/gpdemo/gpdemo-env.sh +source /usr/local/cloudberrydb-db-devel/greenplum_path.sh +make -C ~/workspace/cbdb create-demo-cluster +source ~/workspace/cbdb/gpAux/gpdemo/gpdemo-env.sh ``` -### Setup Hadoop +#### Setup Hadoop Hdfs will be needed to demonstrate functionality. You can choose to start additional hadoop components (hive/hbase) if you need them. Setup [User Impersonation](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) prior to starting the hadoop components (this allows the `gpadmin` user to access hadoop data). + ```bash ~/workspace/pxf/dev/configure_singlecluster.bash ``` @@ -251,6 +225,7 @@ popd ``` Start other optional components based on your need + ```bash pushd ~/workspace/singlecluster/bin # Start Hive @@ -263,7 +238,7 @@ pushd ~/workspace/singlecluster/bin popd ``` -### Setup Minio (optional) +#### Setup Minio (optional) Minio is an S3-API compatible local storage solution. The development docker image comes with Minio software pre-installed. To start the Minio server, run the following script: ```bash source ~/workspace/pxf/dev/start_minio.bash @@ -272,7 +247,7 @@ After the server starts, you can access Minio UI at `http://localhost:9000` from The script also sets `PROTOCOL=minio` so that the automation framework will use the local Minio server when running S3 automation tests. If later you would like to run Hadoop HDFS tests, unset this variable with `unset PROTOCOL` command. -### Setup PXF +#### Setup PXF Install PXF Server ```bash @@ -289,7 +264,7 @@ Install PXF client (ignore if this is already done) psql -d template1 -c "create extension pxf" ``` -### Run PXF Tests +#### Run PXF Tests All tests use a database named `pxfautomation`. ```bash pushd ~/workspace/pxf/automation @@ -316,7 +291,7 @@ cp ${PXF_HOME}/lib/pxf-hbase-*.jar ~/workspace/singlecluster/hbase/lib/pxf-hbase ~/workspace/singlecluster/bin/start-hbase.sh ``` -### Make Changes to PXF +#### Make Changes to PXF To deploy your changes to PXF in the development environment. @@ -331,7 +306,7 @@ yes | $PXF_HOME/bin/pxf init $PXF_HOME/bin/pxf start ``` -# IDE Setup (IntelliJ) +## IDE Setup (IntelliJ) - Start IntelliJ. Click "Open" and select the directory to which you cloned the `pxf` repo. - Select `File > Project Structure`. @@ -344,7 +319,7 @@ no JDK set for Gradle. Just cancel and retry. It goes away the second time. - Optionally you can replace `${PXF_TMP_DIR}` with `${GPHOME}/pxf/tmp` in `automation/pom.xml` - Select `Tools > Create Command-line Launcher...` to enable starting Intellij with the `idea` command, e.g. `cd ~/workspace/pxf && idea .`. -## Debugging the locally running instance of PXF server using IntelliJ +### Debugging the locally running instance of PXF server using IntelliJ - In IntelliJ, click `Edit Configuration` and add a new one of type `Remote` - Change the name to `PXF Service Boot` @@ -354,6 +329,53 @@ no JDK set for Gradle. Just cancel and retry. It goes away the second time. - Debug the new configuration in IntelliJ - Run a query in GPDB that uses PXF to debug with IntelliJ -# To run a Kerberized Hadoop Cluster +## To run a Kerberized Hadoop Cluster + +### Requirements + +- Download bin_gpdb (from any of the pipelines) +- Download pxf_tarball (from any of the pipelines) + +These instructions allow you to run a Kerberized cluster + +```bash +docker run --rm -it \ + --privileged \ + --hostname c6401.ambari.apache.org \ + -p 5432:5432 \ + -p 5888:5888 \ + -p 8000:8000 \ + -p 8080:8080 \ + -p 8020:8020 \ + -p 9000:9000 \ + -p 9090:9090 \ + -p 50070:50070 \ + -w /home/gpadmin/workspace \ + -v ~/workspace/cbdb:/home/gpadmin/workspace/gpdb_src \ + -v ~/workspace/pxf:/home/gpadmin/workspace/pxf_src \ + -v ~/workspace/singlecluster-HDP:/home/gpadmin/workspace/singlecluster \ + -v ~/Downloads/bin_cbdb:/home/gpadmin/workspace/bin_cbdb \ + -v ~/Downloads/pxf_tarball:/home/gpadmin/workspace/pxf_tarball \ + -e CLUSTER_NAME=hdp \ + -e NODE=c6401.ambari.apache.org \ + -e REALM=AMBARI.APACHE.ORG \ + gcr.io/$PROJECT_ID/gpdb-pxf-dev/gpdb6-centos7-test-pxf-hdp2 /bin/bash + +# Inside the container run the following command: +pxf_src/concourse/scripts/test_pxf_secure.bash + +echo "+----------------------------------------------+" +echo "| Kerberos admin principal: admin/admin@$REALM |" +echo "| Kerberos admin password : admin |" +echo "+----------------------------------------------+" + +su - gpadmin +``` + +## Contribute + +See the [CONTRIBUTING](./CONTRIBUTING.md) file for how to make contributions dedicated to the PXF for Cloudberry Database. + +## License -- See instructions in the dev folder for spinning up a kerberized Dataproc cluster in GCP. \ No newline at end of file +Under Apache License V2.0, See the [LICENSE](./LICENSE) for details. diff --git a/SECURITY.md b/SECURITY.md index fabbb9d2..36b48877 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,125 +1,64 @@ -# Security Release Process - -Greenplum Database has adopted this security disclosure and response policy to -ensure we responsibly handle critical issues. - -## Reporting a Vulnerability - Private Disclosure Process - -Security is of the highest importance and all security vulnerabilities or -suspected security vulnerabilities should be reported to Greenplum Database -privately, to minimize attacks against current users of Greenplum Database -before they are fixed. Vulnerabilities will be investigated and patched on the -next patch (or minor) release as soon as possible. This information could be -kept entirely internal to the project. - -If you know of a publicly disclosed security vulnerability for Greenplum -Database, please **IMMEDIATELY** contact the Greenplum Database project team -([email protected]). - -**IMPORTANT: Do not file public issues on GitHub for security vulnerabilities!** - -To report a vulnerability or a security-related issue, please contact the email -address with the details of the vulnerability. The email will be fielded by the -Greenplum Database project team. Emails will be addressed promptly, including a -detailed plan to investigate the issue and any potential workarounds to perform -in the meantime. Do not report non-security-impacting bugs through this -channel. Use [GitHub issues](https://github.com/greenplum-db/gpdb/issues) -instead. - -## Proposed Email Content - -Provide a descriptive subject line and in the body of the email include the -following information: - -* Basic identity information, such as your name and your affiliation or company. -* Detailed steps to reproduce the vulnerability (POC scripts, screenshots, and - logs are all helpful to us). -* Description of the effects of the vulnerability on Greenplum Database and the - related hardware and software configurations, so that the Greenplum Database - project team can reproduce it. -* How the vulnerability affects Greenplum Database usage and an estimation of - the attack surface, if there is one. -* List other projects or dependencies that were used in conjunction with - Greenplum Database to produce the vulnerability. - -## When to report a vulnerability - -* When you think Greenplum Database has a potential security vulnerability. -* When you suspect a potential vulnerability but you are unsure that it impacts - Greenplum Database. -* When you know of or suspect a potential vulnerability on another project that - is used by Greenplum Database. - -## Patch, Release, and Disclosure - -The Greenplum Database project team will respond to vulnerability reports as -follows: - -1. The Greenplum project team will investigate the vulnerability and determine -its effects and criticality. -2. If the issue is not deemed to be a vulnerability, the Greenplum project team -will follow up with a detailed reason for rejection. -3. The Greenplum project team will initiate a conversation with the reporter -promptly. -4. If a vulnerability is acknowledged and the timeline for a fix is determined, -the Greenplum project team will work on a plan to communicate with the -appropriate community, including identifying mitigating steps that affected -users can take to protect themselves until the fix is rolled out. -5. The Greenplum project team will also create a -[CVSS](https://www.first.org/cvss/specification-document) using the [CVSS -Calculator](https://www.first.org/cvss/calculator/3.0). The Greenplum project -team makes the final call on the calculated CVSS; it is better to move quickly -than making the CVSS perfect. Issues may also be reported to -[Mitre](https://cve.mitre.org/) using this [scoring -calculator](https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator). The CVE will -initially be set to private. -6. The Greenplum project team will work on fixing the vulnerability and perform -internal testing before preparing to roll out the fix. -7. A public disclosure date is negotiated by the Greenplum Database project -team, and the bug submitter. We prefer to fully disclose the bug as soon as -possible once a user mitigation or patch is available. It is reasonable to -delay disclosure when the bug or the fix is not yet fully understood, or the -solution is not well-tested. The timeframe for disclosure is from immediate -(especially if itβs already publicly known) to a few weeks. The Greenplum -Database project team holds the final say when setting a public disclosure -date. -8. Once the fix is confirmed, the Greenplum project team will patch the -vulnerability in the next patch or minor release, and backport a patch release -into earlier supported releases as necessary. Upon release of the patched -version of Greenplum Database, we will follow the **Public Disclosure -Process**. - -## Public Disclosure Process - -The Greenplum project team publishes a [public -advisory](https://github.com/greenplum-db/gpdb/security/advisories?state=published) -to the Greenplum Database community via GitHub. In most cases, additional -communication via Slack, Twitter, mailing lists, blog and other channels will -assist in educating Greenplum Database users and rolling out the patched -release to affected users. - -The Greenplum project team will also publish any mitigating steps users can -take until the fix can be applied to their Greenplum Database instances. - -## Mailing lists - -* Use [email protected] to report security concerns to the Greenplum - Database project team, who uses the list to privately discuss security issues - and fixes prior to disclosure. - -## Confidentiality, integrity and availability - -We consider vulnerabilities leading to the compromise of data confidentiality, -elevation of privilege, or integrity to be our highest priority concerns. -Availability, in particular in areas relating to DoS and resource exhaustion, -is also a serious security concern. The Greenplum Database project team takes -all vulnerabilities, potential vulnerabilities, and suspected vulnerabilities -seriously and will investigate them in an urgent and expeditious manner. - -Note that we do not currently consider the default settings for Greenplum -Database to be secure-by-default. It is necessary for operators to explicitly -configure settings, role based access control, and other resource related -features in Greenplum Database to provide a hardened Greenplum Database -environment. We will not act on any security disclosure that relates to a lack -of safe defaults. Over time, we will work towards improved safe-by-default -configuration, taking into account backwards compatibility. +Thanks for helping make Cloudberry Database safe! + +--- + +## Reporting Security Issues + +To report a security issue, please email +[[email protected]](mailto:[email protected]). This +project follows a 90-day disclosure timeline. We will publish the +[security +advisories](https://github.com/cloudberrydb/cloudberrydb/security/advisories) +via GitHub. + +You should receive a response within 2 weeks. If for some reason you +do not, please follow up via email to ensure we received your original +message. + +Please include the requested information listed below (as much as you +can provide) to help us better understand the nature and scope of the +possible issue: + +* Type of issue (e.g. buffer overflow, SQL injection, cross-site + scripting, etc.) +* Full paths of source file(s) related to the manifestation of the + issue +* The location of the affected source code (tag/branch/commit or + direct URL) +* Any special configuration required to reproduce the issue +* Step-by-step instructions to reproduce the issue +* Proof-of-concept or exploit code (if possible) +* Impact of the issue, including how an attacker might exploit the + issue + +This information will help us triage your report more quickly. + +## Do not + +For better collaboration, we hope you: + +- Do not file public issues on GitHub for security vulnerabilities. +- Do not report non-security-impacting bugs through this channel. If + you have any questions on using, development, please use [GitHub + Issues, Discussions or + Slack](https://github.com/cloudberrydb/cloudberrydb/issues/new/choose) + instead. + +## Handling Process + +Here's an overview of the security issues handling process: + +* The reporter reports the security issues to the Cloudberry Database + team. +* The Cloudberry Database team investigates the report and decides to + accept or reject the report. If our team rejects the report, the + team will explain why to the reporter. If we accept the report, our + team will work privately with the reporter to fix the security + issues. +* Release the new version of the Cloudberry Database that includes the + fix. +* Public the security issues. + +## Preferred Languages + +We prefer all communications to be in English. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
