Copilot commented on code in PR #412: URL: https://github.com/apache/incubator-hugegraph-doc/pull/412#discussion_r2444634001
########## content/en/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,518 @@ +--- + +title: "HugeGraph Toolchain Local Testing Guide" +linkTitle: "Toolchain Local Testing" +weight: 7 +--- + +This guide aims to help developers efficiently run tests for the HugeGraph toolchain in a local environment, covering the processes of compiling sub-projects, installing dependent services, running tests, and generating coverage reports. + +## 1. Foreword and Core Concepts + +### 1.1 Core Dependency: HugeGraph Server + +In the testing of the HugeGraph toolchain, **HugeGraph Server is the core dependency for the vast majority of integration and functional tests**. It provides the core services of the graph database, and many components in the toolchain (such as Client, Loader, Hubble, Spark Connector, Tools) need to interact with the Server to perform their functions and be tested. Therefore, a properly configured and running HugeGraph Server is a prerequisite for complete functional testing. This guide will explain how to install/build HugeGraph Server in the sections below. + +### 1.2 Explanation of Test Suite Types + +In the testing of the HugeGraph toolchain, you may encounter the following common types of test suites: + +* **Unit Tests**: + * **Goal**: To verify the correctness of the smallest testable units in the program (usually a single function, method, or class). They typically do not involve external dependencies (like databases, network services, etc.). + +* **API Tests (API Tests / ApiTestSuite)**: + * **Goal**: To verify the correctness, stability, and compliance of the APIs provided by the program. They usually simulate client requests, interact with the server, and check if the API's response data and processing mechanisms meet expectations. + * **Characteristics**: Requires a running server (like HugeGraph Server) to respond to API requests. + +* **Functional Tests (Functional Tests / FuncTestSuite)**: + * **Goal**: To verify that a specific function of a system or component works as required. They are used to simulate user scenarios or business processes, involve the interaction of multiple components, and are end-to-end tests. + * **Characteristics**: They take a relatively long time to execute, require a complete system environment (including all dependent services) to run, and can identify issues at the integration level. + +## 2. Pre-Test Preparation + +### 2.1 System and Software Requirements + +* **Operating System**: Linux or macOS is recommended. For Windows, please use WSL2. +* **JDK**: >= 11. Ensure your `JAVA_HOME` environment variable is correctly configured. +* **Maven**: Version 3.5 or higher is recommended for project building and dependency management. +* **Python**: >= 3.11 (only required for HugeGraph-Hubble related tests). It is recommended to use a virtual environment to manage it and avoid version conflicts. + +### 2.2 Clone the Code Repository + +First, you need to clone the source code repository of the HugeGraph toolchain: + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. Deploying the Test Environment + +Regarding the test environment, since HugeGraph Server is the core dependency for most integration and functional tests, you can refer to the [Community Edition Documentation](https://hugegraph.apache.org/docs/quickstart/hugegraph/hugegraph-server/) for instructions on installing/building HugeGraph-Server. In this testing guide, we will introduce two methods: deployment via script and deployment via Docker. + +**Important Notes:** +* It is recommended to prioritize using a script for local deployment of HugeGraph Server. This method allows you to precisely control the Server version by specifying a Git Commit ID, ensuring high compatibility with your toolchain code version and effectively avoiding test anomalies caused by interface or implementation changes. + +* Docker deployment is more suitable for quickly starting a default-configured HugeGraph Server. However, for fine-grained integration testing, especially when your toolchain code depends on features or fixes from a specific HugeGraph Server version, the version lag or default configuration of the Docker image may cause tests to fail. When there are interface/implementation changes between the toolchain code and HugeGraph Server, the convenience of Docker deployment might lead to test failures. In such cases, it is recommended to fall back to the script deployment method. + +### 3.1 Quick Deployment of Test Environment Using a Script (Recommended) + +This method allows you to compile and install a specific version of HugeGraph Server from the source code, ensuring consistency between the test environment and a specific HugeGraph Server version, which is crucial for reproducing issues or verifying compatibility. + +#### 3.1.1 Variables and Parameters + +* **`$COMMIT_ID`** + * Specifies the Git Commit ID of the HugeGraph Server source code. This variable is used when you need to compile and install a specific version of HugeGraph Server from the source as a test dependency. It ensures consistency between the test environment and the specific HugeGraph Server version, which is crucial for reproducing issues or verifying compatibility. Pass it directly as a parameter to the `install-hugegraph-from-source.sh` script. + +* **`$DB_DATABASE` & `$DB_PASS`** + * Specify the name of the MySQL database and the root user password for the connection used in HugeGraph-Loader's JDBC tests. Providing this database connection information allows the Loader to read and write data correctly. Pass them directly as parameters to the `install-mysql.sh` script. + +#### 3.1.2 Test Processing + +**Install and Start HugeGraph Server** + +If you choose to install manually, you can use the following script to install HugeGraph Server. The script is located in the `/assembly/travis/` directory of any tool's repository. It is used to pull the HugeGraph Server source code from a specified commit ID, compile it, unzip it, and start the service via both http and https. +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`: Specifies the Git Commit ID of HugeGraph Server. +* The default http port is 8080, and the https port is 8443. Please ensure they are not occupied before starting the server. + +**Install and Start Hadoop (HDFS)** (Only required when running HDFS tests for hugegraph-loader): +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**Install and Start MySQL** (Only required when running JDBC tests for hugegraph-loader): +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + +**Health Check** + +```bash +curl http://localhost:8080/graphs +``` +If it returns `{"graphs":["hugegraph"]}`, it means the server is ready to receive requests. + +### 3.2 Using Docker to Deploy the Test Environment + +By using the officially released `hugegraph-server` Docker image, you can quickly start a HugeGraph Server. This method simplifies the setup of the test environment, ensures environmental consistency, and improves test repeatability. **However, please note that the Docker image may not be updated to the latest development version of HugeGraph Server in a timely manner. This means that if your toolchain code depends on the latest interfaces or features of HugeGraph Server, using the Docker image may lead to compatibility issues. In such cases, it is recommended to use the script method to deploy a specific `COMMIT_ID` of HugeGraph Server.** + +#### Quick Start with Docker + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +This quickly starts a HugeGraph server with built-in RocksDB, which meets the requirements for most tests and toolchain components. + +#### Example `docker-compose.yml` File + +The following is an example `docker-compose.yml` file that defines the HugeGraph Server, MySQL, and Hadoop (HDFS) services. You can adjust it according to your actual testing needs. + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # Can be replaced with a specific version, or build your own image + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP port + environment: + # Configure HugeGraph Server parameters as needed, e.g., backend storage + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # If you need to persist data or mount configuration files, add volumes here + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # If you need JDBC tests for hugegraph-loader, you can add the following service + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # Read from environment variable, or use default + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # Read from environment variable, or use default + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # Data persistence + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # If you need Hadoop/HDFS tests for hugegraph-loader, you can add the following services + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### Hadoop Configuration Mounts +📁 `./config/core-site.xml` content: + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 `./config/hdfs-site.xml` content: + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**Explanation**: + +* **`hugegraph-server`**: Uses the `hugegraph/hugegraph:latest` image. You can replace it with a specific version as needed, or if you need to build the Server from source, you can create a custom Dockerfile and reference it here. +* **`mysql`**: Uses the official `mysql:5.7` image. `MYSQL_ROOT_PASSWORD` and `MYSQL_DATABASE` can be passed in via environment variables (`DB_PASS`, `DB_DATABASE`) or use default values. +* **`namenode` and `datanode`** (commented out): If you need to run HDFS tests for HugeGraph-Loader, you can uncomment and configure the Hadoop services. + +#### Starting and Stopping the Docker Environment + +1. **Save `docker-compose.yml`**: Save the above content as a `docker-compose.yml` file, preferably in the root directory of the `hugegraph-toolchain` project or in a separate `docker` directory. + +2. **Start Services**: In the directory where the `docker-compose.yml` file is located, run the following command to start all services: + + ```bash + docker compose up -d + ``` + * The `-d` parameter means running the containers in the background. + +3. **Check Service Status**: You can use the following commands to check the running status of the containers: + + ```bash + docker compose ps + lsof -i:8080 # server port + lsof -i:8020 # hadoop port + lsof -i:3306 # mysql port + ``` + +4. **Stop Services**: After testing is complete, you can stop and remove all containers: + + ```bash + docker compose down + ``` + +## 4. Start Testing + +Generally, the local testing process for each tool is as follows, which will be explained in detail below. + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-4.png" alt="HugeGraph Toolchain Testing Process"> +</div> + + +### 4.1 hugegraph-client Local Testing (Java Version) + +`hugegraph-client` is the Java client library for HugeGraph, used for interacting with HugeGraph Server. Its tests mainly verify the communication and data operations between the client and the server. + +#### 4.1.1 Compile + +First, compile the `hugegraph-client` module: + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`: Specifies to compile only the `hugegraph-client` module. +* `-Dmaven.javadoc.skip=true`: Skips Javadoc generation. +* `-ntp`: No transfer progress. + +#### 4.1.2 Dependent Service Installation + +Follow the instructions in [Deploying the Test Environment](#3-deploying-the-test-environment) to start `hugegraph-server`. + +##### Server Authentication Settings (Authentication tests are not supported for Docker image versions <= 1.5.0) + +Since the client's ApiTest includes authentication tests, you must ensure that the server's password is the same as in the test code; otherwise, data transfer between the client and server will fail. If you use the client's built-in script to install and start the server, you can skip this step. +However, if you start the server using other methods, you must perform the following authentication settings, as the default server does not have them set. For example, enter the container environment with `docker exec -it server bash` to make modifications. + +```bash +# Step 1: Modify the authentication mode +vi conf/rest-server.properties +``` +Change line 23 from `auth.authenticator=` to `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` + +```bash +# Step 2: Set the password +bin/stop-hugegraph.sh +echo -e "pa" | bin/init-store.sh # This script initializes the HugeGraph store and sets default user credentials, including the password for authentication tests +bin/start-hugegraph.sh +``` + +#### 4.1.3 Run Tests + +Go to the `hugegraph-client` module directory and run the tests: + +```bash +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp +mvn test -Dtest=ApiTestSuite -ntp +mvn test -Dtest=FuncTestSuite -ntp +``` + +* The unit test mainly depends on the compilation of `hugegraph-client` itself and is used to test the internal logic of the client. +* Other test modules require a running HugeGraph-Server service. + +### 4.2 hugegraph-loader Local Testing + +`hugegraph-loader` is HugeGraph's data import tool, which supports importing data from various sources. It supports loading data from multiple sources (such as local files, HDFS, relational databases, etc.) into HugeGraph, involving interactions with services like HugeGraph Server, Hadoop, and MySQL. + +#### 4.2.1 Compile + +Compile the `hugegraph-client` and `hugegraph-loader` modules: + +```bash +mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.2.2 Dependent Service Installation (Choose based on test type) + +Follow the instructions in [Deploying the Test Environment](#3-deploying-the-test-environment) to start `hugegraph-server`, `Hadoop (HDFS)` (only required when running HDFS tests), and `MySQL` (only required when running JDBC tests). + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-3.png" alt="HugeGraph Loader Testing Process"> +</div> + +#### 4.2.3 Run Tests + +Go to the `hugegraph-loader` module directory and run the tests. The tests for `hugegraph-loader` are categorized using Maven Profiles: + +```bash +cd hugegraph-loader +mvn test -P unit -ntp +mvn test -P file -ntp +mvn test -P hdfs -ntp +mvn test -P jdbc -ntp +mvn test -P kafka -ntp +``` + +* The unit test mainly depends on the compilation of `hugegraph-loader` itself and is used to test the internal logic of the loader component. +* Other test modules require a running HugeGraph-Server service. + * `hdfs` also requires an available Hadoop (HDFS) environment. + * `jdbc` also requires an available MySQL database. + +**Important Note**: Before running tests for a specific Profile, make sure the corresponding dependent services are started and accessible. + +### 4.3 hugegraph-hubble Backend Local Testing + +`hugegraph-hubble` is the visual management tool for HugeGraph. Its tests include backend unit tests and API tests. + +#### 4.3.1 Compile + +First, install `hugegraph-client` and `hugegraph-loader` (Hubble has an indirect dependency), then compile `hugegraph-hubble`: + +```bash +# First, install hugegraph-client and hugegraph-loader, as Hubble's operation depends on them +mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp +# Then, compile hugegraph-hubble +cd hugegraph-hubble +mvn -e compile -Dmaven.javadoc.skip=true -ntp +``` + +#### 4.3.2 Dependent Service Installation + +Follow the instructions in [Deploying the Test Environment](#3-deploying-the-test-environment) to start `hugegraph-server`. + +**Install Other Hubble Dependencies** + +* **Python Dependencies**: + ```bash + python -m pip install -r hubble-dist/assembly/travis/requirements.txt + ``` + * **Note**: Hubble tests require Python >= 3.11. It is recommended to use a virtual environment: `python -m venv venv && source venv/bin/activate`. + +* **Hubble Packaging**: + ```bash + mvn package -Dmaven.test.skip=true + cd apache-hugegraph-hubble-incubating-${version}/bin/bin + ./start-hubble.sh -d + ./stop-hubble.sh + ``` + Package Hubble, verify that it can start and stop normally to ensure it is built correctly and executable, preparing for subsequent tests. + +#### 4.3.3 Run Tests + +Go to the `hugegraph-hubble` module directory and run the tests: + +```bash +mvn test -P unit-test -pl hugegraph-hubble/hubble-be -ntp # Unit Test +hubble-dist/assembly/travis/run-api-test.sh # API Test +``` + +* The unit test mainly depends on the compilation of `hubble-be` itself and runs the unit tests for the Hubble backend (Java part). +* `run-api-test` requires a running HugeGraph-Server service, as well as the proper functioning of the client and loader. + +**Important Note**: Before running API tests, make sure to install the client and loader, and ensure that both HugeGraph Server and HugeGraph-Hubble services are started and accessible. + +### 4.4 hugegraph-spark-connector Local Testing + +`hugegraph-spark-connector` provides integration capabilities between HugeGraph and Apache Spark. Its tests mainly verify the data connection and operations between Spark and HugeGraph. + +#### 4.4.1 Compile + +Compile the `hugegraph-client` and `hugegraph-spark-connector` modules: + +```bash +mvn install -pl hugegraph-client,hugegraph-spark-connector -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.4.2 Dependent Service Installation + +Follow the instructions in [Deploying the Test Environment](#3-deploying-the-test-environment) to start `hugegraph-server`. + +#### 4.4.3 Run Tests + +Go to the `hugegraph-spark-connector` module directory and run the tests: + +```bash +cd hugegraph-spark-connector +mvn test -ntp +``` + +* Requires a running HugeGraph Server. These tests will connect to HugeGraph Server via Spark. + +### 4.5 hugegraph-tools Local Testing + +`hugegraph-tools` provides a command-line toolset for HugeGraph, used for data management, backup and recovery, etc. Its tests mainly verify the functionality of these tools. + +#### 4.5.1 Compile + +Compile the `hugegraph-client` and `hugegraph-tools` modules: + +```bash +mvn install -pl hugegraph-client,hugegraph-tools -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.5.2 Dependent Service Installation (Choose one) Review Comment: The heading says '(Choose one)' but only one option is presented. This is inconsistent with similar sections like 4.2.2 which says '(Choose based on test type)'. Either remove '(Choose one)' or clarify what the choice refers to. ```suggestion #### 4.5.2 Dependent Service Installation ``` ########## content/cn/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,523 @@ +--- +title: "HugeGraph工具链本地测试指南" +linkTitle: "Toolchain本地测试" +weight: 7 +--- + +本指南旨在帮助开发者高效地在本地环境下运行 HugeGraph 工具链相关测试，涵盖各子项目的编译、依赖服务安装、测试与覆盖率报告生成等流程。 + +## 1. 前言与核心概念 + +### 1.1 核心依赖说明：HugeGraph Server + +在 HugeGraph 工具链的测试中，**HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖**。它提供了图数据库的核心服务，工具链中的许多组件（如 Client、Loader、Hubble、Spark Connector、Tools）都需要与 Server 进行交互才能完成其功能并进行测试。因此，配置好 HugeGraph Server 正常运行是完整进行功能测试的前提，本指南将在下文介绍如何安装/构建HugeGraph Server。 + +### 1.2 测试套件类型解释 + +在 HugeGraph 工具链的测试中，您可能会遇到以下几种常见的测试套件类型： + +* **单元测试 (Unit Tests)**： + * **目标**：验证程序中最小可测试单元（通常是单个函数、方法或类）的正确性。通常不涉及外部依赖（如数据库、网络服务等） + +* **API 测试 (API Tests / ApiTestSuite)**： + * **目标**：验证程序对外提供API的正确性、稳定性和符合性。它们通常模拟客户端请求，与server进行交互，检查 API 的响应数据、处理机制是否符合预期。 + * **特点**：需要一个正在运行的服务端（如 HugeGraph Server）来响应 API 请求。 + + +* **功能测试 (Functional Tests / FuncTestSuite)**： + * **目标**：验证系统或组件的特定功能是否按照需求正常工作。用于模拟用户场景或业务流程，涉及多个组件的交互，是端到端的测试。 + * **特点**：执行时间相对较长，需要完整的系统环境（包括所有依赖服务）来运行，能够发现集成层面的问题。 + +## 2. 测试前准备 + +### 2.1 系统与软件要求 + +* **操作系统**：建议 Linux, macOS。Windows 平台请使用 WSL2。 +* **JDK**：>= 11。确保您的 `JAVA_HOME` 环境变量已正确配置。 +* **Maven**：建议 3.5 及以上。用于项目构建和依赖管理。 +* **Python**：>= 3.11（仅HugeGraph-Hubble 相关测试需用）。建议使用虚拟环境进行管理，以避免版本冲突。 + +### 2.2 克隆代码仓库 + +首先，您需要克隆 HugeGraph 工具链的源代码仓库： + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. 部署测试环境 + +关于测试环境，由于HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖，有关安装/构建 HugeGraph-Server，可参考访问 [社区版文档](https://hugegraph.apache.org/cn/docs/quickstart/hugegraph/hugegraph-server/)。在本测试指南中，我们会介绍通过脚本部署与通过docker部署两种方式。 + +重要提示： +* 推荐优先使用脚本进行本地部署 HugeGraph Server。 这种方式允许您通过指定 Git Commit ID 来精确控制 Server 版本，确保与您的工具链代码版本高度匹配，从而有效避免因接口或实现变动导致测试异常的问题。 + +* Docker 部署方式更适合快速启动一个默认配置的 HugeGraph Server，但在进行精细化的集成测试时，特别是当您的工具链代码依赖于特定 HugeGraph Server 版本的功能或修复时，Docker 镜像的版本滞后或默认配置可能导致测试不通过。当工具链代码与 HugeGraph Server 存在接口/实现变动时，Docker 部署的便捷性可能反而导致测试失败，此时推荐回退到脚本部署方式。 + +### 3.1 使用脚本快速部署测试环境（推荐） + +这种方式允许您从源代码编译和安装特定版本的 HugeGraph Server，确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。 + +#### 3.1.1 变量与参数 + +* **`$COMMIT_ID`** + * 指定 HugeGraph Server 源代码的 Git Commit ID。当您需要从源代码编译和安装特定版本的 HugeGraph Server 作为测试依赖时，会使用此变量,确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。使用时直接接作为参数传递给 install-hugegraph-from-source.sh 脚本。 + +* **`$DB_DATABASE` 与 `$DB_PASS`** + 指定 HugeGraph-Loader 进行 JDBC 测试时所连接的 MySQL 数据库名称与 root 用户密码。请作为参数传递给 `install-mysql.sh` 脚本，供 Loader 正常读写数据。 + +#### 3.1.2 执行流程 + +**安装并启动 HugeGraph Server** + +如果您选择手动安装，可以使用以下脚本来安装 HugeGraph Server。该脚本位于任意工具仓库的`/assembly/travis/` 目录下 +用于从指定 commit id 拉取 HugeGraph Server 源码、编译、解压并分别以 http/https 启动服务 +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`：指定 HugeGraph Server 的 Git Commit ID。 +* 默认http占用端口为8080，https占用端口为8443，请确保其在server启动前未被占用。 + +**安装并启动Hadoop (HDFS)** (仅当运行 hugegraph-loader的HDFS 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**安装并启动MySQL** (仅当运行 hugegraph-loader的JDBC 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + + +**健康性检查** + +```bash +curl http://localhost:8080/graphs +``` +若返回 `{"graphs":["hugegraph"]}`，则表示服务器已准备就绪，可以接收请求。 + +### 3.2 使用 Docker 部署测试环境 + +通过使用官方发布的 hugegraph-server Docker 镜像，您可以快速启动一个 HugeGraph Server。这种方式简化了测试环境的搭建、确保环境一致性并提高测试的可重复性。**然而，请注意，Docker 镜像可能不会及时更新到 HugeGraph Server 的最新开发版本。这意味着如果您的工具链代码依赖于 HugeGraph Server 的最新接口或功能，使用 Docker 镜像可能会导致兼容性问题。在这种情况下，建议使用脚本方式部署特定 `COMMIT_ID` 的 HugeGraph Server。** + +#### docker快速启动 + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +快速启动一个内置了 RocksDB 的 Hugegraph server。满足大部分测试与toolchain组件运行的要求。 + +#### 示例 `docker-compose.yml` 文件 + +以下是一个示例 `docker-compose.yml` 文件，它定义了 HugeGraph Server、MySQL 和 Hadoop (HDFS) 服务。您可以根据实际测试需求进行调整。 + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # 可以替换为特定版本，或构建自己的镜像 + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP 端口 + environment: + # 根据需要配置HugeGraph Server的参数，例如后端存储 + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # 如果需要持久化数据或挂载配置文件，可以在这里添加卷 + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # 如果需要hugegraph-loader的JDBC测试，可以添加以下服务 + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # 从环境变量读取，或使用默认值 + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # 从环境变量读取，或使用默认值 + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # 数据持久化 + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # 如果需要hugegraph-loader的Hadoop/HDFS测试，可以添加以下服务 + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### hadoop配置挂载 +📁 ./config/core-site.xml 内容： + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 ./config/hdfs-site.xml 内容： + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**说明**： + +* **`hugegraph-server`**：使用 `hugegraph/hugegraph:latest` 镜像。您可以根据需要替换为特定版本，或者如果您需要从源代码构建 Server，可以创建一个自定义的 Dockerfile 并在此处引用。 +* **`mysql`**：使用官方 `mysql:5.7` 镜像。`MYSQL_ROOT_PASSWORD` 和 `MYSQL_DATABASE` 可以通过环境变量 (`DB_PASS`, `DB_DATABASE`) 传入，或者使用默认值。 +* **`namenode` 和 `datanode`** (注释掉的部分)：如果您需要运行 HugeGraph-Loader 的 HDFS 测试，可以取消注释并配置 Hadoop 服务。 + + +#### 启动和停止 Docker 环境 + +1. **保存 `docker-compose.yml`**：将上述内容保存为 `docker-compose.yml` 文件，建议放在 `hugegraph-toolchain` 项目的根目录下或一个独立的 `docker` 目录中。 + +2. **启动服务**：在 `docker-compose.yml` 文件所在的目录下，运行以下命令启动所有服务： + + ```bash + docker compose up -d + ``` + * `-d` 参数表示在后台运行容器。 + +3. **检查服务状态**：您可以使用以下命令检查容器的运行状态： + + ```bash + docker compose ps + lsof -i:8080 # server端口 + lsof -i:8020 # hadoop端口 + lsof -i:3306 # mysql端口 + ``` + +4. **停止服务**：测试完成后，您可以停止并移除所有容器： + + ```bash + docker compose down + ``` + +## 4. 开始测试 + +通常来说，各个工具的本地测试大致流程如下，下面将进行细致的说明 + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-2.png" alt="HugeGraph工具链测试流程图"> +</div> + +### 4.1 hugegraph-client 本地测试 (Java 版本) + +`hugegraph-client` 是 HugeGraph 的 Java 客户端库，用于与 HugeGraph Server 进行交互。其测试主要验证客户端与服务端的通信和数据操作。 + +#### 4.1.1 编译 + +首先，编译 `hugegraph-client` 模块： + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`：指定只编译 `hugegraph-client` 模块。 +* `-Dmaven.javadoc.skip=true`：跳过 Javadoc 生成。 +* `-ntp`：不显示传输进度。 + +#### 4.1.2 依赖服务安装 + +按照 [部署测试环境](#3-部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +##### server鉴权设置（docker镜像版本<=1.5.0不支持鉴权测试） + +由于client的ApiTest包含鉴权测试，需确保server的密码与测试代码中相同，否则client与server的数据传递将无法正常进行。若使用client自带的脚本安装并启动server，可跳过此步。 +但若使用其他方式启动，由于默认server并未设置，因此须进行如下鉴权设置。如 `docker exec -it server bash` 进入容器环境进行修改 + +```bash +# 第一步：修改鉴权模式 +vi conf/rest-server.properties +``` +将line 23的 `auth.authenticator=` 修改为 `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` + +```bash +# 第二步：设置密码 +bin/stop-hugegraph.sh +echo -e "pa" | bin/init-store.sh # 此脚本初始化 HugeGraph 存储并设置默认用户凭据，包括用于鉴权测试的密码 +bin/start-hugegraph.sh +``` + +#### 4.1.3 运行测试 + +进入 `hugegraph-client` 模块目录，并运行测试： + +```bash +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp +mvn test -Dtest=ApiTestSuite -ntp +mvn test -Dtest=FuncTestSuite -ntp +``` + +* unit test 主要依赖 `hugegraph-client` 自身的编译, 用于测试客户端内部的逻辑。 +* 其他测试模块都需要依赖一个正在运行的 HugeGraph-Server 服务 + +### 4.2 hugegraph-loader 本地测试 + +`hugegraph-loader` 是 HugeGraph 的数据导入工具，支持从多种数据源导入数据。支持从多种数据源（如本地文件、HDFS、关系型数据库等）加载数据到 HugeGraph 中，涉及与 HugeGraph Server、Hadoop、MySQL 等服务的交互。 + +#### 4.2.1 编译 + +编译 `hugegraph-client` 和 `hugegraph-loader` 模块： + +```bash +mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.2.2 依赖服务安装 (根据测试类型选择) + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server`，`Hadoop (HDFS)` (仅当运行 HDFS 测试时需要)， `MySQL` (仅当运行 JDBC 测试时需要)。 + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-1.png" alt="HugeGraph Loader 测试流程图"> +</div> + +#### 4.2.3 运行测试 + +进入 `hugegraph-loader` 模块目录，并运行测试。`hugegraph-loader` 的测试通过 Maven Profile 进行分类： + +```bash +cd hugegraph-loader +mvn test -P unit -ntp +mvn test -P file -ntp +mvn test -P hdfs -ntp +mvn test -P jdbc -ntp +mvn test -P kafka -ntp +``` + +* unit test 主要依赖 `hugegraph-loader` 自身的编译, 用于测试 loader 组件内部的逻辑。 +* 其他测试模块都需要依赖一个正在运行的 HugeGraph-Server 服务 + * hdfs 还额外依赖 一个可用的 Hadoop (HDFS) 环境; + * jdbc还额外依赖一个可用的 MySQL 数据库。 + + +**重要提示**：运行特定 Profile 的测试前，请务必确保相应的依赖服务已启动并可访问。 + +### 4.3 hugegraph-hubble 后端本地测试 + +`hugegraph-hubble` 是 HugeGraph 的可视化管理工具。其测试包括后端单元测试和 API 测试。 + +#### 4.3.1 编译 + +首先，install `hugegraph-client` 和 `hugegraph-loader` (Hubble 间接依赖)，然后编译 `hugegraph-hubble`： + +```bash +# 首先，安装 hugegraph-client 和 hugegraph-loader，因为 Hubble 运行依赖它们 +mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp +# 然后，编译 hugegraph-hubble +cd hugegraph-hubble +mvn -e compile -Dmaven.javadoc.skip=true -ntp +``` + +#### 4.3.2 依赖服务安装 + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +**安装 Hubble 其他依赖** + +* **Python 依赖**： + ```bash + python -m pip install -r hubble-dist/assembly/travis/requirements.txt + ``` + * **注意**：Hubble 测试需要 Python >= 3.11。建议使用虚拟环境：`python -m venv venv && source venv/bin/activate`。 + +* **Hubble 打包**： + ```bash + mvn package -Dmaven.test.skip=true + cd apache-hugegraph-hubble-incubating-${version}/bin/bin + ./start-hubble.sh -d + ./stop-hubble.sh + ``` + 打包 Hubble，验证其能否正常启动和关闭，确保正确构建并可执行，为后续测试做准备。 + +#### 4.3.3 运行测试 + +进入 `hugegraph-hubble` 模块目录，并运行测试： + +```bash +mvn test -P unit-test -pl hugegraph-hubble/hubble-be -ntp # 单元测试 +hubble-dist/assembly/travis/run-api-test.sh #API测试 +``` + +* unit test 主要依赖 `hubble-be` 自身的编译, 运行 Hubble 后端（Java 部分）的单元测试。 +* run-api-test需要依赖一个正在运行的 HugeGraph-Server 服务，以及client与loader的正常运行。 + + +**重要提示**：运行 API 测试前，请务必完成client与loader的install，并确保 HugeGraph Server 和 HugeGraph-Hubble 服务均已启动并可访问。 + +### 4.4 hugegraph-spark-connector 本地测试 + +`hugegraph-spark-connector` 提供了 HugeGraph 与 Apache Spark 的集成能力。其测试主要验证 Spark 与 HugeGraph 的数据连接和操作。 + +#### 4.4.1 编译 + +编译 `hugegraph-client` 和 `hugegraph-spark-connector` 模块： + +```bash +mvn install -pl hugegraph-client,hugegraph-spark-connector -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.4.2 依赖服务安装 + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +#### 4.4.3 运行测试 + +进入 `hugegraph-spark-connector` 模块目录，并运行测试： + +```bash +cd hugegraph-spark-connector +mvn test -ntp +``` + +* 一个正在运行的 HugeGraph Server。这些测试会通过 Spark 连接 HugeGraph Server。 + +### 4.5 hugegraph-tools 本地测试 + +`hugegraph-tools` 提供了 HugeGraph 的命令行工具集，用于数据管理、备份恢复等操作。其测试主要验证这些工具的功能。 + +#### 4.5.1 编译 + +编译 `hugegraph-client` 和 `hugegraph-tools` 模块： + +```bash +mvn install -pl hugegraph-client,hugegraph-tools -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.5.2 依赖服务安装 (二选一) Review Comment: The heading says '(二选一)' (choose one of two) but only one option is presented. This is inconsistent and confusing. ```suggestion #### 4.5.2 依赖服务安装 ``` ########## content/cn/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,523 @@ +--- +title: "HugeGraph工具链本地测试指南" +linkTitle: "Toolchain本地测试" +weight: 7 +--- + +本指南旨在帮助开发者高效地在本地环境下运行 HugeGraph 工具链相关测试，涵盖各子项目的编译、依赖服务安装、测试与覆盖率报告生成等流程。 + +## 1. 前言与核心概念 + +### 1.1 核心依赖说明：HugeGraph Server + +在 HugeGraph 工具链的测试中，**HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖**。它提供了图数据库的核心服务，工具链中的许多组件（如 Client、Loader、Hubble、Spark Connector、Tools）都需要与 Server 进行交互才能完成其功能并进行测试。因此，配置好 HugeGraph Server 正常运行是完整进行功能测试的前提，本指南将在下文介绍如何安装/构建HugeGraph Server。 + +### 1.2 测试套件类型解释 + +在 HugeGraph 工具链的测试中，您可能会遇到以下几种常见的测试套件类型： + +* **单元测试 (Unit Tests)**： + * **目标**：验证程序中最小可测试单元（通常是单个函数、方法或类）的正确性。通常不涉及外部依赖（如数据库、网络服务等） + +* **API 测试 (API Tests / ApiTestSuite)**： + * **目标**：验证程序对外提供API的正确性、稳定性和符合性。它们通常模拟客户端请求，与server进行交互，检查 API 的响应数据、处理机制是否符合预期。 + * **特点**：需要一个正在运行的服务端（如 HugeGraph Server）来响应 API 请求。 + + +* **功能测试 (Functional Tests / FuncTestSuite)**： + * **目标**：验证系统或组件的特定功能是否按照需求正常工作。用于模拟用户场景或业务流程，涉及多个组件的交互，是端到端的测试。 + * **特点**：执行时间相对较长，需要完整的系统环境（包括所有依赖服务）来运行，能够发现集成层面的问题。 + +## 2. 测试前准备 + +### 2.1 系统与软件要求 + +* **操作系统**：建议 Linux, macOS。Windows 平台请使用 WSL2。 +* **JDK**：>= 11。确保您的 `JAVA_HOME` 环境变量已正确配置。 +* **Maven**：建议 3.5 及以上。用于项目构建和依赖管理。 +* **Python**：>= 3.11（仅HugeGraph-Hubble 相关测试需用）。建议使用虚拟环境进行管理，以避免版本冲突。 + +### 2.2 克隆代码仓库 + +首先，您需要克隆 HugeGraph 工具链的源代码仓库： + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. 部署测试环境 + +关于测试环境，由于HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖，有关安装/构建 HugeGraph-Server，可参考访问 [社区版文档](https://hugegraph.apache.org/cn/docs/quickstart/hugegraph/hugegraph-server/)。在本测试指南中，我们会介绍通过脚本部署与通过docker部署两种方式。 + +重要提示： +* 推荐优先使用脚本进行本地部署 HugeGraph Server。 这种方式允许您通过指定 Git Commit ID 来精确控制 Server 版本，确保与您的工具链代码版本高度匹配，从而有效避免因接口或实现变动导致测试异常的问题。 + +* Docker 部署方式更适合快速启动一个默认配置的 HugeGraph Server，但在进行精细化的集成测试时，特别是当您的工具链代码依赖于特定 HugeGraph Server 版本的功能或修复时，Docker 镜像的版本滞后或默认配置可能导致测试不通过。当工具链代码与 HugeGraph Server 存在接口/实现变动时，Docker 部署的便捷性可能反而导致测试失败，此时推荐回退到脚本部署方式。 + +### 3.1 使用脚本快速部署测试环境（推荐） + +这种方式允许您从源代码编译和安装特定版本的 HugeGraph Server，确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。 + +#### 3.1.1 变量与参数 + +* **`$COMMIT_ID`** + * 指定 HugeGraph Server 源代码的 Git Commit ID。当您需要从源代码编译和安装特定版本的 HugeGraph Server 作为测试依赖时，会使用此变量,确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。使用时直接接作为参数传递给 install-hugegraph-from-source.sh 脚本。 + +* **`$DB_DATABASE` 与 `$DB_PASS`** + 指定 HugeGraph-Loader 进行 JDBC 测试时所连接的 MySQL 数据库名称与 root 用户密码。请作为参数传递给 `install-mysql.sh` 脚本，供 Loader 正常读写数据。 + +#### 3.1.2 执行流程 + +**安装并启动 HugeGraph Server** + +如果您选择手动安装，可以使用以下脚本来安装 HugeGraph Server。该脚本位于任意工具仓库的`/assembly/travis/` 目录下 +用于从指定 commit id 拉取 HugeGraph Server 源码、编译、解压并分别以 http/https 启动服务 +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`：指定 HugeGraph Server 的 Git Commit ID。 +* 默认http占用端口为8080，https占用端口为8443，请确保其在server启动前未被占用。 + +**安装并启动Hadoop (HDFS)** (仅当运行 hugegraph-loader的HDFS 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**安装并启动MySQL** (仅当运行 hugegraph-loader的JDBC 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + + +**健康性检查** + +```bash +curl http://localhost:8080/graphs +``` +若返回 `{"graphs":["hugegraph"]}`，则表示服务器已准备就绪，可以接收请求。 + +### 3.2 使用 Docker 部署测试环境 + +通过使用官方发布的 hugegraph-server Docker 镜像，您可以快速启动一个 HugeGraph Server。这种方式简化了测试环境的搭建、确保环境一致性并提高测试的可重复性。**然而，请注意，Docker 镜像可能不会及时更新到 HugeGraph Server 的最新开发版本。这意味着如果您的工具链代码依赖于 HugeGraph Server 的最新接口或功能，使用 Docker 镜像可能会导致兼容性问题。在这种情况下，建议使用脚本方式部署特定 `COMMIT_ID` 的 HugeGraph Server。** + +#### docker快速启动 + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +快速启动一个内置了 RocksDB 的 Hugegraph server。满足大部分测试与toolchain组件运行的要求。 + +#### 示例 `docker-compose.yml` 文件 + +以下是一个示例 `docker-compose.yml` 文件，它定义了 HugeGraph Server、MySQL 和 Hadoop (HDFS) 服务。您可以根据实际测试需求进行调整。 + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # 可以替换为特定版本，或构建自己的镜像 + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP 端口 + environment: + # 根据需要配置HugeGraph Server的参数，例如后端存储 + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # 如果需要持久化数据或挂载配置文件，可以在这里添加卷 + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # 如果需要hugegraph-loader的JDBC测试，可以添加以下服务 + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # 从环境变量读取，或使用默认值 + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # 从环境变量读取，或使用默认值 + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # 数据持久化 + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # 如果需要hugegraph-loader的Hadoop/HDFS测试，可以添加以下服务 + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### hadoop配置挂载 +📁 ./config/core-site.xml 内容： + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 ./config/hdfs-site.xml 内容： + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**说明**： + +* **`hugegraph-server`**：使用 `hugegraph/hugegraph:latest` 镜像。您可以根据需要替换为特定版本，或者如果您需要从源代码构建 Server，可以创建一个自定义的 Dockerfile 并在此处引用。 +* **`mysql`**：使用官方 `mysql:5.7` 镜像。`MYSQL_ROOT_PASSWORD` 和 `MYSQL_DATABASE` 可以通过环境变量 (`DB_PASS`, `DB_DATABASE`) 传入，或者使用默认值。 +* **`namenode` 和 `datanode`** (注释掉的部分)：如果您需要运行 HugeGraph-Loader 的 HDFS 测试，可以取消注释并配置 Hadoop 服务。 + + +#### 启动和停止 Docker 环境 + +1. **保存 `docker-compose.yml`**：将上述内容保存为 `docker-compose.yml` 文件，建议放在 `hugegraph-toolchain` 项目的根目录下或一个独立的 `docker` 目录中。 + +2. **启动服务**：在 `docker-compose.yml` 文件所在的目录下，运行以下命令启动所有服务： + + ```bash + docker compose up -d + ``` + * `-d` 参数表示在后台运行容器。 + +3. **检查服务状态**：您可以使用以下命令检查容器的运行状态： + + ```bash + docker compose ps + lsof -i:8080 # server端口 + lsof -i:8020 # hadoop端口 + lsof -i:3306 # mysql端口 + ``` + +4. **停止服务**：测试完成后，您可以停止并移除所有容器： + + ```bash + docker compose down + ``` + +## 4. 开始测试 + +通常来说，各个工具的本地测试大致流程如下，下面将进行细致的说明 + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-2.png" alt="HugeGraph工具链测试流程图"> +</div> + +### 4.1 hugegraph-client 本地测试 (Java 版本) + +`hugegraph-client` 是 HugeGraph 的 Java 客户端库，用于与 HugeGraph Server 进行交互。其测试主要验证客户端与服务端的通信和数据操作。 + +#### 4.1.1 编译 + +首先，编译 `hugegraph-client` 模块： + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`：指定只编译 `hugegraph-client` 模块。 +* `-Dmaven.javadoc.skip=true`：跳过 Javadoc 生成。 +* `-ntp`：不显示传输进度。 + +#### 4.1.2 依赖服务安装 + +按照 [部署测试环境](#3-部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +##### server鉴权设置（docker镜像版本<=1.5.0不支持鉴权测试） + Review Comment: [nitpick] Same issue as in the English version: the parenthetical note about Docker image version limitations should be more prominent for better visibility. ```suggestion ##### server鉴权设置 > **注意：** Docker 镜像版本 <= 1.5.0 不支持鉴权测试，请确保使用 1.6.0 及以上版本以运行相关测试。 ``` ########## content/en/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,518 @@ +--- + +title: "HugeGraph Toolchain Local Testing Guide" +linkTitle: "Toolchain Local Testing" +weight: 7 +--- + +This guide aims to help developers efficiently run tests for the HugeGraph toolchain in a local environment, covering the processes of compiling sub-projects, installing dependent services, running tests, and generating coverage reports. + +## 1. Foreword and Core Concepts + +### 1.1 Core Dependency: HugeGraph Server + +In the testing of the HugeGraph toolchain, **HugeGraph Server is the core dependency for the vast majority of integration and functional tests**. It provides the core services of the graph database, and many components in the toolchain (such as Client, Loader, Hubble, Spark Connector, Tools) need to interact with the Server to perform their functions and be tested. Therefore, a properly configured and running HugeGraph Server is a prerequisite for complete functional testing. This guide will explain how to install/build HugeGraph Server in the sections below. + +### 1.2 Explanation of Test Suite Types + +In the testing of the HugeGraph toolchain, you may encounter the following common types of test suites: + +* **Unit Tests**: + * **Goal**: To verify the correctness of the smallest testable units in the program (usually a single function, method, or class). They typically do not involve external dependencies (like databases, network services, etc.). + +* **API Tests (API Tests / ApiTestSuite)**: + * **Goal**: To verify the correctness, stability, and compliance of the APIs provided by the program. They usually simulate client requests, interact with the server, and check if the API's response data and processing mechanisms meet expectations. + * **Characteristics**: Requires a running server (like HugeGraph Server) to respond to API requests. + +* **Functional Tests (Functional Tests / FuncTestSuite)**: + * **Goal**: To verify that a specific function of a system or component works as required. They are used to simulate user scenarios or business processes, involve the interaction of multiple components, and are end-to-end tests. + * **Characteristics**: They take a relatively long time to execute, require a complete system environment (including all dependent services) to run, and can identify issues at the integration level. + +## 2. Pre-Test Preparation + +### 2.1 System and Software Requirements + +* **Operating System**: Linux or macOS is recommended. For Windows, please use WSL2. +* **JDK**: >= 11. Ensure your `JAVA_HOME` environment variable is correctly configured. +* **Maven**: Version 3.5 or higher is recommended for project building and dependency management. +* **Python**: >= 3.11 (only required for HugeGraph-Hubble related tests). It is recommended to use a virtual environment to manage it and avoid version conflicts. + +### 2.2 Clone the Code Repository + +First, you need to clone the source code repository of the HugeGraph toolchain: + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. Deploying the Test Environment + +Regarding the test environment, since HugeGraph Server is the core dependency for most integration and functional tests, you can refer to the [Community Edition Documentation](https://hugegraph.apache.org/docs/quickstart/hugegraph/hugegraph-server/) for instructions on installing/building HugeGraph-Server. In this testing guide, we will introduce two methods: deployment via script and deployment via Docker. + +**Important Notes:** +* It is recommended to prioritize using a script for local deployment of HugeGraph Server. This method allows you to precisely control the Server version by specifying a Git Commit ID, ensuring high compatibility with your toolchain code version and effectively avoiding test anomalies caused by interface or implementation changes. + +* Docker deployment is more suitable for quickly starting a default-configured HugeGraph Server. However, for fine-grained integration testing, especially when your toolchain code depends on features or fixes from a specific HugeGraph Server version, the version lag or default configuration of the Docker image may cause tests to fail. When there are interface/implementation changes between the toolchain code and HugeGraph Server, the convenience of Docker deployment might lead to test failures. In such cases, it is recommended to fall back to the script deployment method. + +### 3.1 Quick Deployment of Test Environment Using a Script (Recommended) + +This method allows you to compile and install a specific version of HugeGraph Server from the source code, ensuring consistency between the test environment and a specific HugeGraph Server version, which is crucial for reproducing issues or verifying compatibility. + +#### 3.1.1 Variables and Parameters + +* **`$COMMIT_ID`** + * Specifies the Git Commit ID of the HugeGraph Server source code. This variable is used when you need to compile and install a specific version of HugeGraph Server from the source as a test dependency. It ensures consistency between the test environment and the specific HugeGraph Server version, which is crucial for reproducing issues or verifying compatibility. Pass it directly as a parameter to the `install-hugegraph-from-source.sh` script. + +* **`$DB_DATABASE` & `$DB_PASS`** + * Specify the name of the MySQL database and the root user password for the connection used in HugeGraph-Loader's JDBC tests. Providing this database connection information allows the Loader to read and write data correctly. Pass them directly as parameters to the `install-mysql.sh` script. + +#### 3.1.2 Test Processing + +**Install and Start HugeGraph Server** + +If you choose to install manually, you can use the following script to install HugeGraph Server. The script is located in the `/assembly/travis/` directory of any tool's repository. It is used to pull the HugeGraph Server source code from a specified commit ID, compile it, unzip it, and start the service via both http and https. +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`: Specifies the Git Commit ID of HugeGraph Server. +* The default http port is 8080, and the https port is 8443. Please ensure they are not occupied before starting the server. + +**Install and Start Hadoop (HDFS)** (Only required when running HDFS tests for hugegraph-loader): +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**Install and Start MySQL** (Only required when running JDBC tests for hugegraph-loader): +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + +**Health Check** + +```bash +curl http://localhost:8080/graphs +``` +If it returns `{"graphs":["hugegraph"]}`, it means the server is ready to receive requests. + +### 3.2 Using Docker to Deploy the Test Environment + +By using the officially released `hugegraph-server` Docker image, you can quickly start a HugeGraph Server. This method simplifies the setup of the test environment, ensures environmental consistency, and improves test repeatability. **However, please note that the Docker image may not be updated to the latest development version of HugeGraph Server in a timely manner. This means that if your toolchain code depends on the latest interfaces or features of HugeGraph Server, using the Docker image may lead to compatibility issues. In such cases, it is recommended to use the script method to deploy a specific `COMMIT_ID` of HugeGraph Server.** + +#### Quick Start with Docker + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +This quickly starts a HugeGraph server with built-in RocksDB, which meets the requirements for most tests and toolchain components. + +#### Example `docker-compose.yml` File + +The following is an example `docker-compose.yml` file that defines the HugeGraph Server, MySQL, and Hadoop (HDFS) services. You can adjust it according to your actual testing needs. + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # Can be replaced with a specific version, or build your own image + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP port + environment: + # Configure HugeGraph Server parameters as needed, e.g., backend storage + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # If you need to persist data or mount configuration files, add volumes here + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # If you need JDBC tests for hugegraph-loader, you can add the following service + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # Read from environment variable, or use default + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # Read from environment variable, or use default + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # Data persistence + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # If you need Hadoop/HDFS tests for hugegraph-loader, you can add the following services + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### Hadoop Configuration Mounts +📁 `./config/core-site.xml` content: + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 `./config/hdfs-site.xml` content: + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**Explanation**: + +* **`hugegraph-server`**: Uses the `hugegraph/hugegraph:latest` image. You can replace it with a specific version as needed, or if you need to build the Server from source, you can create a custom Dockerfile and reference it here. +* **`mysql`**: Uses the official `mysql:5.7` image. `MYSQL_ROOT_PASSWORD` and `MYSQL_DATABASE` can be passed in via environment variables (`DB_PASS`, `DB_DATABASE`) or use default values. +* **`namenode` and `datanode`** (commented out): If you need to run HDFS tests for HugeGraph-Loader, you can uncomment and configure the Hadoop services. + +#### Starting and Stopping the Docker Environment + +1. **Save `docker-compose.yml`**: Save the above content as a `docker-compose.yml` file, preferably in the root directory of the `hugegraph-toolchain` project or in a separate `docker` directory. + +2. **Start Services**: In the directory where the `docker-compose.yml` file is located, run the following command to start all services: + + ```bash + docker compose up -d + ``` + * The `-d` parameter means running the containers in the background. + +3. **Check Service Status**: You can use the following commands to check the running status of the containers: + + ```bash + docker compose ps + lsof -i:8080 # server port + lsof -i:8020 # hadoop port + lsof -i:3306 # mysql port + ``` + +4. **Stop Services**: After testing is complete, you can stop and remove all containers: + + ```bash + docker compose down + ``` + +## 4. Start Testing + +Generally, the local testing process for each tool is as follows, which will be explained in detail below. + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-4.png" alt="HugeGraph Toolchain Testing Process"> +</div> + + +### 4.1 hugegraph-client Local Testing (Java Version) + +`hugegraph-client` is the Java client library for HugeGraph, used for interacting with HugeGraph Server. Its tests mainly verify the communication and data operations between the client and the server. + +#### 4.1.1 Compile + +First, compile the `hugegraph-client` module: + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`: Specifies to compile only the `hugegraph-client` module. +* `-Dmaven.javadoc.skip=true`: Skips Javadoc generation. +* `-ntp`: No transfer progress. + +#### 4.1.2 Dependent Service Installation + +Follow the instructions in [Deploying the Test Environment](#3-deploying-the-test-environment) to start `hugegraph-server`. + +##### Server Authentication Settings (Authentication tests are not supported for Docker image versions <= 1.5.0) + +Since the client's ApiTest includes authentication tests, you must ensure that the server's password is the same as in the test code; otherwise, data transfer between the client and server will fail. If you use the client's built-in script to install and start the server, you can skip this step. +However, if you start the server using other methods, you must perform the following authentication settings, as the default server does not have them set. For example, enter the container environment with `docker exec -it server bash` to make modifications. + +```bash +# Step 1: Modify the authentication mode +vi conf/rest-server.properties +``` +Change line 23 from `auth.authenticator=` to `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` Review Comment: Hard-coding 'line 23' is fragile and may become incorrect if the configuration file format changes. Consider using the property name or a pattern match instruction instead (e.g., 'Find the line containing `auth.authenticator=` and update it to...'). ```suggestion Find the line containing `auth.authenticator=` in `conf/rest-server.properties` and change it to `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` ``` ########## content/en/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,518 @@ +--- + +title: "HugeGraph Toolchain Local Testing Guide" +linkTitle: "Toolchain Local Testing" +weight: 7 +--- + +This guide aims to help developers efficiently run tests for the HugeGraph toolchain in a local environment, covering the processes of compiling sub-projects, installing dependent services, running tests, and generating coverage reports. + +## 1. Foreword and Core Concepts + +### 1.1 Core Dependency: HugeGraph Server + +In the testing of the HugeGraph toolchain, **HugeGraph Server is the core dependency for the vast majority of integration and functional tests**. It provides the core services of the graph database, and many components in the toolchain (such as Client, Loader, Hubble, Spark Connector, Tools) need to interact with the Server to perform their functions and be tested. Therefore, a properly configured and running HugeGraph Server is a prerequisite for complete functional testing. This guide will explain how to install/build HugeGraph Server in the sections below. + +### 1.2 Explanation of Test Suite Types + +In the testing of the HugeGraph toolchain, you may encounter the following common types of test suites: + +* **Unit Tests**: + * **Goal**: To verify the correctness of the smallest testable units in the program (usually a single function, method, or class). They typically do not involve external dependencies (like databases, network services, etc.). + +* **API Tests (API Tests / ApiTestSuite)**: + * **Goal**: To verify the correctness, stability, and compliance of the APIs provided by the program. They usually simulate client requests, interact with the server, and check if the API's response data and processing mechanisms meet expectations. + * **Characteristics**: Requires a running server (like HugeGraph Server) to respond to API requests. + +* **Functional Tests (Functional Tests / FuncTestSuite)**: + * **Goal**: To verify that a specific function of a system or component works as required. They are used to simulate user scenarios or business processes, involve the interaction of multiple components, and are end-to-end tests. + * **Characteristics**: They take a relatively long time to execute, require a complete system environment (including all dependent services) to run, and can identify issues at the integration level. + +## 2. Pre-Test Preparation + +### 2.1 System and Software Requirements + +* **Operating System**: Linux or macOS is recommended. For Windows, please use WSL2. +* **JDK**: >= 11. Ensure your `JAVA_HOME` environment variable is correctly configured. +* **Maven**: Version 3.5 or higher is recommended for project building and dependency management. +* **Python**: >= 3.11 (only required for HugeGraph-Hubble related tests). It is recommended to use a virtual environment to manage it and avoid version conflicts. + +### 2.2 Clone the Code Repository + +First, you need to clone the source code repository of the HugeGraph toolchain: + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. Deploying the Test Environment + +Regarding the test environment, since HugeGraph Server is the core dependency for most integration and functional tests, you can refer to the [Community Edition Documentation](https://hugegraph.apache.org/docs/quickstart/hugegraph/hugegraph-server/) for instructions on installing/building HugeGraph-Server. In this testing guide, we will introduce two methods: deployment via script and deployment via Docker. + +**Important Notes:** +* It is recommended to prioritize using a script for local deployment of HugeGraph Server. This method allows you to precisely control the Server version by specifying a Git Commit ID, ensuring high compatibility with your toolchain code version and effectively avoiding test anomalies caused by interface or implementation changes. + +* Docker deployment is more suitable for quickly starting a default-configured HugeGraph Server. However, for fine-grained integration testing, especially when your toolchain code depends on features or fixes from a specific HugeGraph Server version, the version lag or default configuration of the Docker image may cause tests to fail. When there are interface/implementation changes between the toolchain code and HugeGraph Server, the convenience of Docker deployment might lead to test failures. In such cases, it is recommended to fall back to the script deployment method. + +### 3.1 Quick Deployment of Test Environment Using a Script (Recommended) + +This method allows you to compile and install a specific version of HugeGraph Server from the source code, ensuring consistency between the test environment and a specific HugeGraph Server version, which is crucial for reproducing issues or verifying compatibility. + +#### 3.1.1 Variables and Parameters + +* **`$COMMIT_ID`** + * Specifies the Git Commit ID of the HugeGraph Server source code. This variable is used when you need to compile and install a specific version of HugeGraph Server from the source as a test dependency. It ensures consistency between the test environment and the specific HugeGraph Server version, which is crucial for reproducing issues or verifying compatibility. Pass it directly as a parameter to the `install-hugegraph-from-source.sh` script. + +* **`$DB_DATABASE` & `$DB_PASS`** + * Specify the name of the MySQL database and the root user password for the connection used in HugeGraph-Loader's JDBC tests. Providing this database connection information allows the Loader to read and write data correctly. Pass them directly as parameters to the `install-mysql.sh` script. + +#### 3.1.2 Test Processing + +**Install and Start HugeGraph Server** + +If you choose to install manually, you can use the following script to install HugeGraph Server. The script is located in the `/assembly/travis/` directory of any tool's repository. It is used to pull the HugeGraph Server source code from a specified commit ID, compile it, unzip it, and start the service via both http and https. +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`: Specifies the Git Commit ID of HugeGraph Server. +* The default http port is 8080, and the https port is 8443. Please ensure they are not occupied before starting the server. + +**Install and Start Hadoop (HDFS)** (Only required when running HDFS tests for hugegraph-loader): +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**Install and Start MySQL** (Only required when running JDBC tests for hugegraph-loader): +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + +**Health Check** + +```bash +curl http://localhost:8080/graphs +``` +If it returns `{"graphs":["hugegraph"]}`, it means the server is ready to receive requests. + +### 3.2 Using Docker to Deploy the Test Environment + +By using the officially released `hugegraph-server` Docker image, you can quickly start a HugeGraph Server. This method simplifies the setup of the test environment, ensures environmental consistency, and improves test repeatability. **However, please note that the Docker image may not be updated to the latest development version of HugeGraph Server in a timely manner. This means that if your toolchain code depends on the latest interfaces or features of HugeGraph Server, using the Docker image may lead to compatibility issues. In such cases, it is recommended to use the script method to deploy a specific `COMMIT_ID` of HugeGraph Server.** + +#### Quick Start with Docker + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +This quickly starts a HugeGraph server with built-in RocksDB, which meets the requirements for most tests and toolchain components. + +#### Example `docker-compose.yml` File + +The following is an example `docker-compose.yml` file that defines the HugeGraph Server, MySQL, and Hadoop (HDFS) services. You can adjust it according to your actual testing needs. + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # Can be replaced with a specific version, or build your own image + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP port + environment: + # Configure HugeGraph Server parameters as needed, e.g., backend storage + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # If you need to persist data or mount configuration files, add volumes here + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # If you need JDBC tests for hugegraph-loader, you can add the following service + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # Read from environment variable, or use default + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # Read from environment variable, or use default + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # Data persistence + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # If you need Hadoop/HDFS tests for hugegraph-loader, you can add the following services + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### Hadoop Configuration Mounts +📁 `./config/core-site.xml` content: + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 `./config/hdfs-site.xml` content: + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**Explanation**: + +* **`hugegraph-server`**: Uses the `hugegraph/hugegraph:latest` image. You can replace it with a specific version as needed, or if you need to build the Server from source, you can create a custom Dockerfile and reference it here. +* **`mysql`**: Uses the official `mysql:5.7` image. `MYSQL_ROOT_PASSWORD` and `MYSQL_DATABASE` can be passed in via environment variables (`DB_PASS`, `DB_DATABASE`) or use default values. +* **`namenode` and `datanode`** (commented out): If you need to run HDFS tests for HugeGraph-Loader, you can uncomment and configure the Hadoop services. + +#### Starting and Stopping the Docker Environment + +1. **Save `docker-compose.yml`**: Save the above content as a `docker-compose.yml` file, preferably in the root directory of the `hugegraph-toolchain` project or in a separate `docker` directory. + +2. **Start Services**: In the directory where the `docker-compose.yml` file is located, run the following command to start all services: + + ```bash + docker compose up -d + ``` + * The `-d` parameter means running the containers in the background. + +3. **Check Service Status**: You can use the following commands to check the running status of the containers: + + ```bash + docker compose ps + lsof -i:8080 # server port + lsof -i:8020 # hadoop port + lsof -i:3306 # mysql port + ``` + +4. **Stop Services**: After testing is complete, you can stop and remove all containers: + + ```bash + docker compose down + ``` + +## 4. Start Testing + +Generally, the local testing process for each tool is as follows, which will be explained in detail below. + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-4.png" alt="HugeGraph Toolchain Testing Process"> +</div> + + +### 4.1 hugegraph-client Local Testing (Java Version) + +`hugegraph-client` is the Java client library for HugeGraph, used for interacting with HugeGraph Server. Its tests mainly verify the communication and data operations between the client and the server. + +#### 4.1.1 Compile + +First, compile the `hugegraph-client` module: + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`: Specifies to compile only the `hugegraph-client` module. +* `-Dmaven.javadoc.skip=true`: Skips Javadoc generation. +* `-ntp`: No transfer progress. + +#### 4.1.2 Dependent Service Installation + +Follow the instructions in [Deploying the Test Environment](#3-deploying-the-test-environment) to start `hugegraph-server`. + +##### Server Authentication Settings (Authentication tests are not supported for Docker image versions <= 1.5.0) + Review Comment: [nitpick] The parenthetical note about Docker image version limitations should be presented more prominently, perhaps as a separate note or warning block, to ensure developers are aware of this constraint before starting authentication tests. ```suggestion ##### Server Authentication Settings > **Warning:** Authentication tests are **not supported** for Docker image versions <= 1.5.0. Please ensure you are using a newer version if you intend to run authentication-related tests. ``` ########## content/cn/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,523 @@ +--- +title: "HugeGraph工具链本地测试指南" +linkTitle: "Toolchain本地测试" +weight: 7 +--- + +本指南旨在帮助开发者高效地在本地环境下运行 HugeGraph 工具链相关测试，涵盖各子项目的编译、依赖服务安装、测试与覆盖率报告生成等流程。 + +## 1. 前言与核心概念 + +### 1.1 核心依赖说明：HugeGraph Server + +在 HugeGraph 工具链的测试中，**HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖**。它提供了图数据库的核心服务，工具链中的许多组件（如 Client、Loader、Hubble、Spark Connector、Tools）都需要与 Server 进行交互才能完成其功能并进行测试。因此，配置好 HugeGraph Server 正常运行是完整进行功能测试的前提，本指南将在下文介绍如何安装/构建HugeGraph Server。 + +### 1.2 测试套件类型解释 + +在 HugeGraph 工具链的测试中，您可能会遇到以下几种常见的测试套件类型： + +* **单元测试 (Unit Tests)**： + * **目标**：验证程序中最小可测试单元（通常是单个函数、方法或类）的正确性。通常不涉及外部依赖（如数据库、网络服务等） + +* **API 测试 (API Tests / ApiTestSuite)**： + * **目标**：验证程序对外提供API的正确性、稳定性和符合性。它们通常模拟客户端请求，与server进行交互，检查 API 的响应数据、处理机制是否符合预期。 + * **特点**：需要一个正在运行的服务端（如 HugeGraph Server）来响应 API 请求。 + + +* **功能测试 (Functional Tests / FuncTestSuite)**： + * **目标**：验证系统或组件的特定功能是否按照需求正常工作。用于模拟用户场景或业务流程，涉及多个组件的交互，是端到端的测试。 + * **特点**：执行时间相对较长，需要完整的系统环境（包括所有依赖服务）来运行，能够发现集成层面的问题。 + +## 2. 测试前准备 + +### 2.1 系统与软件要求 + +* **操作系统**：建议 Linux, macOS。Windows 平台请使用 WSL2。 +* **JDK**：>= 11。确保您的 `JAVA_HOME` 环境变量已正确配置。 +* **Maven**：建议 3.5 及以上。用于项目构建和依赖管理。 +* **Python**：>= 3.11（仅HugeGraph-Hubble 相关测试需用）。建议使用虚拟环境进行管理，以避免版本冲突。 + +### 2.2 克隆代码仓库 + +首先，您需要克隆 HugeGraph 工具链的源代码仓库： + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. 部署测试环境 + +关于测试环境，由于HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖，有关安装/构建 HugeGraph-Server，可参考访问 [社区版文档](https://hugegraph.apache.org/cn/docs/quickstart/hugegraph/hugegraph-server/)。在本测试指南中，我们会介绍通过脚本部署与通过docker部署两种方式。 + +重要提示： +* 推荐优先使用脚本进行本地部署 HugeGraph Server。 这种方式允许您通过指定 Git Commit ID 来精确控制 Server 版本，确保与您的工具链代码版本高度匹配，从而有效避免因接口或实现变动导致测试异常的问题。 + +* Docker 部署方式更适合快速启动一个默认配置的 HugeGraph Server，但在进行精细化的集成测试时，特别是当您的工具链代码依赖于特定 HugeGraph Server 版本的功能或修复时，Docker 镜像的版本滞后或默认配置可能导致测试不通过。当工具链代码与 HugeGraph Server 存在接口/实现变动时，Docker 部署的便捷性可能反而导致测试失败，此时推荐回退到脚本部署方式。 + +### 3.1 使用脚本快速部署测试环境（推荐） + +这种方式允许您从源代码编译和安装特定版本的 HugeGraph Server，确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。 + +#### 3.1.1 变量与参数 + +* **`$COMMIT_ID`** + * 指定 HugeGraph Server 源代码的 Git Commit ID。当您需要从源代码编译和安装特定版本的 HugeGraph Server 作为测试依赖时，会使用此变量,确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。使用时直接接作为参数传递给 install-hugegraph-from-source.sh 脚本。 + +* **`$DB_DATABASE` 与 `$DB_PASS`** + 指定 HugeGraph-Loader 进行 JDBC 测试时所连接的 MySQL 数据库名称与 root 用户密码。请作为参数传递给 `install-mysql.sh` 脚本，供 Loader 正常读写数据。 + +#### 3.1.2 执行流程 + +**安装并启动 HugeGraph Server** + +如果您选择手动安装，可以使用以下脚本来安装 HugeGraph Server。该脚本位于任意工具仓库的`/assembly/travis/` 目录下 +用于从指定 commit id 拉取 HugeGraph Server 源码、编译、解压并分别以 http/https 启动服务 +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`：指定 HugeGraph Server 的 Git Commit ID。 +* 默认http占用端口为8080，https占用端口为8443，请确保其在server启动前未被占用。 + +**安装并启动Hadoop (HDFS)** (仅当运行 hugegraph-loader的HDFS 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**安装并启动MySQL** (仅当运行 hugegraph-loader的JDBC 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + + +**健康性检查** + +```bash +curl http://localhost:8080/graphs +``` +若返回 `{"graphs":["hugegraph"]}`，则表示服务器已准备就绪，可以接收请求。 + +### 3.2 使用 Docker 部署测试环境 + +通过使用官方发布的 hugegraph-server Docker 镜像，您可以快速启动一个 HugeGraph Server。这种方式简化了测试环境的搭建、确保环境一致性并提高测试的可重复性。**然而，请注意，Docker 镜像可能不会及时更新到 HugeGraph Server 的最新开发版本。这意味着如果您的工具链代码依赖于 HugeGraph Server 的最新接口或功能，使用 Docker 镜像可能会导致兼容性问题。在这种情况下，建议使用脚本方式部署特定 `COMMIT_ID` 的 HugeGraph Server。** + +#### docker快速启动 + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +快速启动一个内置了 RocksDB 的 Hugegraph server。满足大部分测试与toolchain组件运行的要求。 + +#### 示例 `docker-compose.yml` 文件 + +以下是一个示例 `docker-compose.yml` 文件，它定义了 HugeGraph Server、MySQL 和 Hadoop (HDFS) 服务。您可以根据实际测试需求进行调整。 + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # 可以替换为特定版本，或构建自己的镜像 + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP 端口 + environment: + # 根据需要配置HugeGraph Server的参数，例如后端存储 + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # 如果需要持久化数据或挂载配置文件，可以在这里添加卷 + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # 如果需要hugegraph-loader的JDBC测试，可以添加以下服务 + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # 从环境变量读取，或使用默认值 + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # 从环境变量读取，或使用默认值 + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # 数据持久化 + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # 如果需要hugegraph-loader的Hadoop/HDFS测试，可以添加以下服务 + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### hadoop配置挂载 +📁 ./config/core-site.xml 内容： + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 ./config/hdfs-site.xml 内容： + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**说明**： + +* **`hugegraph-server`**：使用 `hugegraph/hugegraph:latest` 镜像。您可以根据需要替换为特定版本，或者如果您需要从源代码构建 Server，可以创建一个自定义的 Dockerfile 并在此处引用。 +* **`mysql`**：使用官方 `mysql:5.7` 镜像。`MYSQL_ROOT_PASSWORD` 和 `MYSQL_DATABASE` 可以通过环境变量 (`DB_PASS`, `DB_DATABASE`) 传入，或者使用默认值。 +* **`namenode` 和 `datanode`** (注释掉的部分)：如果您需要运行 HugeGraph-Loader 的 HDFS 测试，可以取消注释并配置 Hadoop 服务。 + + +#### 启动和停止 Docker 环境 + +1. **保存 `docker-compose.yml`**：将上述内容保存为 `docker-compose.yml` 文件，建议放在 `hugegraph-toolchain` 项目的根目录下或一个独立的 `docker` 目录中。 + +2. **启动服务**：在 `docker-compose.yml` 文件所在的目录下，运行以下命令启动所有服务： + + ```bash + docker compose up -d + ``` + * `-d` 参数表示在后台运行容器。 + +3. **检查服务状态**：您可以使用以下命令检查容器的运行状态： + + ```bash + docker compose ps + lsof -i:8080 # server端口 + lsof -i:8020 # hadoop端口 + lsof -i:3306 # mysql端口 + ``` + +4. **停止服务**：测试完成后，您可以停止并移除所有容器： + + ```bash + docker compose down + ``` + +## 4. 开始测试 + +通常来说，各个工具的本地测试大致流程如下，下面将进行细致的说明 + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-2.png" alt="HugeGraph工具链测试流程图"> +</div> + +### 4.1 hugegraph-client 本地测试 (Java 版本) + +`hugegraph-client` 是 HugeGraph 的 Java 客户端库，用于与 HugeGraph Server 进行交互。其测试主要验证客户端与服务端的通信和数据操作。 + +#### 4.1.1 编译 + +首先，编译 `hugegraph-client` 模块： + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`：指定只编译 `hugegraph-client` 模块。 +* `-Dmaven.javadoc.skip=true`：跳过 Javadoc 生成。 +* `-ntp`：不显示传输进度。 + +#### 4.1.2 依赖服务安装 + +按照 [部署测试环境](#3-部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +##### server鉴权设置（docker镜像版本<=1.5.0不支持鉴权测试） + +由于client的ApiTest包含鉴权测试，需确保server的密码与测试代码中相同，否则client与server的数据传递将无法正常进行。若使用client自带的脚本安装并启动server，可跳过此步。 +但若使用其他方式启动，由于默认server并未设置，因此须进行如下鉴权设置。如 `docker exec -it server bash` 进入容器环境进行修改 + +```bash +# 第一步：修改鉴权模式 +vi conf/rest-server.properties +``` +将line 23的 `auth.authenticator=` 修改为 `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` Review Comment: Same issue as in the English version: hard-coding 'line 23' is fragile. Consider using a pattern-based instruction instead. ```suggestion 找到 `auth.authenticator=` 这一行，并将其修改为 `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` ``` ########## content/cn/docs/guides/toolchain-local-test.md: ########## @@ -0,0 +1,523 @@ +--- +title: "HugeGraph工具链本地测试指南" +linkTitle: "Toolchain本地测试" +weight: 7 +--- + +本指南旨在帮助开发者高效地在本地环境下运行 HugeGraph 工具链相关测试，涵盖各子项目的编译、依赖服务安装、测试与覆盖率报告生成等流程。 + +## 1. 前言与核心概念 + +### 1.1 核心依赖说明：HugeGraph Server + +在 HugeGraph 工具链的测试中，**HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖**。它提供了图数据库的核心服务，工具链中的许多组件（如 Client、Loader、Hubble、Spark Connector、Tools）都需要与 Server 进行交互才能完成其功能并进行测试。因此，配置好 HugeGraph Server 正常运行是完整进行功能测试的前提，本指南将在下文介绍如何安装/构建HugeGraph Server。 + +### 1.2 测试套件类型解释 + +在 HugeGraph 工具链的测试中，您可能会遇到以下几种常见的测试套件类型： + +* **单元测试 (Unit Tests)**： + * **目标**：验证程序中最小可测试单元（通常是单个函数、方法或类）的正确性。通常不涉及外部依赖（如数据库、网络服务等） + +* **API 测试 (API Tests / ApiTestSuite)**： + * **目标**：验证程序对外提供API的正确性、稳定性和符合性。它们通常模拟客户端请求，与server进行交互，检查 API 的响应数据、处理机制是否符合预期。 + * **特点**：需要一个正在运行的服务端（如 HugeGraph Server）来响应 API 请求。 + + +* **功能测试 (Functional Tests / FuncTestSuite)**： + * **目标**：验证系统或组件的特定功能是否按照需求正常工作。用于模拟用户场景或业务流程，涉及多个组件的交互，是端到端的测试。 + * **特点**：执行时间相对较长，需要完整的系统环境（包括所有依赖服务）来运行，能够发现集成层面的问题。 + +## 2. 测试前准备 + +### 2.1 系统与软件要求 + +* **操作系统**：建议 Linux, macOS。Windows 平台请使用 WSL2。 +* **JDK**：>= 11。确保您的 `JAVA_HOME` 环境变量已正确配置。 +* **Maven**：建议 3.5 及以上。用于项目构建和依赖管理。 +* **Python**：>= 3.11（仅HugeGraph-Hubble 相关测试需用）。建议使用虚拟环境进行管理，以避免版本冲突。 + +### 2.2 克隆代码仓库 + +首先，您需要克隆 HugeGraph 工具链的源代码仓库： + +```bash +git clone https://github.com/${GITHUB_USER_NAME}/hugegraph-toolchain.git +cd hugegraph-toolchain +``` + +## 3. 部署测试环境 + +关于测试环境，由于HugeGraph Server 是绝大多数集成测试和功能测试的核心依赖，有关安装/构建 HugeGraph-Server，可参考访问 [社区版文档](https://hugegraph.apache.org/cn/docs/quickstart/hugegraph/hugegraph-server/)。在本测试指南中，我们会介绍通过脚本部署与通过docker部署两种方式。 + +重要提示： +* 推荐优先使用脚本进行本地部署 HugeGraph Server。 这种方式允许您通过指定 Git Commit ID 来精确控制 Server 版本，确保与您的工具链代码版本高度匹配，从而有效避免因接口或实现变动导致测试异常的问题。 + +* Docker 部署方式更适合快速启动一个默认配置的 HugeGraph Server，但在进行精细化的集成测试时，特别是当您的工具链代码依赖于特定 HugeGraph Server 版本的功能或修复时，Docker 镜像的版本滞后或默认配置可能导致测试不通过。当工具链代码与 HugeGraph Server 存在接口/实现变动时，Docker 部署的便捷性可能反而导致测试失败，此时推荐回退到脚本部署方式。 + +### 3.1 使用脚本快速部署测试环境（推荐） + +这种方式允许您从源代码编译和安装特定版本的 HugeGraph Server，确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。 + +#### 3.1.1 变量与参数 + +* **`$COMMIT_ID`** + * 指定 HugeGraph Server 源代码的 Git Commit ID。当您需要从源代码编译和安装特定版本的 HugeGraph Server 作为测试依赖时，会使用此变量,确保测试环境与特定 HugeGraph Server 版本的一致性，这对于复现问题或验证兼容性至关重要。使用时直接接作为参数传递给 install-hugegraph-from-source.sh 脚本。 + +* **`$DB_DATABASE` 与 `$DB_PASS`** + 指定 HugeGraph-Loader 进行 JDBC 测试时所连接的 MySQL 数据库名称与 root 用户密码。请作为参数传递给 `install-mysql.sh` 脚本，供 Loader 正常读写数据。 + +#### 3.1.2 执行流程 + +**安装并启动 HugeGraph Server** + +如果您选择手动安装，可以使用以下脚本来安装 HugeGraph Server。该脚本位于任意工具仓库的`/assembly/travis/` 目录下 +用于从指定 commit id 拉取 HugeGraph Server 源码、编译、解压并分别以 http/https 启动服务 +```bash +hugegraph-*/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID +``` + +* `$COMMIT_ID`：指定 HugeGraph Server 的 Git Commit ID。 +* 默认http占用端口为8080，https占用端口为8443，请确保其在server启动前未被占用。 + +**安装并启动Hadoop (HDFS)** (仅当运行 hugegraph-loader的HDFS 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-hadoop.sh +``` + +**安装并启动MySQL** (仅当运行 hugegraph-loader的JDBC 测试时需要)： +```bash +hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS +``` + + +**健康性检查** + +```bash +curl http://localhost:8080/graphs +``` +若返回 `{"graphs":["hugegraph"]}`，则表示服务器已准备就绪，可以接收请求。 + +### 3.2 使用 Docker 部署测试环境 + +通过使用官方发布的 hugegraph-server Docker 镜像，您可以快速启动一个 HugeGraph Server。这种方式简化了测试环境的搭建、确保环境一致性并提高测试的可重复性。**然而，请注意，Docker 镜像可能不会及时更新到 HugeGraph Server 的最新开发版本。这意味着如果您的工具链代码依赖于 HugeGraph Server 的最新接口或功能，使用 Docker 镜像可能会导致兼容性问题。在这种情况下，建议使用脚本方式部署特定 `COMMIT_ID` 的 HugeGraph Server。** + +#### docker快速启动 + +```bash +docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:latest +``` + +快速启动一个内置了 RocksDB 的 Hugegraph server。满足大部分测试与toolchain组件运行的要求。 + +#### 示例 `docker-compose.yml` 文件 + +以下是一个示例 `docker-compose.yml` 文件，它定义了 HugeGraph Server、MySQL 和 Hadoop (HDFS) 服务。您可以根据实际测试需求进行调整。 + +```yaml +version: '3.8' + +services: + hugegraph-server: + image: hugegraph/hugegraph:latest # 可以替换为特定版本，或构建自己的镜像 + container_name: hugegraph-server + ports: + - "8080:8080" # HugeGraph Server HTTP 端口 + environment: + # 根据需要配置HugeGraph Server的参数，例如后端存储 + - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb" + volumes: + # 如果需要持久化数据或挂载配置文件，可以在这里添加卷 + # - ./hugegraph-data:/opt/hugegraph/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/graphs || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + networks: + - hugegraph-net + + # 如果需要hugegraph-loader的JDBC测试，可以添加以下服务 + # mysql: + # image: mysql:5.7 + # container_name: mysql-db + # environment: + # MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # 从环境变量读取，或使用默认值 + # MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # 从环境变量读取，或使用默认值 + # ports: + # - "3306:3306" + # volumes: + # - ./mysql-data:/var/lib/mysql # 数据持久化 + # healthcheck: + # test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # 如果需要hugegraph-loader的Hadoop/HDFS测试，可以添加以下服务 + # namenode: + # image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 + # container_name: namenode + # ports: + # - "0.0.0.0:9870:9870" + # - "0.0.0.0:8020:8020" + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_NAMENODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - namenode_data:/hadoop/dfs/name + # command: bash -c "hdfs namenode -format && /entrypoint.sh" + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + + # datanode: + # image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 + # container_name: datanode + # depends_on: + # - namenode + # environment: + # - CLUSTER_NAME=test-cluster + # - HDFS_DATANODE_USER=root + # - HADOOP_CONF_DIR=/hadoop/etc/hadoop + # volumes: + # - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml + # - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml + # - datanode_data:/hadoop/dfs/data + # healthcheck: + # test: ["CMD", "hdfs", "dfsadmin", "-report"] + # interval: 5s + # timeout: 3s + # retries: 5 + # networks: + # - hugegraph-net + +networks: + hugegraph-net: + driver: bridge +``` + +#### hadoop配置挂载 +📁 ./config/core-site.xml 内容： + +```xml +<configuration> + <property> + <name>fs.defaultFS</name> + <value>hdfs://namenode:8020</value> + </property> +</configuration> +``` + +📁 ./config/hdfs-site.xml 内容： + +```xml +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>/opt/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>/opt/hdfs/data</value> + </property> + <property> + <name>dfs.permissions.superusergroup</name> + <value>hadoop</value> + </property> + <property> + <name>dfs.support.append</name> + <value>true</value> + </property> +</configuration> +``` + +**说明**： + +* **`hugegraph-server`**：使用 `hugegraph/hugegraph:latest` 镜像。您可以根据需要替换为特定版本，或者如果您需要从源代码构建 Server，可以创建一个自定义的 Dockerfile 并在此处引用。 +* **`mysql`**：使用官方 `mysql:5.7` 镜像。`MYSQL_ROOT_PASSWORD` 和 `MYSQL_DATABASE` 可以通过环境变量 (`DB_PASS`, `DB_DATABASE`) 传入，或者使用默认值。 +* **`namenode` 和 `datanode`** (注释掉的部分)：如果您需要运行 HugeGraph-Loader 的 HDFS 测试，可以取消注释并配置 Hadoop 服务。 + + +#### 启动和停止 Docker 环境 + +1. **保存 `docker-compose.yml`**：将上述内容保存为 `docker-compose.yml` 文件，建议放在 `hugegraph-toolchain` 项目的根目录下或一个独立的 `docker` 目录中。 + +2. **启动服务**：在 `docker-compose.yml` 文件所在的目录下，运行以下命令启动所有服务： + + ```bash + docker compose up -d + ``` + * `-d` 参数表示在后台运行容器。 + +3. **检查服务状态**：您可以使用以下命令检查容器的运行状态： + + ```bash + docker compose ps + lsof -i:8080 # server端口 + lsof -i:8020 # hadoop端口 + lsof -i:3306 # mysql端口 + ``` + +4. **停止服务**：测试完成后，您可以停止并移除所有容器： + + ```bash + docker compose down + ``` + +## 4. 开始测试 + +通常来说，各个工具的本地测试大致流程如下，下面将进行细致的说明 + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-2.png" alt="HugeGraph工具链测试流程图"> +</div> + +### 4.1 hugegraph-client 本地测试 (Java 版本) + +`hugegraph-client` 是 HugeGraph 的 Java 客户端库，用于与 HugeGraph Server 进行交互。其测试主要验证客户端与服务端的通信和数据操作。 + +#### 4.1.1 编译 + +首先，编译 `hugegraph-client` 模块： + +```bash +mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp +``` + +* `-pl hugegraph-client`：指定只编译 `hugegraph-client` 模块。 +* `-Dmaven.javadoc.skip=true`：跳过 Javadoc 生成。 +* `-ntp`：不显示传输进度。 + +#### 4.1.2 依赖服务安装 + +按照 [部署测试环境](#3-部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +##### server鉴权设置（docker镜像版本<=1.5.0不支持鉴权测试） + +由于client的ApiTest包含鉴权测试，需确保server的密码与测试代码中相同，否则client与server的数据传递将无法正常进行。若使用client自带的脚本安装并启动server，可跳过此步。 +但若使用其他方式启动，由于默认server并未设置，因此须进行如下鉴权设置。如 `docker exec -it server bash` 进入容器环境进行修改 + +```bash +# 第一步：修改鉴权模式 +vi conf/rest-server.properties +``` +将line 23的 `auth.authenticator=` 修改为 `auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator` + +```bash +# 第二步：设置密码 +bin/stop-hugegraph.sh +echo -e "pa" | bin/init-store.sh # 此脚本初始化 HugeGraph 存储并设置默认用户凭据，包括用于鉴权测试的密码 +bin/start-hugegraph.sh +``` + +#### 4.1.3 运行测试 + +进入 `hugegraph-client` 模块目录，并运行测试： + +```bash +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp +mvn test -Dtest=ApiTestSuite -ntp +mvn test -Dtest=FuncTestSuite -ntp +``` + +* unit test 主要依赖 `hugegraph-client` 自身的编译, 用于测试客户端内部的逻辑。 +* 其他测试模块都需要依赖一个正在运行的 HugeGraph-Server 服务 + +### 4.2 hugegraph-loader 本地测试 + +`hugegraph-loader` 是 HugeGraph 的数据导入工具，支持从多种数据源导入数据。支持从多种数据源（如本地文件、HDFS、关系型数据库等）加载数据到 HugeGraph 中，涉及与 HugeGraph Server、Hadoop、MySQL 等服务的交互。 + +#### 4.2.1 编译 + +编译 `hugegraph-client` 和 `hugegraph-loader` 模块： + +```bash +mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.2.2 依赖服务安装 (根据测试类型选择) + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server`，`Hadoop (HDFS)` (仅当运行 HDFS 测试时需要)， `MySQL` (仅当运行 JDBC 测试时需要)。 + +<div style="text-align: center;"> + <img src="/docs/images/toolchain-test-mermaid-1.png" alt="HugeGraph Loader 测试流程图"> +</div> + +#### 4.2.3 运行测试 + +进入 `hugegraph-loader` 模块目录，并运行测试。`hugegraph-loader` 的测试通过 Maven Profile 进行分类： + +```bash +cd hugegraph-loader +mvn test -P unit -ntp +mvn test -P file -ntp +mvn test -P hdfs -ntp +mvn test -P jdbc -ntp +mvn test -P kafka -ntp +``` + +* unit test 主要依赖 `hugegraph-loader` 自身的编译, 用于测试 loader 组件内部的逻辑。 +* 其他测试模块都需要依赖一个正在运行的 HugeGraph-Server 服务 + * hdfs 还额外依赖 一个可用的 Hadoop (HDFS) 环境; + * jdbc还额外依赖一个可用的 MySQL 数据库。 + + +**重要提示**：运行特定 Profile 的测试前，请务必确保相应的依赖服务已启动并可访问。 + +### 4.3 hugegraph-hubble 后端本地测试 + +`hugegraph-hubble` 是 HugeGraph 的可视化管理工具。其测试包括后端单元测试和 API 测试。 + +#### 4.3.1 编译 + +首先，install `hugegraph-client` 和 `hugegraph-loader` (Hubble 间接依赖)，然后编译 `hugegraph-hubble`： + +```bash +# 首先，安装 hugegraph-client 和 hugegraph-loader，因为 Hubble 运行依赖它们 +mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp +# 然后，编译 hugegraph-hubble +cd hugegraph-hubble +mvn -e compile -Dmaven.javadoc.skip=true -ntp +``` + +#### 4.3.2 依赖服务安装 + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +**安装 Hubble 其他依赖** + +* **Python 依赖**： + ```bash + python -m pip install -r hubble-dist/assembly/travis/requirements.txt + ``` + * **注意**：Hubble 测试需要 Python >= 3.11。建议使用虚拟环境：`python -m venv venv && source venv/bin/activate`。 + +* **Hubble 打包**： + ```bash + mvn package -Dmaven.test.skip=true + cd apache-hugegraph-hubble-incubating-${version}/bin/bin + ./start-hubble.sh -d + ./stop-hubble.sh + ``` + 打包 Hubble，验证其能否正常启动和关闭，确保正确构建并可执行，为后续测试做准备。 + +#### 4.3.3 运行测试 + +进入 `hugegraph-hubble` 模块目录，并运行测试： + +```bash +mvn test -P unit-test -pl hugegraph-hubble/hubble-be -ntp # 单元测试 +hubble-dist/assembly/travis/run-api-test.sh #API测试 +``` + +* unit test 主要依赖 `hubble-be` 自身的编译, 运行 Hubble 后端（Java 部分）的单元测试。 +* run-api-test需要依赖一个正在运行的 HugeGraph-Server 服务，以及client与loader的正常运行。 + + +**重要提示**：运行 API 测试前，请务必完成client与loader的install，并确保 HugeGraph Server 和 HugeGraph-Hubble 服务均已启动并可访问。 + +### 4.4 hugegraph-spark-connector 本地测试 + +`hugegraph-spark-connector` 提供了 HugeGraph 与 Apache Spark 的集成能力。其测试主要验证 Spark 与 HugeGraph 的数据连接和操作。 + +#### 4.4.1 编译 + +编译 `hugegraph-client` 和 `hugegraph-spark-connector` 模块： + +```bash +mvn install -pl hugegraph-client,hugegraph-spark-connector -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.4.2 依赖服务安装 + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +#### 4.4.3 运行测试 + +进入 `hugegraph-spark-connector` 模块目录，并运行测试： + +```bash +cd hugegraph-spark-connector +mvn test -ntp +``` + +* 一个正在运行的 HugeGraph Server。这些测试会通过 Spark 连接 HugeGraph Server。 + +### 4.5 hugegraph-tools 本地测试 + +`hugegraph-tools` 提供了 HugeGraph 的命令行工具集，用于数据管理、备份恢复等操作。其测试主要验证这些工具的功能。 + +#### 4.5.1 编译 + +编译 `hugegraph-client` 和 `hugegraph-tools` 模块： + +```bash +mvn install -pl hugegraph-client,hugegraph-tools -am -Dmaven.javadoc.skip=true -DskipTests -ntp +``` + +#### 4.5.2 依赖服务安装 (二选一) + +按照 [部署测试环境](#部署测试环境) 部署测试环境 中的说明，启动 `hugegraph-server` 。 + +#### 4.5.3 运行测试 + +进入 `hugegraph-tools` 模块目录，并运行功能测试： + +```bash +cd hugegraph-tools +mvn test -Dtest=FuncTestSuite -pl hugegraph-tools -ntp +``` + +* 依赖一个正在运行的 HugeGraph Server 和 `hugegraph-client` 的正常编译。 + + +## 5. 常见问题与故障排除 + +本节列举了在 HugeGraph 工具链本地测试过程中可能遇到的一些常见问题及其排查方法。 + +* **服务未启动或端口冲突**： + * **问题描述**：测试失败，提示无法连接到 HugeGraph Server、MySQL 或其他依赖服务。 + * **排查方法**： + * 确认所有必要的依赖服务（HugeGraph Server、MySQL、Hadoop 等）已正确启动，且必须确保server的http服务运行在8080端口。 + * 检查服务监听的端口是否与测试配置一致，并且没有被其他程序占用。您可以使用 `lsof -i:<端口号>` (Linux/macOS) 或 `netstat -ano | findstr :<端口号>` (Windows) 来检查端口占用情况。 + * 如果使用 Docker，请检查 `docker compose ps` 输出，确保所有容器都处于 `Up` 状态，并检查容器日志 (`docker compose logs <service_name>`)。 + +* **环境变量或参数配置错误**： + * **问题描述**：命令执行失败，提示找不到文件、权限不足或参数无效。 + * **排查方法**： + * 仔细检查您设置的环境变量（如 `$COMMIT_ID`, `$DB_DATABASE`, `$DB_PASS`）是否正确，并且在执行命令的 shell 会话中已生效。 + * 确认 Maven 命令参数和 Shell 脚本参数的拼写和用法是否正确，参考 参考 [3.1.1 变量与参数](#3-1-1-变量与参数) 章节。 章节。 Review Comment: The text contains duplicated words: '参考 参考' (reference reference) and '章节。 章节。' (section. section.). Remove the duplicates. ```suggestion * 确认 Maven 命令参数和 Shell 脚本参数的拼写和用法是否正确，参考 [3.1.1 变量与参数](#3-1-1-变量与参数) 章节。 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
