PHILO-HE commented on code in PR #10793: URL: https://github.com/apache/incubator-gluten/pull/10793#discussion_r2386550339
########## docs/developers/NewToGluten.md: ########## @@ -4,158 +4,123 @@ title: New To Gluten nav_order: 2 parent: Developer Overview --- -Help users to debug and test with Gluten. -# Environment +# Guide for New Developers -Gluten supports Ubuntu20.04, Ubuntu22.04, CentOS8, CentOS7 and MacOS. +## Environment -## JDK +Gluten supports Ubuntu 20.04/22.04, CentOS 7/8, and MacOS. -Currently, Gluten supports JDK 8 for Spark 3.2/3.3/3.4/3.5. For Spark 3.3 and higher versions, Gluten -supports JDK 11 and 17. Please note since Spark 4.0, JDK 8 will not be supported. So we recommend Velox -backend users to use higher JDK version now to ease the migration for deploying Gluten with Spark-4.0 -in the future. And we may probably upgrade Arrow from 15.0.0 to some higher version, which also requires -JDK 11 is the minimum version. +### JDK -### JDK 8 +Currently, Gluten supports JDK 8 for Spark 3.2, 3.3, 3.4, and 3.5. For Spark 3.3 and later versions, Gluten +also supports JDK 11 and 17. -#### Environment Setting +Note: Starting with Spark 4.0, the minimum required JDK version is 17. -For root user, the environment variables file is `/etc/profile`, it will take effect for all the users. +We recommend using a higher JDK version now to ease migration when deploying Gluten for Spark 4.0 +in the future. In addition, we may upgrade Arrow from 15.0.0 to a newer release, which will require +JDK 11 as the minimum version. -For other user, you can set in `~/.bashrc`. +By default, Gluten compiles packages using JDK 8. Enable maven profile by `-Pjava-17` or `-Pjava-11` to use the corresponding JDK version, and ensure that the JDK version is available in your environment. -#### Guide for Ubuntu - -The default JDK version in ubuntu is java11, we need to set to java8. - -```bash -apt install openjdk-8-jdk -update-alternatives --config java -java -version -``` - -`--config java` to config java executable path, `javac` and other commands can also use this command to config. -For some other uses, we suggest to set `JAVA_HOME`. - -```bash -export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ -JRE_HOME=$JAVA_HOME/jre -export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar -# pay attention to $PATH double quote -export PATH="$PATH:$JAVA_HOME/bin" -``` - -> Must set PATH with double quote in ubuntu. - -### JDK 11/17 - -By default, Gluten compiles package using JDK8. Enable maven profile by `-Pjava-17` to use JDK17 or `-Pjava-11` to use JDK 11, and please make sure your JAVA_HOME is set correctly. - -Apache Spark and Arrow requires setting java args `-Dio.netty.tryReflectionSetAccessible=true`, see [SPARK-29924](https://issues.apache.org/jira/browse/SPARK-29924) and [ARROW-6206](https://issues.apache.org/jira/browse/ARROW-6206). -So please add following configs in `spark-defaults.conf`: +If JDK 11 or a higher version is used, Spark and Arrow require setting the java args `-Dio.netty.tryReflectionSetAccessible=true`, see [SPARK-29924](https://issues.apache.org/jira/browse/SPARK-29924) and [ARROW-6206](https://issues.apache.org/jira/browse/ARROW-6206). +Add the following configs in `spark-defaults.conf`: ``` spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true ``` -## Maven 3.6.3 or above - -[Maven Download Page](https://maven.apache.org/docs/history.html) -And then set the environment setting. +### Maven 3.6.3 or above -## GCC 11 or above +### GCC 11 or above -# Compile Gluten using debug mode +## Development -If you want to just debug java/scala code, there is no need to compile cpp code with debug mode. -You can just refer to [build-gluten-with-velox-backend](../get-started/Velox.md#build-gluten-with-velox-backend). +To debug Java/Scala code, follow the steps in [build-gluten-with-velox-backend](../get-started/Velox.md#build-gluten-with-velox-backend). -If you need to debug cpp code, please compile the backend code and gluten cpp code with debug mode. +To debug C++ code, compile the backend code and gluten C++ code in debug mode. ```bash ## compile Velox backend with benchmark and tests to debug gluten_home/dev/builddeps-veloxbe.sh --build_tests=ON --build_benchmarks=ON --build_type=Debug ``` -If you need to debug the tests in <gluten>/gluten-ut, You need to compile java code with `-P spark-ut`. +Note: To debug the tests in <gluten>/gluten-ut, you must compile java code with `-Pspark-ut`. -# Java/scala code development with Intellij +### Java/scala code development -## Linux IntelliJ local debug +#### Linux IntelliJ local debug Install the Linux IntelliJ version, and debug code locally. - Ask your linux maintainer to install the desktop, and then restart the server. -- If you use Moba-XTerm to connect linux server, you don't need to install x11 server, If not (e.g. putty), please follow this guide: -[X11 Forwarding: Setup Instructions for Linux and Mac](https://www.businessnewsdaily.com/11035-how-to-use-x11-forwarding.html) +- If you use Moba-XTerm to connect, you don't need to install x11 server. If you are using another tool, such as putty, follow this guide: + [X11 Forwarding: Setup Instructions for Linux and Mac](https://www.businessnewsdaily.com/11035-how-to-use-x11-forwarding.html) -- Download [IntelliJ Linux community version](https://www.jetbrains.com/idea/download/?fromIDE=#section=linux) to Linux server -- Start Idea, `bash <idea_dir>/idea.sh` +- Download [IntelliJ Linux community version](https://www.jetbrains.com/idea/download/?fromIDE=#section=linux) to Linux server. +- Start Idea using the following command: + `bash <idea_dir>/idea.sh` Review Comment: It makes sense. I've updated. Thanks. ########## docs/developers/NewToGluten.md: ########## @@ -4,158 +4,123 @@ title: New To Gluten nav_order: 2 parent: Developer Overview --- -Help users to debug and test with Gluten. -# Environment +# Guide for New Developers -Gluten supports Ubuntu20.04, Ubuntu22.04, CentOS8, CentOS7 and MacOS. +## Environment -## JDK +Gluten supports Ubuntu 20.04/22.04, CentOS 7/8, and MacOS. -Currently, Gluten supports JDK 8 for Spark 3.2/3.3/3.4/3.5. For Spark 3.3 and higher versions, Gluten -supports JDK 11 and 17. Please note since Spark 4.0, JDK 8 will not be supported. So we recommend Velox -backend users to use higher JDK version now to ease the migration for deploying Gluten with Spark-4.0 -in the future. And we may probably upgrade Arrow from 15.0.0 to some higher version, which also requires -JDK 11 is the minimum version. +### JDK -### JDK 8 +Currently, Gluten supports JDK 8 for Spark 3.2, 3.3, 3.4, and 3.5. For Spark 3.3 and later versions, Gluten +also supports JDK 11 and 17. -#### Environment Setting +Note: Starting with Spark 4.0, the minimum required JDK version is 17. -For root user, the environment variables file is `/etc/profile`, it will take effect for all the users. +We recommend using a higher JDK version now to ease migration when deploying Gluten for Spark 4.0 +in the future. In addition, we may upgrade Arrow from 15.0.0 to a newer release, which will require +JDK 11 as the minimum version. -For other user, you can set in `~/.bashrc`. +By default, Gluten compiles packages using JDK 8. Enable maven profile by `-Pjava-17` or `-Pjava-11` to use the corresponding JDK version, and ensure that the JDK version is available in your environment. -#### Guide for Ubuntu - -The default JDK version in ubuntu is java11, we need to set to java8. - -```bash -apt install openjdk-8-jdk -update-alternatives --config java -java -version -``` - -`--config java` to config java executable path, `javac` and other commands can also use this command to config. -For some other uses, we suggest to set `JAVA_HOME`. - -```bash -export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ -JRE_HOME=$JAVA_HOME/jre -export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar -# pay attention to $PATH double quote -export PATH="$PATH:$JAVA_HOME/bin" -``` - -> Must set PATH with double quote in ubuntu. - -### JDK 11/17 - -By default, Gluten compiles package using JDK8. Enable maven profile by `-Pjava-17` to use JDK17 or `-Pjava-11` to use JDK 11, and please make sure your JAVA_HOME is set correctly. - -Apache Spark and Arrow requires setting java args `-Dio.netty.tryReflectionSetAccessible=true`, see [SPARK-29924](https://issues.apache.org/jira/browse/SPARK-29924) and [ARROW-6206](https://issues.apache.org/jira/browse/ARROW-6206). -So please add following configs in `spark-defaults.conf`: +If JDK 11 or a higher version is used, Spark and Arrow require setting the java args `-Dio.netty.tryReflectionSetAccessible=true`, see [SPARK-29924](https://issues.apache.org/jira/browse/SPARK-29924) and [ARROW-6206](https://issues.apache.org/jira/browse/ARROW-6206). +Add the following configs in `spark-defaults.conf`: ``` spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true ``` -## Maven 3.6.3 or above - -[Maven Download Page](https://maven.apache.org/docs/history.html) -And then set the environment setting. +### Maven 3.6.3 or above Review Comment: Yes, I agree. It looks better after the update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
