lenoxzhao opened a new pull request, #4133:
URL: https://github.com/apache/incubator-streampark/pull/4133
<!--
Thank you for contributing to StreamPark! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
## Contribution Checklist
- If this is your first time, please read our contributor guidelines:
[Submit Code](https://streampark.apache.org/community/submit_guide/submit_code).
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/incubator-streampark/issues).
- Name the pull request in the form "[Feature] Title of the pull request",
where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- If the PR is unfinished, add `[WIP]` in your PR title, e.g.,
`[WIP][Feature] Title of the pull request`.
-->
## What changes were proposed in this pull request
<!--(For example: This pull request proposed to add checkstyle plugin).-->
## Brief change log
Added `KUBERNETES_NATIVE_CLUSTER` and `KUBERNETES_NATIVE_CLIENT` deployment
modes.
Updated pod template related fields to spark scheme.
cURL Request For Test:
```
curl --location 'http://localhost:10000/spark/app/create' \
--header 'Authorization: {your authorization}' \
--header 'Cookie: JSESSIONID={your cookie}' \
--form 'jobType="1"' \
--form 'appType="4"' \
--form 'deployMode="4"' \
--form 'versionId="1"' \
--form 'appName="test-packaging3"' \
--form 'resourceFrom="2"' \
--form 'jar="example3"' \
--form 'mainClass="org.apache.spark.examples.SparkPi"' \
--form 'teamId="100000"' \
--form 'k8sMasterUrl="k8s://https://127.0.0.1:6443";' \
--form 'k8sContainerImage="apache/spark:v3.2.1"' \
--form 'k8sServiceAccount="spark"' \
--form 'k8sNamespace="default"' \
--form 'k8sDriverPodTemplate="apiVersion: v1
kind: Pod
metadata:
name: test-driver-spark
spec:
initContainers:
- name: install
image: busybox
command:
- wget
- \"-O\"
- \"/var/index.html\"
- \"https://www.baidu.com\"";'
```
Test Result Log:
```
16:13:37.958 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[1/7] running =>
Create building workspace | appName=test-packaging3
16:13:37.960 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Recreate building workspace:
/tmp/streampark/workspace/100004/default | appName=test-packaging3
16:13:37.960 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[1/7] success |
appName=test-packaging3
16:13:37.961 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[2/7] running =>
Export kubernetes pod template | appName=test-packaging3
16:13:37.962 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Export spark podTemplates:
/tmp/streampark/workspace/100004/default/driver-pod-template.yaml |
appName=test-packaging3
16:13:37.962 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[2/7] success |
appName=test-packaging3
16:13:37.963 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[3/7] running =>
Prepare spark job jar | appName=test-packaging3
16:13:37.968 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Prepared spark job jar:
/tmp/streampark/workspace/100004/default/spark-examples_2.12-3.1.2.jar |
appName=test-packaging3
16:13:37.969 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[3/7] success |
appName=test-packaging3
16:13:37.970 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[4/7] running =>
Export spark app dockerfile | appName=test-packaging3
16:13:37.974 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Output spark dockerfile:
/tmp/streampark/workspace/100004/default/Dockerfile, content:
FROM apache/spark:v3.2.1
USER root
RUN mkdir -p $SPARK_HOME/usrlib
COPY spark-examples_2.12-3.1.2.jar
$SPARK_HOME/usrlib/spark-examples_2.12-3.1.2.jar
COPY lib $SPARK_HOME/lib/
| appName=test-packaging3
16:13:37.974 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[4/7] success |
appName=test-packaging3
16:13:37.976 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[5/7] running => Pull
spark app base docker image | appName=test-packaging3
16:13:48.176 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Already pulled docker image from remote
register, imageTag=apache/spark:v3.2.1 | appName=test-packaging3
16:13:48.178 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[5/7] success |
appName=test-packaging3
16:13:48.204 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[6/7] running =>
Build spark app docker image | appName=test-packaging3
16:13:48.371 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Built docker image, imageId=5f694e272ece,
imageTag=crpi-xedl4flgdfs4nujg.cn-shenzhen.personal.cr.aliyuncs.com/streampark_lenox/streampark-sparkjob-default-test-packaging3
| appName=test-packaging3
16:13:48.371 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[6/7] success |
appName=test-packaging3
16:13:48.374 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[7/7] running => Push
spark app docker image | appName=test-packaging3
16:14:02.487 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Already pushed docker image,
imageTag=crpi-xedl4flgdfs4nujg.cn-shenzhen.personal.cr.aliyuncs.com/streampark_lenox/streampark-sparkjob-default-test-packaging3
| appName=test-packaging3
16:14:02.488 [streampark-pipeline-watcher-executor-2] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline step[7/7] success |
appName=test-packaging3
16:14:02.512 [streampark-build-pipeline-executor-1] INFO
org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline
- [StreamPark] [streampark-packer] Building pipeline has finished
successfully. | appName=test-packaging3
```
Command For Image Test:
```
./spark-submit \
--master k8s://https://127.0.0.1:6443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.executor.request.cores=0.1 \
--conf
spark.kubernetes.container.image=crpi-xedl4flgdfs4nujg.cn-shenzhen.personal.cr.aliyuncs.com/streampark_lenox/streampark-sparkjob-default-test-packaging3
\
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf
spark.kubernetes.driver.podTemplateFile={your_local_workspace}/driver-pod-template.yaml
\
local:///opt/spark/usrlib/spark-examples_2.12-3.1.2.jar
```
## Verifying this change
<!--*(Please pick either of the following options)*-->
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added integration tests for end-to-end.*
- *Added *Test to verify the change.*
- *Manually verified the change by testing locally.* -->
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): (yes / no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
