junyuc25 commented on code in PR #43510:
URL: https://github.com/apache/spark/pull/43510#discussion_r1385868179
##########
.github/workflows/build_and_test.yml:
##########
@@ -1064,7 +1064,7 @@ jobs:
kubectl create clusterrolebinding serviceaccounts-cluster-admin
--clusterrole=cluster-admin --group=system:serviceaccounts || true
kubectl apply -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.8.0/installer/volcano-development.yaml
|| true
eval $(minikube docker-env)
- build/sbt -Psparkr -Pkubernetes -Pvolcano
-Pkubernetes-integration-tests -Dspark.kubernetes.test.driverRequestCores=0.5
-Dspark.kubernetes.test.executorRequestCores=0.2
-Dspark.kubernetes.test.volcanoMaxConcurrencyJobNum=1 -Dtest.exclude.tags=local
"kubernetes-integration-tests/test"
+ build/sbt -Phadoop-3 -Psparkr -Pkubernetes -Pvolcano
-Pkubernetes-integration-tests -Dspark.kubernetes.test.driverRequestCores=0.5
-Dspark.kubernetes.test.executorRequestCores=0.2
-Dspark.kubernetes.test.volcanoMaxConcurrencyJobNum=1 -Dtest.exclude.tags=local
"kubernetes-integration-tests/test"
Review Comment:
Without explicitly activating this profile, I'm seeing compilation issues
during build:
```
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:30:
not found: object software
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:31:
not found: object software
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:32:
not found: object software
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:33:
not found: object software
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:34:
not found: object software
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:309:
not found: type S3Client
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:310:
not found: value AwsBasicCredentials
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:311:
not found: value S3Client
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:312:
not found: value StaticCredentialsProvider
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:314:
not found: value Region
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:323:
not found: value CreateBucketRequest
[ERROR]
/Users/xxx/repositories/Spark-upgrade-k8s/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:342:
not found: value PutObjectRequest
```
In the pom file of this integration test module, hadoop-3 profile is set to
`<activeByDefault>true</activeByDefault>`. This setting means, when another
profile in this pom is activated via -P in command line (i.e. `-Pvolcano` in
this case), the hadoop-3 profile would be deactivated. This behavior is also
explained
[here](https://maven.apache.org/guides/introduction/introduction-to-profiles.html).
Since the `volcano` profile is explicitly activated, we would also need to
explicitly activate `hadoop-3` to address the above errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
