Yes, it looks inside the docker containers folder. It will work if you are
using s3 og gs.
ons. 12. apr. 2023, 18:02 skrev Mich Talebzadeh <mich.talebza...@gmail.com>:
> Hi,
>
> In my spark-submit to eks cluster, I use the standard code to submit to
> the cluster as below:
>
> spark-submit --verbose \
> --master k8s://$KUBERNETES_MASTER_IP:443 \
> --deploy-mode cluster \
> --name sparkOnEks \
> --py-files local://$CODE_DIRECTORY/spark_on_eks.zip \
> local:///home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py
>
> In Google Kubernetes Engine (GKE) I simply load them from gs:// storage
> bucket.and it works fine.
>
> I am getting the following error in driver pod
>
> + CMD=("$SPARK_HOME/bin/spark-submit" --conf
> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
> "$@")
> + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf
> spark.driver.bindAddress=192.168.39.251 --deploy-mode client
> --properties-file /opt/spark/conf/spark.properties --class
> org.apache.spark.deploy.PythonRunner
> local:///home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py
> 23/04/11 23:07:23 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> /usr/bin/python3: can't open file
> '/home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py': [Errno
> 2] No such file or directory
> log4j:WARN No appenders could be found for logger
> (org.apache.spark.util.ShutdownHookManager).
> It says can't open file
> '/home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py':
>
>
> [Errno 2] No such file or directory but it is there!
>
> ls -l /home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py
> -rw-rw-rw- 1 hduser hadoop 5060 Mar 18 14:16
> /home/hduser/dba/bin/python/spark_on_eks/src/RandomDataBigQuery.py
> So not sure what is going on. I have suspicion that it is looking inside the
> docker itself for this file?
>
>
> Is that a correct assumption?
>
>
> Thanks
>
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
