[
https://issues.apache.org/jira/browse/HUDI-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677076#comment-17677076
]
kazdy commented on HUDI-5273:
-----------------------------
Hi [~jonvex]
I started working on this, wanted to use the existing hudi docker image as a
base, but it seems like it comes with a pretty old Debian distribution and
therefore with Python 3.5.3 alongside Spark 3.2.1. When running "pyspark" in
shell within container, I'm getting:
{code:java}
root@e9fb3f81bdc9:/opt# pyspark
Python 3.5.3 (default, Nov 4 2021, 15:29:10)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/opt/spark/python/pyspark/shell.py", line 29, in <module>
from pyspark.context import SparkContext
File "/opt/spark/python/pyspark/__init__.py", line 53, in <module>
from pyspark.rdd import RDD, RDDBarrier
File "/opt/spark/python/pyspark/rdd.py", line 48, in <module>
from pyspark.traceback_utils import SCCallSiteSync
File "/opt/spark/python/pyspark/traceback_utils.py", line 23, in <module>
CallSite = namedtuple("CallSite", "function file linenum")
File "/opt/spark/python/pyspark/serializers.py", line 390, in namedtuple
for k, v in _old_namedtuple_kwdefaults.items():
AttributeError: 'NoneType' object has no attribute 'items' {code}
The image I used was:
https://hub.docker.com/r/apachehudi/hudi-hadoop_3.1.0-hive_3.1.2-sparkadhoc_3.2.1/tags
Spark 3.2.1 requires Python 3.6+, Spark 3.3 requires Python 3.7+.
I see that base image for Java 8 uses openjdk:8u212-jdk-slim-stretch. Would it
be possible to upgrade to
[8u342-jdk-slim-bullseye|https://hub.docker.com/layers/library/openjdk/8u342-jdk-slim-bullseye/images/sha256-ecb89bb055c1ee4db9da38713b953f6daafefe575c77c6439eabbb85e3168402?context=explore]?
It will also bump python version and allow me to use pyspark I think.
> Notebook demo with pyspark - improve new users experience
> ---------------------------------------------------------
>
> Key: HUDI-5273
> URL: https://issues.apache.org/jira/browse/HUDI-5273
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: kazdy
> Assignee: kazdy
> Priority: Trivial
>
> Make it easier to get started with Hudi for new users.
> Provide docker image with spark + metastore + notebook with examples to
> showcase hudi capabilities.
> The goal is to provide ready to use quickstart guide in mentioned notebook.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
