Hi Adam,
Unfortunately, I didn't have the opportunity to test keytab/principal
settings. Maybe it makes sense to move the static part into the
constructor. There you will then also have access to the loaded Zeppelin
configuration (zeppelin-site.xml). Please note, however, that the
constructor can potentially be called several times per JVM process and
that it can potentially contain different configurations each time.
The configuration using system properties and environment variables are
always the same for each JVM process and can therefore be returned
statically.
Best Regards
Philipp Dallig
Am 11.09.24 um 16:04 schrieb Adam Binford:
this sounds like a regression that should be fixed, I’m not sure what’s
the issue exactly being
This was specifically what I hit:
https://github.com/apache/zeppelin/blob/940cf13935ec9ed989c7afeec98447c2e7eb5a7e/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/FileSystemStorage.java#L61
which was updated to use "getStaticString" in
https://github.com/apache/zeppelin/commit/3eae255c109aa224c5b5ae3f2c986510d2a26ee9
I'll clean up and make the PR for Spark 4 stuff soon.
On Wed, Sep 11, 2024 at 9:52 AM Cheng Pan <pan3...@gmail.com> wrote:
Adam, good to know that you are working on Spark 4 support, and Spark 4.0
preview2 is likely coming soon. It would be great if you can submit a draft
PR even Spark 4.0 is not GA.
it looks like there was a change that forced keytab/principal settings to
only be loaded from env vars or system props, and not the
zeppelin-site.xml
this sounds like a regression that should be fixed, I’m not sure what’s
the issue exactly being, if you are willing to debug, this project[1] can
leverage docker compose to simplify setup a kerberized Hadoop environment.
[1] https://github.com/awesome-kyuubi/hadoop-testing
Thanks,
Cheng Pan
On Sep 11, 2024, at 18:30, Adam Binford <adam...@gmail.com> wrote:
+1 for deprecating Java 8 and dropping Spark 3.2.
I also have a working version of "unofficial" Spark 4.0 support I can
submit a PR for. Unofficial because I don't want to mark 4.0.0 as
supported
before it's released, but I got the Spark interpreter tests to pass with
4.0.0-preview1.
One thing I noticed while trying to build a release with Spark 4.0
support
is that the 0.12.0-SNAPSHOT build I made broke in our deployment, because
it looks like there was a change that forced keytab/principal settings to
only be loaded from env vars or system props, and not the
zeppelin-site.xml
config. I ended up just backporting it to a 0.11.2 build and Spark
4.0.0-preview1 seems to be working fine.
Adam
On Wed, Sep 11, 2024 at 2:21 AM Cheng Pan <pan3...@gmail.com> wrote:
Thanks for bringing up this discussion, generally, I agree that we
should
have a short release cycle.
For the version number, I prefer to 0.12 instead of 1.0, because some
interpreters relies on external systems were not fully tested with Java
11,
I would like to give it more time to collect user feedbacks before enter
1.0.
+1 for marking Java 8 deprecated in the next release, meanwhile, if
someone is interested, we should also consider adding support for Java
17,
to prepare support for Spark 4.0.
I would also like to propose dropping support for Spark 3.2, and making
Spark 3.5 as the default version.
Currently, Zeppelin supports Spark 3.2 to 3.5.
Spark 3.2 released at Oct 2021, was EOL at Apr 2023.
Spark 3.3 released at Jun 2022, was EOL at Dec 2023.
Spark 3.4 released at Apr 2023, will be EOL at Oct 2024.
Spark 3.5 released at Aug 2024, it’s a LTS version.
For users who want to use lower Spark versions, Livy and Kyuubi(for now,
only supports Spark SQL via the JDBC interpreter) are alternatives.
Thanks,
Cheng Pan
On Sep 11, 2024, at 10:24, Jongyoul Lee <jongy...@gmail.com> wrote:
Hello,
I would like to discuss a new release with the community.
Here is what we are working on:
- Docker enhancement
- Java 8 deprecation
- New UI
- Raft deprecation
If you have any questions or ideas for the next release please feel
free to leave your comments.
Focusing on contributing and reviewing relevant tasks, I hope we do
the next release by the end of this month.
--
Best regards,
Jongyoul Lee
--
Adam Binford
