Thanks Danny, I agree that adding specific auth logic to core worker logic is not a great idea - It's about adding features at worker level that will allow optional customizations like adding Kerberos.
Original option: - set env and jvm parameters which you mentioned may be hard - add ability to stage files that are not part of java classpath/python env Alternative option: - leverage sdk initializer work ( https://lists.apache.org/thread/8r8mq601ztlwo37qvm2ycx8q7z6cv2op) and JVM Initializer - stage files like above I've added alternative design, assuming related design python based initialization actions will be available. On Mon, Dec 16, 2024 at 4:44 PM Danny McCormick <dannymccorm...@google.com> wrote: > Upleveling my high level feedback from the doc in case others have > thoughts: > > I'm a little skeptical about baking specific auth logic mostly needed for > Kafka into the core worker logic, I wonder if we could make this easier > without going this far - one option would be to provide a templated > dockerfile example users can easily use to build their container. Or we > could try to make something more generic to allow users to override envs > (doing this well seems hard though). We could also build a function users > could call as part of JVMInitializer which takes care of the messy details > there (I am leaning in this direction since it is opt-in, pretty easy to > use, and fits into our existing flows). > > In general, I'm +1 to making this space easier, but I think it is not a > universal enough problem to justify a lot of worker setup logic for all > users, or for maintaining separate worker infrastructure. I am, however, > curious if others have run into issues in this space or if folks have > thoughts on how to solve this more generically. > > Lastly, noting that this is loosely related to > https://lists.apache.org/thread/8r8mq601ztlwo37qvm2ycx8q7z6cv2op; there > may be a trend of needing more control over the environment without wanting > to go through the whole custom container flow. We also may need something > similar for x-lang containers. > > Thanks, > Danny > > On Mon, Dec 16, 2024 at 7:28 AM Radek Stankiewicz via dev < > dev@beam.apache.org> wrote: > >> Hi everyone, >> >> Kerberizing workers is not an easy thing for many users, especially for >> Python users where some of the transforms are implemented via expansion >> service. >> I've drafted some ideas on how we could simplify this process - please >> read this design doc >> <https://docs.google.com/document/d/1T3Py6VZhP-FNQMjiURj38ddZyhWQRa_vDEUEc4f1P5A/edit?tab=t.0> >> . >> >> Looking for your feedback - maybe you have a better way of configuring >> kerberos or do you see some scenario where this design won't work with your >> IOs you have used. >> >> Thank you, >> Radek >> >> >> >> >>
