Re: Parametrisable output metadata path

low hanging fruit improvement that helps with a few issues. I might be biased obviously. :) -- Kind regards/ Pozdrawiam, Wojciech Indyk pon., 17 kwi 2023 o 22:42 Jungtaek Lim napisał(a): > small correction: "I intentionally didn't enumerate." The meaning could be > quite different so

Re: Parametrisable output metadata path

Hi Jungtaek, integration with Delta Lake is not an option to me, I raised a PR for improvement of FileStreamSink with the new parameter: https://github.com/apache/spark/pull/40821. Can you please take a look? -- Kind regards/ Pozdrawiam, Wojciech Indyk niedz., 16 kwi 2023 o 04:45 Jungtaek Lim

Parametrisable output metadata path

and hasMetadata works well. If you have any corner case in mind where the parametrised output metadata path can break something please describe it. -- Kind regards/ Pozdrawiam, Wojciech Indyk

Delegation of Kerberos tokens

to workers? Does it depend on spark mode? As I saw in the code the method obtainTokenForHBase is calling when yarn-client mode is on, but not for yarn-cluster. Am I right? Is it ok? -- Kind regards/ Pozdrawiam, Wojciech Indyk http://datacentric.pl