Yeah I'm in favor of fast-fail if things are not working out as end users intended. Spark should only fail back when it doesn't make any difference but only some sort of performance. (like whole stage codegen) This fail back brings behavioral differences, which should be considered as a bug.
I'll file an issue and raise a PR sooner. Thanks for providing voices! On Sat, Oct 24, 2020 at 2:03 AM Ryan Blue <rb...@netflix.com> wrote: > I agree. If the user configures an invalid catalog, it should fail and > propagate the exception. Running with a catalog other than the one the user > requested is incorrect. > > On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer <russell.spit...@gmail.com> > wrote: > >> I was convinced that we should probably just fail, but if that is too >> much of a change, then logging the exception is also acceptable. >> >> On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> >> wrote: >> >>> Hi devs, >>> >>> I got another report regarding configuring v2 session catalog, when >>> Spark fails to instantiate the configured catalog. For now, it just simply >>> logs error message without exception information, and silently uses the >>> default session catalog. >>> >>> >>> https://github.com/apache/spark/blob/3819d39607392aa968595e3d97b84fedf83d08d9/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala#L75-L95 >>> >>> IMO, as the user intentionally provides the session catalog, it >>> shouldn't fail back and just throw the exception. Otherwise (if we still >>> want to do the failback), we need to add the exception information in the >>> error log message at least. >>> >>> Would like to hear the voices. >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >> > > -- > Ryan Blue > Software Engineer > Netflix >
