Disable/Remove datasources in Spark

Hi, I am trying to force all users to use only 1 datasource (A custom datasource I plan to write) to read/write data. So, I was looking at the DataSource api in Spark: 1. I was able to figure out how to create my own Datasource (Reference

Re: structured streaming- checkpoint metadata growing indefinetely

For posterity: the problem was FileStreamSourceLog class. I needed to overwrite method shouldRetain, that by default returns true and its doc say: Default implementation retains all log entries. Implementations should override the method to change the behavior. -- Kind regards/ Pozdrawiam, Wojciec

Re: Spark error with jupyter

Hi, looks like spark listener is not working? Is your session still running? Try to see the SPARK UI to find out whether the session is still active or not Regards, Gourav On Tue, May 3, 2022 at 7:37 PM Bjørn Jørgensen wrote: > I use jupyterlab and spark and I have not seen this before. > > Ju