RE: DataSource API v2 & Spark-SQL

Thanks for following up, I will give this a go! ~ Shawn -Original Message- From: roizaig Sent: Thursday, April 29, 2021 7:42 AM To: user@spark.apache.org Subject: Re: DataSource API v2 & Spark-SQL You can create a custom data source following this blog <http://roizaig.blogs

Re: DataSource API v2 & Spark-SQL

You can create a custom data source following this blog . It shows how to read a java log file using spark v3 api as an example. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

RE: DataSource API v2 & Spark-SQL

Thanks for clarifying, Russel. Is spark native catalog reference on the roadmap for dsv2 or should I be trying to use something else? ~ Shawn From: Russell Spitzer [mailto:russell.spit...@gmail.com] Sent: Monday, August 3, 2020 8:27 AM To: Lavelle, Shawn Cc: user Subject: Re: DataSource API

Re: DataSource API v2 & Spark-SQL

That's a bad error message. Basically you can't make a spark native catalog reference for a dsv2 source. You have to use that Datasources catalog or use the programmatic API. Both dsv1 and dsv2 programattic apis work (plus or minus some options) On Mon, Aug 3, 2020, 7:28 AM Lavelle, Shawn wrote: