Many thanks Russell. That worked val *HiveDF* = Try(spark.read. format("jdbc"). option("url", jdbcUrl). option("dbtable", HiveSchema+"."+HiveTable). option("user", HybridServerUserName). option("password", HybridServerPassword). load()) match { * case Success(df) => df* case Failure(e) => throw new Exception("Error Encountered reading Hive table") }
HiveDF: org.apache.spark.sql.DataFrame = [id: int, clustered: int ... 5 more fields] Appreciated your help Sean and Russell Mich On Fri, 2 Oct 2020 at 01:22, Russell Spitzer <russell.spit...@gmail.com> wrote: > You can't use df as the name of the return from the try and the name of > the match variable in success. You also probably want to match the name of > the variable in the match with the return from the match. > > So > > val df = Try(spark.read. > > format("jdbc"). > > option("url", jdbcUrl). > > option("dbtable", HiveSchema+"."+HiveTable). > > option("user", HybridServerUserName). > > option("password", HybridServerPassword). > > load()) match { > > * case Success(validDf) => validDf* > > case Failure(e) => throw new Exception("Error > Encountered reading Hive table") > > } > > On Thu, Oct 1, 2020 at 5:53 PM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> >> Many thanks SEan. >> >> >> Maybe I misunderstood your point? >> >> >> var DF = Try(spark.read. >> >> format("jdbc"). >> >> option("url", jdbcUrl). >> >> option("dbtable", HiveSchema+"."+HiveTable). >> >> option("user", HybridServerUserName). >> >> option("password", HybridServerPassword). >> >> load()) match { >> >> * case Success(DF) => HiveDF* >> >> case Failure(e) => throw new Exception("Error >> Encountered reading Hive table") >> >> } >> >> Still getting the error >> >> >> <console>:74: error: recursive method DF needs type >> >> case Success(DF) => HiveDF >> >> Do I need to define DF as DataFrame beforehand because at that moment >> Spark does not know what DF type is >> >> Thanks again >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Thu, 1 Oct 2020 at 23:08, Sean Owen <sro...@gmail.com> wrote: >> >>> You are reusing HiveDF for two vars and it ends up ambiguous. Just >>> rename one. >>> >>> On Thu, Oct 1, 2020, 5:02 PM Mich Talebzadeh <mich.talebza...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> >>>> Spark version 2.3.3 on Google Dataproc >>>> >>>> >>>> I am trying to use databricks to other databases >>>> >>>> >>>> https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html >>>> >>>> >>>> to read from Hive table on Prem using Spark in Cloud >>>> >>>> >>>> This works OK without a Try enclosure. >>>> >>>> >>>> import spark.implicits._ >>>> >>>> import scala.util.{Try, Success, Failure} >>>> >>>> val HiveDF = Try(spark.read. >>>> >>>> format("jdbc"). >>>> >>>> option("url", jdbcUrl). >>>> >>>> option("dbtable", HiveSchema+"."+HiveTable). >>>> >>>> option("user", HybridServerUserName). >>>> >>>> option("password", HybridServerPassword). >>>> >>>> load()) match { >>>> >>>> case Success(HiveDF) => HiveDF >>>> >>>> case Failure(e) => throw new Exception("Error >>>> Encountered reading Hive table") >>>> >>>> } >>>> >>>> However, with Try I am getting the following error >>>> >>>> >>>> <console>:66: error: recursive value HiveDF needs type >>>> >>>> case Success(HiveDF) => HiveDF >>>> >>>> Wondering what is causing this. I have used it before (say reading from >>>> an XML file) and it worked the, >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>