Hi Chris, Thanks for the email. You're right. but it's like Sqoop job gets launched based on dataframe values in spark job. Certainly it can be isolated and broken.
On Sat, Aug 31, 2019 at 8:07 AM Chris Teoh <[email protected]> wrote: > I'd say this is an uncommon approach, could you use a workflow/scheduling > system to call Sqoop outside of Spark? Spark is usually multiprocess > distributed so putting in this Sqoop job in the Spark code seems to imply > you want to run Sqoop first, then Spark. If you're really insistent on > this, call it from the driver using Sqoop Java APIs. > > On Fri, 30 Aug 2019 at 06:02, Chetan Khatri <[email protected]> > wrote: > >> Sorry, >> I call sqoop job from above function. Can you help me to resolve this. >> >> Thanks >> >> On Fri, Aug 30, 2019 at 1:31 AM Chetan Khatri < >> [email protected]> wrote: >> >>> Hi Users, >>> I am launching a Sqoop job from Spark job and would like to FAIL Spark >>> job if Sqoop job fails. >>> >>> def executeSqoopOriginal(serverName: String, schemaName: String, username: >>> String, password: String, >>> query: String, splitBy: String, fetchSize: Int, >>> numMappers: Int, targetDir: String, jobName: String, dateColumns: String) = >>> { >>> >>> val connectionString = "jdbc:sqlserver://" + serverName + ";" + >>> "databaseName=" + schemaName >>> var parameters = Array("import") >>> parameters = parameters :+ "-Dmapreduce.job.user.classpath.first=true" >>> parameters = parameters :+ "--connect" >>> parameters = parameters :+ connectionString >>> parameters = parameters :+ "--mapreduce-job-name" >>> parameters = parameters :+ jobName >>> parameters = parameters :+ "--username" >>> parameters = parameters :+ username >>> parameters = parameters :+ "--password" >>> parameters = parameters :+ password >>> parameters = parameters :+ "--hadoop-mapred-home" >>> parameters = parameters :+ "/usr/hdp/2.6.5.0-292/hadoop-mapreduce/" >>> parameters = parameters :+ "--hadoop-home" >>> parameters = parameters :+ "/usr/hdp/2.6.5.0-292/hadoop/" >>> parameters = parameters :+ "--query" >>> parameters = parameters :+ query >>> parameters = parameters :+ "--split-by" >>> parameters = parameters :+ splitBy >>> parameters = parameters :+ "--fetch-size" >>> parameters = parameters :+ fetchSize.toString >>> parameters = parameters :+ "--num-mappers" >>> parameters = parameters :+ numMappers.toString >>> if (dateColumns.length() > 0) { >>> parameters = parameters :+ "--map-column-java" >>> parameters = parameters :+ dateColumns >>> } >>> parameters = parameters :+ "--target-dir" >>> parameters = parameters :+ targetDir >>> parameters = parameters :+ "--delete-target-dir" >>> parameters = parameters :+ "--as-avrodatafile" >>> >>> } >>> >>> > > -- > Chris >
