Hi Chris, Thanks for the email. You're right. but it's like Sqoop job gets
launched based on dataframe values in spark job. Certainly it can be
isolated and broken.
On Sat, Aug 31, 2019 at 8:07 AM Chris Teoh <chris.t...@gmail.com> wrote:
> I'd say this is an uncommon approach, could you use a workflow/scheduling
> system to call Sqoop outside of Spark? Spark is usually multiprocess
> distributed so putting in this Sqoop job in the Spark code seems to imply
> you want to run Sqoop first, then Spark. If you're really insistent on
> this, call it from the driver using Sqoop Java APIs.
>
> On Fri, 30 Aug 2019 at 06:02, Chetan Khatri <chetan.opensou...@gmail.com>
> wrote:
>
>> Sorry,
>> I call sqoop job from above function. Can you help me to resolve this.
>>
>> Thanks
>>
>> On Fri, Aug 30, 2019 at 1:31 AM Chetan Khatri <
>> chetan.opensou...@gmail.com> wrote:
>>
>>> Hi Users,
>>> I am launching a Sqoop job from Spark job and would like to FAIL Spark
>>> job if Sqoop job fails.
>>>
>>> def executeSqoopOriginal(serverName: String, schemaName: String, username:
>>> String, password: String,
>>> query: String, splitBy: String, fetchSize: Int,
>>> numMappers: Int, targetDir: String, jobName: String, dateColumns: String) =
>>> {
>>>
>>> val connectionString = "jdbc:sqlserver://" + serverName + ";" +
>>> "databaseName=" + schemaName
>>> var parameters = Array("import")
>>> parameters = parameters :+ "-Dmapreduce.job.user.classpath.first=true"
>>> parameters = parameters :+ "--connect"
>>> parameters = parameters :+ connectionString
>>> parameters = parameters :+ "--mapreduce-job-name"
>>> parameters = parameters :+ jobName
>>> parameters = parameters :+ "--username"
>>> parameters = parameters :+ username
>>> parameters = parameters :+ "--password"
>>> parameters = parameters :+ password
>>> parameters = parameters :+ "--hadoop-mapred-home"
>>> parameters = parameters :+ "/usr/hdp/2.6.5.0-292/hadoop-mapreduce/"
>>> parameters = parameters :+ "--hadoop-home"
>>> parameters = parameters :+ "/usr/hdp/2.6.5.0-292/hadoop/"
>>> parameters = parameters :+ "--query"
>>> parameters = parameters :+ query
>>> parameters = parameters :+ "--split-by"
>>> parameters = parameters :+ splitBy
>>> parameters = parameters :+ "--fetch-size"
>>> parameters = parameters :+ fetchSize.toString
>>> parameters = parameters :+ "--num-mappers"
>>> parameters = parameters :+ numMappers.toString
>>> if (dateColumns.length() > 0) {
>>> parameters = parameters :+ "--map-column-java"
>>> parameters = parameters :+ dateColumns
>>> }
>>> parameters = parameters :+ "--target-dir"
>>> parameters = parameters :+ targetDir
>>> parameters = parameters :+ "--delete-target-dir"
>>> parameters = parameters :+ "--as-avrodatafile"
>>>
>>> }
>>>
>>>
>
> --
> Chris
>
