[
https://issues.apache.org/jira/browse/SQOOP-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026553#comment-14026553
]
Souvik Bose edited comment on SQOOP-1334 at 6/10/14 3:17 PM:
-------------------------------------------------------------
As you can see above the input format is
org.apache.hadoop.mapred.TextInputFormat
and output format is
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat. I was digging
through the logs in more details, and one thing I noticed which is unnatural if
we compare that with other logs.
14/06/05 08:45:24 INFO mapred.FileInputFormat: Total input paths to process :
35244
14/06/05 08:45:43 DEBUG hcat.SqoopHCatExportFormat: Expected split count 4
14/06/05 08:45:43 DEBUG hcat.SqoopHCatExportFormat: HCatInputFormat provided
split count 35244
In all of the successful cases I see that all the counts match upto 4. But
those are pretty obvious as the hcat imported tables using sqoops have 4 hdfs
files generates as part0000-m like that. But this is a case of external table
which is referencing to a hdfs location with lots of files. So in this case
Expected split cpount should be 35244 instead of only 4. Why it is assuming 4
only that may be issue.
was (Author: souvik2008):
As you can see above the input format is
org.apache.hadoop.mapred.TextInputFormat
and output format is
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat. I was digging
through the logs in more details, and one thing I noticed which is unnatural if
we compare that with other logs.
14/06/05 08:45:24 INFO mapred.FileInputFormat: Total input paths to process :
35244
14/06/05 08:45:43 DEBUG hcat.SqoopHCatExportFormat: Expected split count 4
14/06/05 08:45:43 DEBUG hcat.SqoopHCatExportFormat: HCatInputFormat provided
split count 35244
In all of the successful cases I see that all the counts match upto 4. But
those are pretty obvious as the hcat imported tables using sqoops have 4 hdfs
files generates as part0000-m like that. But this is a case of external table
which is referencing to a hdfs location with lots of files.
> Sqoop Hcatalog export failed with exception of Failed to load foster storage
> handler during map reduce
> ------------------------------------------------------------------------------------------------------
>
> Key: SQOOP-1334
> URL: https://issues.apache.org/jira/browse/SQOOP-1334
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.3
> Environment: CDH4.4.0 with cloudera manager.
> Reporter: Souvik Bose
> Priority: Blocker
> Labels: hcatalog, sqoop, storage-handler
> Attachments: Sqoop_Hcatalog_Export_Log_verbose.txt
>
>
> I tried to export a hive external table to mysql using sqoop hcatalog export
> command.
> sqoop export --connect jdbc:mysql://example.com:3306/demo1 --username
> demo-password demo1234 bd_sqoop1 --hcatalog-table test_data
> I have set the HCAT_HOME, HADOOP_CLASSPATH, HIVE_HOME and paths of Jars. Now
> the job is progressing upto the map reduce stage, but stops at map 0% reduce
> 0% with an exception as below:
> 14/06/04 13:27:21 INFO mapred.JobClient: Task Id :
> attempt_201406021302_0043_m_000002_1, Status : FAILED
> java.io.IOException: Error initializing HCat record reader
> at
> org.apache.sqoop.mapreduce.hcat.SqoopHCatRecordReader.initNextRecordReader(SqoopHCatRecordReader.java:146)
> at
> org.apache.sqoop.mapreduce.hcat.SqoopHCatRecordReader.<init>(SqoopHCatRecordReader.java:57)
> at
> org.apache.sqoop.mapreduce.hcat.SqoopHCatExportFormat.createRecordReader(SqoopHCatExportFormat.java:128)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:644)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.io.IOException: Failed to load foster storage handler
> at
> org.apache.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:416)
> at org.apache.hcatalog.common.HCatUtil.getStorageHandl
--
This message was sent by Atlassian JIRA
(v6.2#6252)
