Hi All,
SET hive.exec.compress.output=true;
SET io.seqfile.compression.type=BLOCK;
SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;
set mapred.job.priority=VERY_HIGH;
set mapred.job.name=loading data from YYY to XXXXXX;
insert overwrite table XXXXXXXXX partition (pixeldate, pixelhr, pixelmin,
reportts, clustertype, logStatus, reportgroup, requesttype, clicktype,
plugintype) select key, ipaddress, timezone, url, responsecode,
responsesize, referrer, useragent, decodedurl, domain , specificdomain,
sourceurl, destinationurl, destinationdomain, cid, cpc, rpc,
geocountry,geoDefault,userid, sessionid, requestid, pagetypeid, articleid,
keywords, responsetime, widgetid, pagetype, sourceIP, geocity, isnewuser,
errormessage, logValue, contractstatus, pagetypederived, domainid, noofads,
noofrelatedposts, adopt, addomain, prid, topposition, leftposition,
requestTimestamp, occurenceFreq, suspectStatus, isbillable, platform,
to_date(logdate) pixeldate, hour(logdate) as pixelhr, (hour(logdate) * 60)
+ (floor(minute(logdate) / 15) * 15) as pixelmin, reportts, clustertype,
logStatus, reportgroup, requesttype, clicktype, plugintype from YYYYYYY
where reportts=20130619042507 and clustertype = 'DEFAULT' and
reportdate='2013-06-19' and logstatus in('VALID','Invalid','Error')
When I run this query. I am seeing this error in the tasktracker log.
In task tracker System Logs we are seeing the following errors.
In ERRORS :
We see "*Error:
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.getRemaining(J)I*"
*syslog logs*
2013-07-15 05:39:41,239 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-07-15 05:39:41,241 ERROR org.apache.hadoop.io.nativeio.NativeIO:
*Unable to initialize NativeIO libraries*
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
Can someone please throw light in to it.
Some added Info :
I have upgraded Hadoop from .20.2 to 1.1.2
and hive from .7.0 to .11.
After Upgrade I faced lot of Jar file issues and cleared each with lot
of search in the internet.
But this haunting us for few days. Any help would be grateful.
Thanks,
Pradeep
