Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

2018-06-25 Thread Mich Talebzadeh 
Thanks

I assume two things

Create these ORC tables from scratch and populate them. These are older
tables from 2.7

Do I need to upgrade to Hadoop 3.1 as suggested as well? Or I can keep the
current 3.0.3 of Hadoop and just redo these ORC tables again

Sounds like I need to upgrade Hadoop to 3.1?


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 25 Jun 2018 at 18:18, Gopal Vijayaraghavan 
wrote:

> > This is Hadoop 3.0.3
> > java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> (state=08S01,code=1)
> > Something is missing here! Is this specific to ORC tables?
>
> No, it is a Hadoop BUG.
>
> https://issues.apache.org/jira/browse/HADOOP-1468
>
> Fixed in Hadoop-2.8.2+ & most likely this is happening because you have
> built something against Hadoop-2.7.3 (Parquet and ORC default to the 2.7.3,
> so you need to rebuild them over for Hadoop3).
>
> Cheers,
> Gopal
>
>
>

Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

2018-06-25 Thread Gopal Vijayaraghavan 
> This is Hadoop 3.0.3
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I 
> (state=08S01,code=1)
> Something is missing here! Is this specific to ORC tables?

No, it is a Hadoop BUG.

https://issues.apache.org/jira/browse/HADOOP-1468

Fixed in Hadoop-2.8.2+ & most likely this is happening because you have built 
something against Hadoop-2.7.3 (Parquet and ORC default to the 2.7.3, so you 
need to rebuild them over for Hadoop3).

Cheers,
Gopal

Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

2018-06-25 Thread Mich Talebzadeh 
Thanks

This is Hadoop 3.0.3

hadoop version
Hadoop 3.0.3
Source code repository
https://yjzhan...@git-wip-us.apache.org/repos/asf/hadoop.git -r
37fd7d752db73d984dc31e0cdfd590d252f5e075
Compiled by yzhang on 2018-05-31T17:12Z
Compiled with protoc 2.5.0
>From source with checksum 736cdcefa911261ad56d2d120bf1fa
This command was run using
/home/hduser/hadoop-3.0.3/share/hadoop/common/hadoop-common-3.0.3.jar

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 25 Jun 2018 at 18:05, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> Hi
> What hadoop version are you using? I am guessing this is from old hadoop
> version. Can you try with hadoop-3.1?
>
> Thanks
> Prasanth
>
>
>
> On Mon, Jun 25, 2018 at 9:55 AM -0700, "Mich Talebzadeh" <
> mich.talebza...@gmail.com> wrote:
>
> Hive version 3
>>
>> An ORC partitioned table
>>
>> 0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
>> Error: Error while processing statement: FAILED: Execution Error, return
>> code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split
>> generation failed with exception: java.lang.NoSuchMethodError:
>> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
>> (state=08S01,code=1)
>>
>> Something is missing here! Is this specific to ORC tables?
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>

Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

2018-06-25 Thread Prasanth Jayachandran 
Hi
What hadoop version are you using? I am guessing this is from old hadoop 
version. Can you try with hadoop-3.1?

Thanks
Prasanth



On Mon, Jun 25, 2018 at 9:55 AM -0700, "Mich Talebzadeh" 
mailto:mich.talebza...@gmail.com>> wrote:

Hive version 3

An ORC partitioned table

0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
Error: Error while processing statement: FAILED: Execution Error, return code 1 
from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed 
with exception: java.lang.NoSuchMethodError: 
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I 
(state=08S01,code=1)

Something is missing here! Is this specific to ORC tables?


Dr Mich Talebzadeh



LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destruction of data or any other property which may arise from 
relying on this email's technical content is explicitly disclaimed. The author 
will in no case be liable for any monetary damages arising from such loss, 
damage or destruction.

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

2018-06-25 Thread Mich Talebzadeh 
Hive version 3

An ORC partitioned table

0: jdbc:hive2://rhes75:10099/default> select count(1) from sales;
Error: Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split
generation failed with exception: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=1)

Something is missing here! Is this specific to ORC tables?

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Update on ORC transactional table fails with org.apache.hadoop.fs.FileStatus.compareTo.. error

2018-06-25 Thread Mich Talebzadeh 
Hi,

hadoop version
Hadoop 3.0.3

Hive version
Apache Hive (version 3.0.0)

ORC transactional table is created as follows:

create table t (
 owner   varchar(30)
,object_name varchar(30)
,subobject_name  varchar(30)
,object_id   bigint
,data_object_id  bigint
,object_type varchar(19)
,created timestamp
,last_ddl_time   timestamp
,timestamp2   varchar(19)
,status  varchar(7)
,temporary2  varchar(1)
,generated   varchar(1)
,secondary   varchar(1)
,namespace   bigint
,edition_namevarchar(30)
,padding1varchar(4000)
,padding2varchar(3500)
,attribute   varchar(32)
,op_type int
,op_time timestamp
)
CLUSTERED BY (object_id) INTO 256 BUCKETS
STORED AS ORC
TBLPROPERTIES ( "orc.compress"="SNAPPY",
"transactional"="true",
"orc.create.index"="true",
"orc.bloom.filter.columns"="object_id",
"orc.bloom.filter.fpp"="0.05",
"orc.stripe.size"="268435456",
"orc.row.index.stride"="1" )
;

The following a simple update

use asehadoop;
set hive.support.concurrency=true;
set hive.enforce.bucketing=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.compactor.initiator.on=true;
set hive.compactor.worker.threads=20;
UPDATE t set object_name = 'Mich' WHERE object_id = 594688;

And this is the error I get at the end

Error: Error while processing statement: FAILED: Execution Error, return
code -101 from org.apache.hadoop.hive.ql.exec.StatsTask.
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
(state=08S01,code=-101)

Appreciate any info.

Regards,

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.