[ 
https://issues.apache.org/jira/browse/CARBONDATA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237543#comment-17237543
 ] 

Karan commented on CARBONDATA-4021:
-----------------------------------

Caching parquet or ORC segments in Index Server is not supported. Please do not 
enable index server while querying on carbonn data table having parquet or ORC 
segments. Even if index Server is ON, please make sure that fallback is not 
disabled.

> With Index server running, Upon executing count* we are getting the below 
> error, after adding the parquet and ORC segment. 
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-4021
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-4021
>             Project: CarbonData
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Prasanna Ravichandran
>            Priority: Major
>
> We are getting below issues while index server enable and index server 
> fallback disable is configured as true. With count* we are getting the below 
> error, after adding the parquet and ORC segment.
> Queries and error:
> > use rps;
> +---------+|
> Result  |
> +---------+
> +---------+
> No rows selected (0.054 seconds)
> > drop table if exists uniqdata;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.229 seconds)
> > CREATE TABLE uniqdata (cust_id int,cust_name String,active_emui_version 
> > string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 
> > bigint,decimal_column1 decimal(30,10), decimal_column2 
> > decimal(36,36),double_column1 double, double_column2 double,integer_column1 
> > int) stored as carbondata;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.756 seconds)
> > load data inpath 'hdfs://hacluster/user/prasanna/2000_UniqData.csv' into 
> > table uniqdata 
> > options('fileheader'='cust_id,cust_name,active_emui_version,dob,doj,bigint_column1,bigint_column2,decimal_column1,decimal_column2,double_column1,double_column2,integer_column1','bad_records_action'='force');
> INFO  : Execution ID: 95
> +---------+|
> Result  |
> +---------+
> +---------+
> No rows selected(2.789 seconds)
>  > use default;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.052 seconds)
>  > drop table if exists uniqdata;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (1.122 seconds)
> > CREATE TABLE uniqdata (cust_id int,cust_name String,active_emui_version 
> > string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 
> > bigint,decimal_column1 decimal(30,10), decimal_column2 
> > decimal(36,36),double_column1 double, double_column2 double,integer_column1 
> > int) stored as carbondata;
> +---------+
> |  Result  |
> +---------+
> +---------+
> No rows selected (0.508 seconds)
> > load data inpath 'hdfs://hacluster/user/prasanna/2000_UniqData.csv' into 
> > table uniqdata 
> > options('fileheader'='cust_id,cust_name,active_emui_version,dob,doj,bigint_column1,bigint_column2,decimal_column1,decimal_column2,double_column1,double_column2,integer_column1','bad_records_action'='force');
> INFO  : Execution ID: 108
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (1.316 seconds)
> > drop table if exists uniqdata_parquet;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.668 seconds)
> > CREATE TABLE uniqdata_parquet (cust_id int,cust_name 
> > String,active_emui_version string, dob timestamp, doj timestamp, 
> > bigint_column1 bigint,bigint_column2 bigint,decimal_column1 decimal(30,10), 
> > decimal_column2 decimal(36,36),double_column1 double, double_column2 
> > double,integer_column1 int) stored as parquet;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.397 seconds)
> > insert into uniqdata_parquet select * from uniqdata;
> INFO  : Execution ID: 116
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (4.805 seconds)
> >  drop table if exists uniqdata_orc;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.553 seconds)
> > CREATE TABLE uniqdata_orc (cust_id int,cust_name String,active_emui_version 
> > string, dob timestamp, doj timestamp, bigint_column1 bigint,bigint_column2 
> > bigint,decimal_column1 decimal(30,10), decimal_column2 
> > decimal(36,36),double_column1 double, double_column2 double,integer_column1 
> > int) using orc;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.396 seconds)
> > insert into uniqdata_orc select * from uniqdata;
> INFO  : Execution ID: 122
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (3.403 seconds)
> > use rps;
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.06 seconds)
> > Alter table uniqdata add segment options 
> > ('path'='hdfs://hacluster/user/hive/warehouse/uniqdata_parquet','format'='parquet');
> INFO  : Execution ID: 126
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (1.511 seconds)
> > Alter table uniqdata add segment options 
> > ('path'='hdfs://hacluster/user/hive/warehouse/uniqdata_orc','format'='orc');
> +---------+
> |Result|
> +---------+
> +---------+
> No rows selected (0.716 seconds)
> > select count(*) from uniqdata;
> Error: java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): 
> java.security.PrivilegedActionException: org.apache.spark.SparkException: Job 
> aborted due to stage failure: Task 2 in stage 54.0 failed 4 times, most 
> recent failure: Lost task 2.3 in stage 54.0 (TID 118, linux-16, executor 2): 
> java.io.IOException: Problem in loading segment blocks: java.io.IOException: 
> org.apache.thrift.protocol.TProtocolException: Required field 'version' was 
> not found in serialized data! Struct: 
> org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 
> at 
> org.apache.carbondata.core.indexstore.BlockletIndexStore.getAll(BlockletIndexStore.java:201)
>  at 
> org.apache.carbondata.core.indexstore.blockletindex.BlockletIndexFactory.getIndexes(BlockletIndexFactory.java:181)
>  at 
> org.apache.carbondata.core.indexstore.blockletindex.BlockletIndexFactory.getIndexes(BlockletIndexFactory.java:141)
>  at org.apache.carbondata.core.index.TableIndex.prune(TableIndex.java:131) at 
> org.apache.carbondata.core.index.IndexInputFormat$1.initialize(IndexInputFormat.java:174)
>  at 
> org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1$$anonfun$apply$1.apply(DistributedPruneRDD.scala:141)
>  at 
> org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1$$anonfun$apply$1.apply(DistributedPruneRDD.scala:138)
>  at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>  at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>  at scala.collection.immutable.List.foreach(List.scala:392) at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at 
> scala.collection.immutable.List.flatMap(List.scala:355) at 
> org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1.apply(DistributedPruneRDD.scala:138)
>  at 
> org.apache.carbondata.indexserver.DistributedPruneRDD$$anonfun$org$apache$carbondata$indexserver$DistributedPruneRDD$$generateFuture$1.apply(DistributedPruneRDD.scala:138)
>  at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>  at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) 
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)Caused by: 
> java.lang.RuntimeException: java.io.IOException: 
> org.apache.thrift.protocol.TProtocolException: Required field 'version' was 
> not found in serialized data! Struct: 
> org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 
> at 
> org.apache.carbondata.core.indexstore.BlockletIndexStore.get(BlockletIndexStore.java:160)
>  at 
> org.apache.carbondata.core.indexstore.BlockletIndexStore.getAll(BlockletIndexStore.java:190)
>  ... 18 moreCaused by: java.io.IOException: 
> org.apache.thrift.protocol.TProtocolException: Required field 'version' was 
> not found in serialized data! Struct: 
> org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 
> at org.apache.carbondata.core.reader.ThriftReader.read(ThriftReader.java:146) 
> at 
> org.apache.carbondata.core.reader.CarbonIndexFileReader.readIndexHeader(CarbonIndexFileReader.java:54)
>  at 
> org.apache.carbondata.core.util.AbstractDataFileFooterConverter.getIndexInfo(AbstractDataFileFooterConverter.java:151)
>  at 
> org.apache.carbondata.core.util.BlockletIndexUtil.getBlockMetaInfoMap(BlockletIndexUtil.java:113)
>  at 
> org.apache.carbondata.core.indexstore.BlockletIndexStore.get(BlockletIndexStore.java:112)
>  ... 19 moreCaused by: org.apache.thrift.protocol.TProtocolException: 
> Required field 'version' was not found in serialized data! Struct: 
> org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme@30eec3e5 
> at 
> org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme.read(IndexHeader.java:829)
>  at 
> org.apache.carbondata.format.IndexHeader$IndexHeaderStandardScheme.read(IndexHeader.java:748)
>  at org.apache.carbondata.format.IndexHeader.read(IndexHeader.java:656) at 
> org.apache.carbondata.core.reader.ThriftReader.read(ThriftReader.java:144) 
> ... 23 more
> Driver stacktrace: at java.security.AccessController.doPrivileged(Native 
> Method) at javax.security.auth.Subject.doAs(Subject.java:360) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1717)
>  at 
> org.apache.carbondata.indexserver.IndexServer$.doAs(IndexServer.scala:116) at 
> org.apache.carbondata.indexserver.IndexServer$.getSplits(IndexServer.scala:171)
>  at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:550)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at 
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985) at 
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> javax.security.auth.Subject.doAs(Subject.java:422) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2876) (state=,code=0)
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to