[
https://issues.apache.org/jira/browse/CARBONDATA-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633208#comment-16633208
]
jiangmanhua commented on CARBONDATA-2928:
-----------------------------------------
Hi, the error message said it can not found the carbondata file, so it should
have already read the carbon index/merge index file and the pruning results
told us to read the carbondata file.
I am confusing the value `9965100033100001` in
`part-0-9965100033100001_batchno0-0-9865-1536751682754.carbondata`. Please
check the file.
More check I done:
1. let the thread sleep before starting to merge carbon index and query is OK ;
2. While writing merge index, carbon uses `AtoicFileOperations` which first
write to a temp file and then rename it.
3. Under `<TablePath>/Metadata/segment/`, you can see `.segment` file which
records the index file. After the merge index is written, this file will be
updated. So carbon knows to load which index file(s)
> query failed when doing merge index during load
> -----------------------------------------------
>
> Key: CARBONDATA-2928
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2928
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
> Affects Versions: 1.4.1
> Reporter: ocean
> Priority: Major
> Fix For: NONE
>
>
> In carbondata version 1.4.1, carbonindex file be merged in every load. But
> when query through thriftserver(about 10QPS), if merge index is in progress,
> An error will occurs.
>
> java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: File does
> not exist:
> /warehouse/spark/ae_event_cb_40e_std/productid=534/part-0-9965100033100001_batchno0-0-9865-1536751682754.carbondata
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
