Padma,
I've already modified the configuration as specified, but the error is still
there.
Running hdfs dfs -ls /folder does return list of files
I enabled Verbose error logging, here's the full error message:
org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
ERROR: Failed to create schema tree. [Error Id:
28c0c9a2-460d-460e-b93b-1d34e341cc65 on server:31010]
(java.io.IOException) Failed on local exception: java.io.IOException:
Couldn't set up IO streams; Host Details : local host is:
"server/10.61.60.113"; destination host is: "hdfs-server":8020;
org.apache.hadoop.net.NetUtils.wrapException():776
org.apache.hadoop.ipc.Client.call():1480
org.apache.hadoop.ipc.Client.call():1407
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke():229
com.sun.proxy.$Proxy63.getListing():-1
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.g
etListing():573 sun.reflect.GeneratedMethodAccessor3.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():497
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():187
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():102
com.sun.proxy.$Proxy64.getListing():-1
org.apache.hadoop.hdfs.DFSClient.listPaths():2094
org.apache.hadoop.hdfs.DFSClient.listPaths():2077
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal():791
org.apache.hadoop.hdfs.DistributedFileSystem.access$700():106
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():853
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():849
org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus():860
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():16
0
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSche
ma.():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchema
s():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFacto
ry.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617
java.lang.Thread.run():745 Caused By (java.io.IOException) Couldn't
set up IO streams
org.apache.hadoop.ipc.Client$Connection.setupIOstreams():788
org.apache.hadoop.ipc.Client$Connection.access$2800():370
org.apache.hadoop.ipc.Client.getConnection():1529
org.apache.hadoop.ipc.Client.call():1446
org.apache.hadoop.ipc.Client.call():1407
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke():229
com.sun.proxy.$Proxy63.getListing():-1
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.g
etListing():573 sun.reflect.GeneratedMethodAccessor3.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():497
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():187
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():102
com.sun.proxy.$Proxy64.getListing():-1
org.apache.hadoop.hdfs.DFSClient.listPaths():2094
org.apache.hadoop.hdfs.DFSClient.listPaths():2077
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal():791
org.apache.hadoop.hdfs.DistributedFileSystem.access$700():106
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():853
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():849
org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus():860
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():16
0
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSche
ma.():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchema
s():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFacto
ry.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617
java.lang.Thread.run():745 Caused By (java.lang.NoClassDefFoundError)
org/apache/hadoop/yarn/api/ApplicationClientProtocolPB
org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo.getTokenIn
fo():65 org.apache.hadoop.security.SecurityUtil.getTokenInfo():331
org.apache.hadoop.security.SaslRpcClient.getServerToken():263
org.apache.hadoop.security.SaslRpcClient.createSaslClient():219
org.apache.hadoop.security.SaslRpcClient.selectSaslClient():159
org.apache.hadoop.security.SaslRpcClient.saslConnect():396
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection():555
org.apache.hadoop.ipc.Client$Connection.access$1800():370
org.apache.hadoop.ipc.Client$Connection$2.run():724
org.apache.hadoop.ipc.Client$Connection$2.run():720
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1657
org.apache.hadoop.ipc.Client$Connection.setupIOstreams():720
org.apache.hadoop.ipc.Client$Connection.access$2800():370
org.apache.hadoop.ipc.Client.getConnection():1529
org.apache.hadoop.ipc.Client.call():1446
org.apache.hadoop.ipc.Client.call():1407
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke():229
com.sun.proxy.$Proxy63.getListing():-1
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.g
etListing():573 sun.reflect.GeneratedMethodAccessor3.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():497
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():187
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():102
com.sun.proxy.$Proxy64.getListing():-1
org.apache.hadoop.hdfs.DFSClient.listPaths():2094
org.apache.hadoop.hdfs.DFSClient.listPaths():2077
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal():791
org.apache.hadoop.hdfs.DistributedFileSystem.access$700():106
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():853
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():849
org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus():860
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():16
0
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSche
ma.():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchema
s():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFacto
ry.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617
java.lang.Thread.run():745
-----Original Message-----
From: Padma Penumarthy [mailto:[email protected]]
Sent: August 23, 17 9:09 PM
To: [email protected]
Subject: Re: Apache Drill unable to read files from HDFS (Resource
error: Failed to create schema tree)
For HDFS, your storage plugin configuration should be something like this:
{
"type": "file",
"enabled": true,
"connection": "hdfs://<IP Address>:<Port>”, // IP address and port number
of name node metadata service
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": true,
"defaultInputFormat": null
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null
}
},
Also, try hadoop dfs -ls command to see if you can list the files.
Thanks,
Padma
On Aug 23, 2017, at 12:18 PM, Lee, David
<[email protected]<mailto:[email protected]>> wrote:
HDFS storage plugin should be set to your HDFS name node url..
-----Original Message-----
From: Zubair, Muhammad [mailto:[email protected]]
Sent: Wednesday, August 23, 2017 11:33 AM
To: [email protected]<mailto:[email protected]>
Subject: Apache Drill unable to read files from HDFS (Resource error:
Failed to create schema tree)
Hello,
After setting up drill on one of the edge nodes of our HDFS cluster, I am
unable to read any hdfs files. I can query data from local files (as long as
they are in a folder that has 777 permissions) but querying data from hdfs
fails with the following error:
Error: RESOURCE ERROR: Failed to create schema tree.
[Error Id: d9f7908c-6c3b-49c0-a11e-71c004d27f46 on server-name:31010]
(state=,code=0)
Query:
0: jdbc:drill:zk=local> select * from hdfs.`/names/city.parquet` limit 2;
Querying from local file works fine:
0: jdbc:drill:zk=local> select * from dfs.`/tmp/city.parquet` limit 2; My HDFS
settings are similar to the DFS settings, except for the connection URL being the
server address instead of file:/// I can't find anything online regarding this
error for drill.
______________________________________________________________________
_ If you received this email in error, please advise the sender (by
return email or otherwise) immediately. You have consented to receive the
attached electronically at the above-noted email address; please retain a copy
of this confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté
de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse
courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation
pour les fins de reference future.
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/en-us/compliance/email-disclaimers for
further information. Please refer to
http://www.blackrock.com/corporate/en-us/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/en-us/about-us/contacts-locations.
© 2017 BlackRock, Inc. All rights reserved.
______________________________________________________________________
_ If you received this email in error, please advise the sender (by
return email or otherwise) immediately. You have consented to receive the
attached electronically at the above-noted email address; please retain a copy
of this confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté
de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse
courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation
pour les fins de reference future.
