One of possible problems is a mismatch of yarn libraries on the edge
node. What hadoop distro and version do you have on the edge node? Can
you provide output (classpath) of "jinfo <pid>" where pid is
Foreman/Drillbit process id.
Caused By (java.lang.NoClassDefFoundError)
org/apache/hadoop/yarn/api/ApplicationClientProtocolPB
org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo.getTokenInfo():65
org.apache.hadoop.security.SecurityUtil.getTokenInfo():331
org.apache.hadoop.security.SaslRpcClient.getServerToken():263
org.apache.hadoop.security.SaslRpcClient.createSaslClient():219
org.apache.hadoop.security.SaslRpcClient.selectSaslClient():159
org.apache.hadoop.security.SaslRpcClient.saslConnect():396
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection():555
org.apache.hadoop.ipc.Client$Connection.access$1800():370
org.apache.hadoop.ipc.Client$Connection$2.run():724
Thank you,
Vlad
On 8/24/17 08:39, Zubair, Muhammad wrote:
Padma,
I've already modified the configuration as specified, but the error is still
there.
Running hdfs dfs -ls /folder does return list of files
I enabled Verbose error logging, here's the full error message:
org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: Failed to create schema
tree. [Error Id: 28c0c9a2-460d-460e-b93b-1d34e341cc65 on server:31010] (java.io.IOException) Failed
on local exception: java.io.IOException: Couldn't set up IO streams; Host Details : local host is:
"server/10.61.60.113"; destination host is: "hdfs-server":8020;
org.apache.hadoop.net.NetUtils.wrapException():776 org.apache.hadoop.ipc.Client.call():1480
org.apache.hadoop.ipc.Client.call():1407
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke():229 com.sun.proxy.$Proxy63.getListing():-1
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing():573
sun.reflect.GeneratedMethodAccessor3.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43 java.lang.reflect.Method.invoke():497
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():187
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():102
com.sun.proxy.$Proxy64.getListing():-1 org.apache.hadoop.hdfs.DFSClient.listPaths():2094
org.apache.hadoop.hdfs.DFSClient.listPaths():2077
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal():791
org.apache.hadoop.hdfs.DistributedFileSystem.access$700():106
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():853
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():849
org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus():860
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 Caused By
(java.io.IOException) Couldn't set up IO streams
org.apache.hadoop.ipc.Client$Connection.setupIOstreams():788
org.apache.hadoop.ipc.Client$Connection.access$2800():370
org.apache.hadoop.ipc.Client.getConnection():1529 org.apache.hadoop.ipc.Client.call():1446
org.apache.hadoop.ipc.Client.call():1407
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke():229 com.sun.proxy.$Proxy63.getListing():-1
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing():573
sun.reflect.GeneratedMethodAccessor3.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43 java.lang.reflect.Method.invoke():497
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():187
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():102
com.sun.proxy.$Proxy64.getListing():-1 org.apache.hadoop.hdfs.DFSClient.listPaths():2094
org.apache.hadoop.hdfs.DFSClient.listPaths():2077
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal():791
org.apache.hadoop.hdfs.DistributedFileSystem.access$700():106
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():853
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():849
org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus():860
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 Caused By
(java.lang.NoClassDefFoundError) org/apache/hadoop/yarn/api/ApplicationClientProtocolPB
org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo.getTokenInfo():65
org.apache.hadoop.security.SecurityUtil.getTokenInfo():331
org.apache.hadoop.security.SaslRpcClient.getServerToken():263
org.apache.hadoop.security.SaslRpcClient.createSaslClient():219
org.apache.hadoop.security.SaslRpcClient.selectSaslClient():159
org.apache.hadoop.security.SaslRpcClient.saslConnect():396
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection():555
org.apache.hadoop.ipc.Client$Connection.access$1800():370
org.apache.hadoop.ipc.Client$Connection$2.run():724
org.apache.hadoop.ipc.Client$Connection$2.run():720
java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1657
org.apache.hadoop.ipc.Client$Connection.setupIOstreams():720
org.apache.hadoop.ipc.Client$Connection.access$2800():370
org.apache.hadoop.ipc.Client.getConnection():1529 org.apache.hadoop.ipc.Client.call():1446
org.apache.hadoop.ipc.Client.call():1407
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke():229 com.sun.proxy.$Proxy63.getListing():-1
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing():573
sun.reflect.GeneratedMethodAccessor3.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43 java.lang.reflect.Method.invoke():497
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod():187
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke():102
com.sun.proxy.$Proxy64.getListing():-1 org.apache.hadoop.hdfs.DFSClient.listPaths():2094
org.apache.hadoop.hdfs.DFSClient.listPaths():2077
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal():791
org.apache.hadoop.hdfs.DistributedFileSystem.access$700():106
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():853
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall():849
org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus():860
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745
-----Original Message-----
From: Padma Penumarthy [mailto:[email protected]]
Sent: August 23, 17 9:09 PM
To: [email protected]
Subject: Re: Apache Drill unable to read files from HDFS (Resource error:
Failed to create schema tree)
For HDFS, your storage plugin configuration should be something like this:
{
"type": "file",
"enabled": true,
"connection": "hdfs://<IP Address>:<Port>”, // IP address and port number
of name node metadata service
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": true,
"defaultInputFormat": null
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null
}
},
Also, try hadoop dfs -ls command to see if you can list the files.
Thanks,
Padma
On Aug 23, 2017, at 12:18 PM, Lee, David
<[email protected]<mailto:[email protected]>> wrote:
HDFS storage plugin should be set to your HDFS name node url..
-----Original Message-----
From: Zubair, Muhammad [mailto:[email protected]]
Sent: Wednesday, August 23, 2017 11:33 AM
To: [email protected]<mailto:[email protected]>
Subject: Apache Drill unable to read files from HDFS (Resource error: Failed to
create schema tree)
Hello,
After setting up drill on one of the edge nodes of our HDFS cluster, I am
unable to read any hdfs files. I can query data from local files (as long as
they are in a folder that has 777 permissions) but querying data from hdfs
fails with the following error:
Error: RESOURCE ERROR: Failed to create schema tree.
[Error Id: d9f7908c-6c3b-49c0-a11e-71c004d27f46 on server-name:31010]
(state=,code=0)
Query:
0: jdbc:drill:zk=local> select * from hdfs.`/names/city.parquet` limit 2;
Querying from local file works fine:
0: jdbc:drill:zk=local> select * from dfs.`/tmp/city.parquet` limit 2; My HDFS
settings are similar to the DFS settings, except for the connection URL being the
server address instead of file:/// I can't find anything online regarding this
error for drill.
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email
or otherwise) immediately. You have consented to receive the attached
electronically at the above-noted email address; please retain a copy of this
confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté
de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse
courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation
pour les fins de reference future.
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/en-us/compliance/email-disclaimers for
further information. Please refer to
http://www.blackrock.com/corporate/en-us/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/en-us/about-us/contacts-locations.
© 2017 BlackRock, Inc. All rights reserved.
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email
or otherwise) immediately. You have consented to receive the attached
electronically at the above-noted email address; please retain a copy of this
confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté
de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse
courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation
pour les fins de reference future.
Thank you,
Vlad
