Just trying to get up and running with Drill, and most of my data is in GCS.
Brand new Drill user, I'm familiar with Apache Beam and a tiny bit of
Spark, not familiar with Hadoop/HDFS and I don't need Hadoop for my current
project.
- bin/drill-embedded
- configured S3, queried successfully
- built github.com/GoogleCloudPlatform/bigdata-interop
- copied gcs-connector-1.6.1-hadoop2-shaded.jar to apache-drill-1.11.0/jars
- bin/drill-embedded does not throw an error (it was with the non-shaded
jar)
- created a storage configuration in the drill UI
{
"type": "file",
"enabled": true,
"connection": "gs://my-bucket/",
"config": {
"fs.gs.project.id": "my-project"
},
"workspaces": {
(if the rest is important let me know)
- GS and S3 queries fail
Surely this is just a configuration thing, but the GCS log isn't very
helpful. The relevant log is after my signature.
Jacob
===========
Drill UI:
RESOURCE ERROR: Failed to create schema tree.
===========
log/sqlline.log:
[Error Id: 82c064c0-ee57-401f-9276-9c8848dbeadd ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
~[drill-common-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:115)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:99)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:164)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:153)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema(QueryContext.java:139)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.planner.sql.SqlConverter.<init>(SqlConverter.java:111)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:101)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79)
[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1050)
[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:280)
[drill-java-exec-1.11.0.jar:1.11.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_141]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
Caused by: java.io.IOException: Multiple IOExceptions.
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageExceptions.createCompositeException(GoogleCloudStorageExceptions.java:61)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfos(GoogleCloudStorageImpl.java:1496)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.ForwardingGoogleCloudStorage.getItemInfos(ForwardingGoogleCloudStorage.java:221)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfosRaw(GoogleCloudStorageFileSystem.java:1256)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.listFileInfo(GoogleCloudStorageFileSystem.java:1032)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.listStatus(GoogleHadoopFileSystemBase.java:1305)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:522)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:160)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>(FileSystemSchemaFactory.java:77)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:64)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:149)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:396)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:110)
[drill-java-exec-1.11.0.jar:1.11.0]
... 12 common frames omitted
