Hi David,
We do something very similar for testing in the ClusterFixture class. You may
be able to use those classes directly, or mine them for ideas about how to set
up the local mode that has a "local cluster coordinator" (i.e. no ZK) and a
local persistence mechanism (write files to /tmp by default.)
Thanks,
- Paul
On Friday, October 19, 2018, 9:20:57 AM PDT, David Cuesta
<[email protected]> wrote:
Hi,
My name is David Cuesta and I am very interested on using Drill as a Library,
in embedded mode. The intention is to launch a drillbit from my application
without using Zookeeper and use it to change the Datasource configurations in
runtime, do queries to my local file system, to an external Hadoop, etc.
How should I do that?
Right now, what I have done is the next (Kotlin code):
import java.sql.DriverManager
fun main(args: Array<String>) {
Class.forName("org.apache.drill.jdbc.Driver")
val connection = DriverManager.getConnection("jdbc:drill:zk=local")
val st = connection.createStatement()
val rs = st.executeQuery("SELECT FROM
dfs.`/Users/david.cuesta/Downloads/userdata1.parquet` WHERE
first_name='Patricia'")
while (rs.next()) {
println(rs.getString(1))
}
}
But there is an error when the library is going to do the bit.run in
DrillConnectionImpl.java: 134:
try {
try {
String connect = null;
DrillConfig dConfig;
if (this.config.isLocal()) {
try {
Class.forName("org.eclipse.jetty.server.Handler");
} catch (ClassNotFoundException var11) {
throw new SQLNonTransientConnectionException("Running Drill
in embedded mode using Drill's jdbc-all JDBC driver Jar file alone is not
supported.", var11);
}
dConfig = DrillConfig.create(info);
this.allocator = RootAllocatorFactory.newRoot(dConfig);
RemoteServiceSet set =
(RemoteServiceSet)GlobalServiceSetReference.SETS.get();
if (set == null) {
this.serviceSet = RemoteServiceSet.getLocalServiceSet();
set = this.serviceSet;
try {
this.bit = new Drillbit(dConfig, this.serviceSet);
this.bit.run();
} catch (UserException var9) {
throw new SQLException("Failure in starting embedded
Drillbit: " + var9.getMessage(), var9);
} catch (Exception var10) {
throw new SQLException("Failure in starting embedded
Drillbit: " + var10, var10);
}
} else {
this.serviceSet = null;
this.bit = null;
}
makeTmpSchemaLocationsUnique(this.bit.getContext().getStorage(), info);
this.client = new DrillClient(dConfig, set.getCoordinator());
} else if (this.config.isDirect()) {
dConfig = DrillConfig.forClient();
this.allocator = RootAllocatorFactory.newRoot(dConfig);
this.client = new DrillClient(dConfig, true);
connect = this.config.getZookeeperConnectionString();
} else {
dConfig = DrillConfig.forClient();
this.allocator = RootAllocatorFactory.newRoot(dConfig);
this.client = new DrillClient();
connect = this.config.getZookeeperConnectionString();
}
this.client.setClientName("Apache Drill JDBC Driver");
this.client.connect(connect, info);
} catch (OutOfMemoryException var12) {
throw new SQLNonTransientConnectionException("Failure creating root
allocator", var12);
} catch (InvalidConnectionInfoException var13) {
throw new SQLNonTransientConnectionException("Invalid parameter in
connection string: " + var13.getMessage(), var13);
} catch (RpcException var14) {
throw new SQLNonTransientConnectionException("Failure in connecting
to Drill: " + var14, var14);
} catch (SQLException var15) {
throw var15;
} catch (Exception var16) {
throw new SQLException("Failure in creating DrillConnectionImpl: "
+ var16, var16);
}
} catch (Throwable var17) {
this.cleanup();
throw var17;
}
}
The tracktrace error is the next one:
-----------------------------------------------------------------------------------------------------
log4j:WARN No appenders could be found for logger
(oadd.org.apache.drill.common.scanner.ClassPathScanner).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Exception in thread "main" java.lang.NoClassDefFoundError:
oadd/org/apache/drill/exec/store/sys/PersistentStoreProvider
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:134)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at AppKt.main(app.kt:18)
Caused by: java.lang.ClassNotFoundException:
oadd.org.apache.drill.exec.store.sys.PersistentStoreProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
So after a intense research, I have seen that the class
“oadd/org/apache/drill/exec/store/sys/PersistentStoreProvider” is not exposed
by Drill, so when the classLoad is done, it is not able to load this class.
Apart of that, I am not sure If will be able to change the Drill dataSources
configuration at runtime.
The dependencies that I have used are the next ones:
compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8"
compile group: 'org.apache.drill.exec', name: 'drill-jdbc-all', version:
'1.13.0'
compile group: 'org.eclipse.jetty', name: 'jetty-server', version:
'9.4.12.v20180830'
compile group: 'org.apache.drill.exec', name: 'drill-java-exec', version:
'1.13.0'
compile group: 'org.apache.calcite', name: 'calcite-core', version: '1.17.0'
Could you help me?
Please, it is very important as we would like to use Drill in our OpenSource
project.
Best regards,
David.
