Hi, One step closer. Drill now seems to be communicating with at least the Tachyon client library, picked up by using "tachyon:///" configuration, and jets3t library is loaded.
The strange thing, at this stage, is that the Tachyon client seems to be using the jets3t librarry and is complaining about missing s3 config. I would think it enough for the Tachyon client to talk to the Tachyon worker that is already handling all S3 communications just fine. Anyways; if anyone here has been down this hole then please share. Regards, -Stefan On Sun, Jul 26, 2015 at 3:22 PM, Stefán Baxter <[email protected]> wrote: > Thank you. > > After a bit more digging I see that Tachyon is not HDFS compatible > (explains the "tachyon://" connection prefix) and I doubt that the it will > work at all until Drill starts officially supporting it. > > I will do this and report any positive progress. > > It's a shame that it does not work as Tachyon+S3 could be a really great > Drill setup where frequently used files are available "locally" and S3 is > used more like a deep storage than anything else. > > Regards, > -Stefán > > > > On Sun, Jul 26, 2015 at 3:11 PM, David Tucker <[email protected]> > wrote: > >> One quick hint : >> The file $DRILL_HOME/bin/hadoop-excludes.txt has a list of jars that ARE >> NOT loaded during the bootstrap of Drill … and jets3t is one of them. >> Commenting out the jets3t line in that file and restarting the drill bits >> will at least get you past the first java dependency problem. >> >> — David >> >> > On Jul 26, 2015, at 4:52 AM, Stefán Baxter <[email protected]> >> wrote: >> > >> > Hi, >> > >> > I'm trying to run Drill with Tachyon on top of S3. >> > >> > My Drill-Source config looks like this: >> > { >> > "type": "file", >> > "enabled": true, >> > "connection": "tachyon://localhost:19998/", >> > "workspaces": { >> > "root": { >> > "location": "/", >> > "writable": true, >> > "defaultInputFormat": null >> > } >> > }, >> > "formats": { >> > "psv": { >> > "type": "text", >> > "extensions": [ >> > "tbl" >> > ], >> > "delimiter": "|" >> > }, >> > "csv": { >> > "type": "text", >> > "extensions": [ >> > "csv" >> > ], >> > "delimiter": "," >> > }, >> > "tsv": { >> > "type": "text", >> > "extensions": [ >> > "tsv" >> > ], >> > "delimiter": "\t" >> > }, >> > "parquet": { >> > "type": "parquet" >> > }, >> > "json": { >> > "type": "json" >> > }, >> > "avro": { >> > "type": "avro" >> > } >> > } >> > } >> > >> > Initilly I get: >> > Error: SYSTEM ERROR: IOException: No FileSystem for scheme: tachyon >> > >> > When I copy the tachyon client (tachyon-client-0.7.0.jar) to the >> > jar/3rdparty the system complains about missing tependencies (jets3). >> > >> > If I copy the full Tachyon client >> > (tachyon-client-0.7.0-jar-with-dependencies.jar) to the jar/3rds party >> the >> > system will not start. >> > >> > Exception in thread "main" java.lang.ExceptionInInitializerError >> > at org.apache.log4j.LogManager.getLogger(LogManager.java:44) >> > at >> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64) >> > at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277) >> > at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288) >> > at org.apache.drill.exec.server.Drillbit.<clinit>(Drillbit.java:58) >> > Caused by: java.lang.IllegalStateException: Detected both >> > log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting >> > StackOverflowError. See also >> > http://www.slf4j.org/codes.html#log4jDelegationLoop for more details. >> > at >> org.apache.log4j.Log4jLoggerFactory.<clinit>(Log4jLoggerFactory.java:51) >> > ... 5 more >> > >> > >> > Has anyone here done this already and is willing to share some info? >> > >> > Regards, >> > -Stefán >> >> >
