Copying info over from Slack. For anyone who finds this thread, empty drill-module.conf files do not cause problems. The issue was a misunderstanding about the function of drill-override.conf. Values cannot be added to existing property lists using this file, it is designed for overriding the default values.
If you put your UDFs in a package that is already being scanned as part of the default list then there is no need to add a new package to the list. The default list is generated by merging the drill-module.conf files in all of the default Drill packages. Here is one example from the drill exec module [1] To augment the list, add a drill-module.conf file to the classpath, the easiest way to do this is just include it in your jar. [1] - https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf <https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf> - Jason On Mon, Jan 11, 2016 at 11:24 AM, rahul challapalli < [email protected]> wrote: > Sure! > > On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse < > [email protected]> > wrote: > > > The error you posted originally is from deserializing a storage plugin > > config, not the drill-override or drill-module files. That being said, we > > find the list of available storage plugins using classpath scanning. > > > > I don't know why the inclusion or exclusion of contents inside of a > > module.conf file would impact our scanning for storage plugins, but I > > believe this is the issue you are seeing. Somehow the package that > contains > > the hbase plugin is being removed from the list of packages to scan when > > you add the empty file. > > > > Do you want to jump on slack to chat about this? > > > > - Jason > > > > On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli < > > [email protected]> wrote: > > > > > Julien, > > > > > > I have an empty drill-module.conf file in the udf jar file. Below are > the > > > contents of my global drill-override.conf file > > > > > > drill.classpath.scanning.packages : > > ${?drill.classpath.scanning.packages} [ > > > org.apache.drill.udfs ] > > > drill.exec: { > > > cluster-id: "rahul_cluster_com-drillbits", > > > zk.connect: "localhost:5181" > > > } > > > > > > With this, the drillbit fails to start with the error I posted in the > > first > > > email. May be I am getting the syntax wrong? > > > > > > The reason I am insisting an empty drill-module.conf file is because, > > udf's > > > developed prior to the "classpath scanning change" had an empty > > > drill-module.conf file and we used to override the udf package name in > > the > > > global conf file. > > > > > > - Rahul > > > > > > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <[email protected]> > > wrote: > > > > > > > Yes I believe that should work: > > > > - add an empty drill-module.conf in the root of the udf jar > > > > - add the package to drill.classpath.scanning.packages in the drill > > conf > > > > (possibly using drill-override.conf) > > > > > > > > However if you are adding the drill-module.conf file to the jar, you > > > might > > > > as well add the package in it. (unless there's some other reason) > > > > > > > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli < > > > > [email protected]> wrote: > > > > > > > > > Just to be sure, If I have an empty drill-module.conf in the root > of > > my > > > > udf > > > > > jar, then there is no way to add the package information to the > > global > > > > > drill-override.conf file? > > > > > > > > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <[email protected] > > > > > > wrote: > > > > > > > > > > > You are correct: > > > > > > The jar containing the UDFs should have a drill-module.conf at > the > > > root > > > > > > adding your package to the property > > drill.classpath.scanning.packages > > > > for > > > > > > scanning > > > > > > drill.classpath.scanning.packages : > > > > > ${?drill.classpath.scanning.packages} [ > > > > > > my.package.containing.my.udfs > > > > > > ] > > > > > > Jars that don't contain a drill-module.conf will not get scanned. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Thanks for your reply Jason. > > > > > > > > > > > > > > If we cannot override the global configuration file, then for > > > > existing > > > > > > > UDF's we have to re-compile them by modifying the > > drill-module.conf > > > > > file. > > > > > > > If so our UDF's are not backward compatible. Appreciate it if > > > someone > > > > > can > > > > > > > confirm this. > > > > > > > > > > > > > > - Rahul > > > > > > > > > > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse < > > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Rahul, > > > > > > > > > > > > > > > > The error message you are seeing is in reading a storage > plugin > > > > > > > > configuration file. I am planning to fix these kinds of > > messages > > > to > > > > > > > > actually direct users at the file that is failing parsing. I > > have > > > > > seen > > > > > > > this > > > > > > > > in the past when the classpath was incorrect and one of the > > > plugins > > > > > > (like > > > > > > > > Hbase) was not included. > > > > > > > > > > > > > > > > Julien can confirm, but I think this might be intentional to > > have > > > > the > > > > > > > paths > > > > > > > > read out of the modules configuration rather than the global > > one > > > to > > > > > > save > > > > > > > > time when scanning the path (rather than scanning all of the > > jars > > > > for > > > > > > all > > > > > > > > paths given in the override file). > > > > > > > > > > > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Before 1.2, my udfs project contained an empty > > > > drill-override.conf > > > > > > file > > > > > > > > and > > > > > > > > > I used to update the drill-override.conf on all the > drillbits > > > to > > > > > > > specify > > > > > > > > > the package of my UDF. This is no longer working for me. I > > > tried > > > > a > > > > > > few > > > > > > > > > things and below is how my drill-override.conf file looks > now > > > > > > > > > > > > > > > > > > drill.classpath.scanning.packages : > > > > > > > > ${?drill.classpath.scanning.packages} [ > > > > > > > > > org.apache.drill.udfs ] > > > > > > > > > drill.exec: { > > > > > > > > > cluster-id: "rahul_cluster_com-drillbits", > > > > > > > > > zk.connect: "localhost:5181" > > > > > > > > > } > > > > > > > > > > > > > > > > > > When I restart the drillbits, I get this strange error " > > Caused > > > > by: > > > > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could > > not > > > > > > resolve > > > > > > > > type > > > > > > > > > id 'hbase' into a subtype of [simple type, class > > > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]" > > > > > > > > > > > > > > > > > > If I moved the package information to the drill-module.conf > > in > > > my > > > > > > udf's > > > > > > > > > project, then things are working fine. However this > requires > > > > > > > re-compiling > > > > > > > > > the udfs which is not desirable. Is there any other way > > around > > > > > this ? > > > > > > > > > > > > > > > > > > - Rahul > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Julien > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Julien > > > > > > > > > >
