Hi JC,
Unless something has changed recently, it turns out that system/session options
are global: they must be defined in the one big file you discovered, and
default values must be listed in the master drill-module.conf file.
It would be a handy feature to modify this to allow modules to add options.
Easy to define the defaults using the drill-module.conf file for your module.
But, we'd need some way for Drill to invoke code in your module that would
populate the master options list.
Jyothsna, you did a lot of work on options. Any ideas for JC on how to solve
the problems he outlined?
Thanks,
- Paul
On Sunday, October 7, 2018, 6:18:09 AM PDT, Jean-Claude Cote
<[email protected]> wrote:
Hi
I'm writing a new msgpack data source for Drill. I would like to be able to
configure the reader using the alter session mechanism using sqlline,
something like "alter session set `store.msgpack.reader.learnschema` =
true".
However I'm unable to make this work. In my format plugin I have added this
code to my MsgpackRecordReader. Note I'm taking inspiration from the
JSONRecordReader.
public static final String MSGPACK_LEARN_SCHEMA =
"store.msgpack.reader.learnschema";
public static final BooleanValidator MSGPACK_LEARN_SCHEMA_VALIDATOR = new
BooleanValidator(MSGPACK_LEARN_SCHEMA, null);
public MsgpackRecordReader(MsgpackFormatConfig config, final
FragmentContext fragmentContext, final String inputPath,
final DrillFileSystem fileSystem, final List<SchemaPath> columns) {
Preconditions.checkArgument((inputPath != null), "InputPath must be
set.");
this.hadoopPath = new Path(inputPath);
this.fileSystem = fileSystem;
this.fragmentContext = fragmentContext;
this.learnSchema =
fragmentContext.getOptions().getOption(MSGPACK_LEARN_SCHEMA_VALIDATOR);
setColumns(columns);
}
However this is not working. I noticed that the JSONRecordReader registers
it's properties in the SystemOptionManager like so. Is there a way for a
format plugin in the "contrib" folder to be able to do the same?
public class SystemOptionManager extends BaseOptionManager implements
AutoCloseable {
/**
* Creates all the OptionDefinitions to be registered with the {@link
SystemOptionManager}.
* @return A map
*/
public static CaseInsensitiveMap<OptionDefinition>
createDefaultOptionDefinitions() {
new
OptionDefinition(ExecConstants.JSON_SKIP_MALFORMED_RECORDS_VALIDATOR),
Thank you
jc
