Okay. That's what I thought I could do however I was hoping not to have to modify the java-exec module. I'm writing my plugin in the contrib folder.
For now I've added these options to my MsgpackFormatConfig which means I have to edit the storage plugin configuration to change it which is not ideal but works. On Mon, Oct 8, 2018 at 5:17 PM Jyothsna Reddy <[email protected]> wrote: > -----Formatting the previous email-------- > Hi JC, > Like Paul suggested the default values must be added to drill-module.conf > file. Please follow these steps to add a new system/session option: > > Step-1 : Add a validator in the ExecConstants.java similar to Kafka reader > *public static final String MSGPACK_LEARN_SCHEMA = > "store.msgpack.reader.learnschema"; * > * BooleanValidator MSGPACK_LEARN_SCHEMA_VALIDATOR = new > BooleanValidator(MSGPACK_LEARN_SCHEMA, null);* > > Step-2: Add the validator from the above step to the *definitions* in > SystemOptionManager.java > > Step-3 : Add the option in the conf file (drill-module.conf or > drill-override.conf file ) under the name space drill.exec.options > > > > *drill.exec.options : { store.msgpack.reader.learnschema : > true }* > > Step-4: Set the default value for the option in the conf file as shown > above > > Step-5: OptionManager loads default values from the conf file, user can > override the default values by issuing alter SYSTEM/SESSION. > > Thank you, > Jyothsna > > > On Mon, Oct 8, 2018 at 1:58 PM Venkata Donapati <[email protected]> > wrote: > > > Hi JC, > > Like Paul suggested the default values must be added to drill-module.conf > > file. Please follow these steps to add a new system/session option: > > > > Step-1 : Add a validator in the ExecConstants.java similar to Kafka > reader > > > > > > > > *public static final String MSGPACK_LEARN_SCHEMA = * > > *"store.msgpack.reader.learnschema"; BooleanValidator > > MSGPACK_LEARN_SCHEMA_VALIDATOR = new > > BooleanValidator(MSGPACK_LEARN_SCHEMA, null);* > > > > Step-2: Add the validator from the above step to the > > *definitions* in SystemOptionManager.java > > > > Step-3 : Add the option in the conf file (drill-module.conf or > > drill-override.conf file ) under the name space drill.exec.options > > > > > > *drill.exec.options : {* > > > > * store.msgpack.reader.learnschema : true* > > > > * }* > > > > Step-4: Set the default value for the option in the conf file as shown > > above > > > > Step-5: OptionManager loads default values from the conf file, > user > > can override the default values by issuing alter SYSTEM/SESSION. > > > > > > > > Thank you, > > > > Jyothsna > > > > > > On Sun, Oct 7, 2018 at 9:43 PM Paul Rogers <[email protected]> > > wrote: > > > > > Hi JC, > > > > > > Unless something has changed recently, it turns out that system/session > > > options are global: they must be defined in the one big file you > > > discovered, and default values must be listed in the master > > > drill-module.conf file. > > > > > > It would be a handy feature to modify this to allow modules to add > > > options. Easy to define the defaults using the drill-module.conf file > for > > > your module. But, we'd need some way for Drill to invoke code in your > > > module that would populate the master options list. > > > > > > Jyothsna, you did a lot of work on options. Any ideas for JC on how to > > > solve the problems he outlined? > > > > > > Thanks, > > > - Paul > > > > > > > > > > > > On Sunday, October 7, 2018, 6:18:09 AM PDT, Jean-Claude Cote < > > > [email protected]> wrote: > > > > > > Hi > > > > > > I'm writing a new msgpack data source for Drill. I would like to be > able > > to > > > configure the reader using the alter session mechanism using sqlline, > > > something like "alter session set `store.msgpack.reader.learnschema` = > > > true". > > > > > > However I'm unable to make this work. In my format plugin I have added > > this > > > code to my MsgpackRecordReader. Note I'm taking inspiration from the > > > JSONRecordReader. > > > > > > public static final String MSGPACK_LEARN_SCHEMA = > > > "store.msgpack.reader.learnschema"; > > > public static final BooleanValidator MSGPACK_LEARN_SCHEMA_VALIDATOR = > > new > > > BooleanValidator(MSGPACK_LEARN_SCHEMA, null); > > > > > > > > > public MsgpackRecordReader(MsgpackFormatConfig config, final > > > FragmentContext fragmentContext, final String inputPath, > > > final DrillFileSystem fileSystem, final List<SchemaPath> > columns) { > > > > > > Preconditions.checkArgument((inputPath != null), "InputPath must be > > > set."); > > > this.hadoopPath = new Path(inputPath); > > > > > > this.fileSystem = fileSystem; > > > this.fragmentContext = fragmentContext; > > > this.learnSchema = > > > fragmentContext.getOptions().getOption(MSGPACK_LEARN_SCHEMA_VALIDATOR); > > > setColumns(columns); > > > } > > > > > > However this is not working. I noticed that the JSONRecordReader > > registers > > > it's properties in the SystemOptionManager like so. Is there a way for > a > > > format plugin in the "contrib" folder to be able to do the same? > > > > > > public class SystemOptionManager extends BaseOptionManager implements > > > AutoCloseable { > > > > > > /** > > > * Creates all the OptionDefinitions to be registered with the {@link > > > SystemOptionManager}. > > > * @return A map > > > */ > > > public static CaseInsensitiveMap<OptionDefinition> > > > createDefaultOptionDefinitions() { > > > new > > > OptionDefinition(ExecConstants.JSON_SKIP_MALFORMED_RECORDS_VALIDATOR), > > > > > > Thank you > > > jc > > > > > >
