paul-rogers opened a new pull request #2054: DRILL-6168: Revise format plugin table functions URL: https://github.com/apache/drill/pull/2054 # [DRILL-6168](https://issues.apache.org/jira/browse/DRILL-6168): Revise format plugin table functions ## Description [DRILL-7612](https://issues.apache.org/jira/browse/DRILL-7612) notes a conflict with how we handle the properties of format plugins. This PR fixes that conflict. On the one hand, we want all plugins (storage and format) to be immutable as they are used as keys in internal maps. Those internal maps will become invalid if we change the values of any plugin property used to compute the hash value. On the other hand, format plugins participate in table functions. The table function implementation is based on Java introspection: it creates an empty instance, then sets public fields to the values requested in the table function. Clearly, these two use cases conflict: one requires immutability, the other mutability. The solution is to revise table functions to not require mutability. We observe the format plugins are JSON-serializable by definition. Jackson allows serializing an object to and from a generic JSON object. This becomes the foundation for the new implementation. A side benefit is that we can start with the existing plugin config values (DRILL-6168): * If an existing format plugin is available, convert it to a generic JSON object, else create an empty object. * Apply each table function parameter as a key/value pair to this object, overwriting properties obtained above. * Convert the resulting object into a new format plugin config object. With this change, we can now change every format plugin to make fields immutable. This requires adding a `@JsonCreator` constructor and sometimes "getter" functions. Also used this opportunity to clean up `equals()` and `hashCode()` functions, and add `toString()` where missing. Doing this work, discovered that the `bootstrap-storage-plugins.json` file used the wrong (old) field name for the text format `fieldDelimiter` field: `delimiter`. Updated the file with the modern name, and added an alias to ensure the existing plugins will continue to load. ## Documentation None for end users. For developers, we must revise descriptions which say that fields should be mutable and public. ## Testing Reran all unit tests to verify no regressions.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
