kishoreg opened a new pull request #4899: (WIP) Initial code dump for dynamically loading pinot plugins URL: https://github.com/apache/incubator-pinot/pull/4899 Submitting the PR for early review/feedback. Here is how we want the end state of Pinot distribution - Pinot-distribution - bin/<all_launchers> - repo/<all_jars_needed_for_pinot> - plugins/ - record_readers - pinot-avro/pinot-avro-shaded.jar - pinot-parquet/pinot-avro-parquet.jar - ... - connectors - pinot-kafka-0.9/pinot-kafka-0.9-shaded.jar - pinot-kafka-2.0/pinot-kafka-2.0-shaded.jar - ... - file-sytem - pinot-hdfs/pinot-hdfs-shaded.jar - pinot-s3/pinot-s3-shaded.jar - pinot-azure/pinot-azure-shaded.jar - ... All launchers will take two parameters plugin.directory and plugin.includes. These two parameters can be set as a system property and PluginManager will iterate recursively through the plugin directory and loaded them. As of now, Pinot will load all the plugin jars into system/App Classloader. I tried loading into separate classloaders but some libraries don't work (e.g. Parquet) since it depends on Hadoop (Hadoop has Class.forName all over the code).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
