Hi,
Sqoop is based on Hadoop Tools infrastructure, which allows users to develop
their own plugins for sqoop and extend sqoop as per their requirements without
changing the base sqoop. Users can develop their plugins as separate jars,
deploy them in $SQOOP_LIB and register with sqoop as :
<property>
<name>sqoop.tool.plugins</name>
<value>com.expedia.sqoop.tool.HiveMergePlugin</value>
<description>A comma-delimited list of ToolPlugin implementations
which are consulted, in order, to register SqoopTool instances which
allow third-party tools to be used.
</description>
</property>
This functionality is already there and we are able to use this develop our own
plugins. Infact there are a couple of teams in Expedia which are using their
own plugins with sqoop. I am sure there might be others who will be using this
or willing to use this functionality.. Our Team have also a tool called
hive-merge for sqoop which merges two hive tables, but this tool needs to use
some option names which are not available in SqoopOptions. The limitation with
user defined plugin is that it doesn't support custom defined options as all
the sqoop options for sqoop are tied to names and types in SqoopOptions. We
have made a small enhancement which adds a map member to SqoopOptions. This is
a very small change but it opens up Sqoop for user defined tools with custom
options. It will open up sqoop for user defined plugins. We have also added
support for its persistence into metastore. The patch is already available for
the issue :
https://issues.apache.org/jira/browse/SQOOP-2333
The review request for the fix is available at :
https://reviews.apache.org/r/36042/
Although the change is small, it will be a great addition to Sqoop and will
enable developers to develop their own plugins with user defined options and
contribute back to the community without knowing much details of sqoop. We have
also added documentation about how to develop user defined plugins and custom
options for user defined plugins.
Please review and suggest.
Thanks and Regards,
Rakesh.
