Proposal:
DIL (LinkedIn internal project name) is a generic multi-stage Gobblin connector
library. The code can be accessed here:
https://github.com/linkedin/gobblin-connectors. Its core features and high
level descriptions are shared here:
https://engineering.linkedin.com/blog/2021/data-integration-library.
Per initial discussion with members of Gobblin community, we are here proposing
a separate sub-repo for this library.
Why:
Some thoughts/justifications of a sub-repo vs. a module in the main
Gobblin repo.
1. Gobblin connectors are important part of Gobblin ecosystem, but the
development of connectors is relatively independent of Gobblin core.
2. Gobblin connector is where open source communities can contribute the
most, and it will be growing much faster than Gobblin core.
3. The new connector library is a comprehensive package of unique design
patterns. This is where the data integration diversity challenge will be
addressed. The importance of this code base grows by day as more integration
scenarios are becoming supported.
4. The new connector library evolves and replaces many prior Gobblin
connectors under the “gobblin-modules” module. A separate repo will help avoid
confusion.
5. Separating core and ecosystem modules can help improve isolation and
reduce the number of defects.
Regards,
Chris
