Hi, community!
Our team is working on the new CdapIO connector implementation and we prepared initial PRs for components of the CdapIO package. We are reviewing the design document [1] with members of CDAP, and working towards resolving comments and having CDAP plugin artifacts published to Maven, so we could include them as dependencies (this delays finalizing some PRs like [2]). We prepared PRs for CDAP context classes and plugin wrapper classes. While they depend on [2], wanted to ask if someone would be open to help with starting on code reviews? PRs ready for code review: [3] CDAP context classes for CDAP plugins https://github.com/apache/beam/pull/17104 [https://opengraph.githubassets.com/9260a94844067566a77c159de11543f854fc4c3b1a735395873f5460c9beaeb2/apache/beam/pull/17104]<https://github.com/apache/beam/pull/17104> [BEAM-14081] [CdapIO] Add context classes for CDAP plugins by Krasavinigor · Pull Request #17104 · apache/beam · GitHub<https://github.com/apache/beam/pull/17104> github.com WARNING! This PR depends on PR #17051 Changes: Implements classes for BatchSourceContext, BatchSinkContext and StreamingSourceContext interfaces. Implements BatchContextImpl as part of has-a references for classes. Implements tests. We move all common logic to BatchContextImpl which implements the CDAP interface BatchContext. Each class extends BatchContextImpl and can override it if needed and contains its own logic. After creating the plugin config and an instance of [PluginName]BatchSink/BatchSource/StreamingSource we pass our context through the prepareRun method for in instance. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: Choose reviewer(s) and mention them in a comment (R: @username). Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. Update [4] CDAP plugin wrapper classes https://github.com/apache/beam/pull/17150 [https://opengraph.githubassets.com/d3edcbc6791ade4994b3cfab1d0b97494257c961fa514a97f0491537311ba7b1/apache/beam/pull/17150]<https://github.com/apache/beam/pull/17150> [BEAM-14053] [CdapIO] Add wrapper class for CDAP plugin by ktttnv · Pull Request #17150 · apache/beam · GitHub<https://github.com/apache/beam/pull/17150> github.com Changes: Added wrapper classes for source and sink CDAP plugins Added builder for plugin wrapper class Added possibility to provide Hadoop configuration Warning: This PR contains changes from PR "[BEAM-14048] [CdapIO] Add ConfigWrapper for building CDAP PluginConfigs" and should be merged after it Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: Choose reviewer(s) and mention them in a comment (R: @username). Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. Update CHANGES.md with noteworthy changes. If this contribution is large, please file an Apache Individual Contributor License Agreement. See the Contributor Guide for more tips on how to make review process smoother. To check the build health, please visit https://github.com/apache/beam We would appreciate it very much if you could review ready PRs and leave comments. P.S. CommunityMetrics PR check is finishing with an error every time. We don't clearly understand the issue with this check. Any guidance on that would be helpful? Thank you for your attention to it, Elizaveta [1] TDD https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit?usp=sharing [2] CDAP plugin's configuration wrapper https://github.com/apache/beam/pull/17051, <https://github.com/apache/beam/pull/17051> https://issues.apache.org/jira/browse/BEAM-14048 [https://opengraph.githubassets.com/2568e6c2ec87aea825c75e3f7adf9808ee50f571bd44d034da842d1663066153/apache/beam/pull/17051]<https://github.com/apache/beam/pull/17051> [BEAM-14048] [CdapIO] Add ConfigWrapper for building CDAP PluginConfigs by Amar3tto · Pull Request #17051 · apache/beam · GitHub<https://github.com/apache/beam/pull/17051> github.com CDAP plugins use their own configuration classes (ex. SalesforceSourceConfig, SalesforceStreamingSourceConfig) to set the required parameters to prepare and run pipeline in Source or Sink classes. All these config classes are extended from the base io.cdap.cdap.api.plugin.PluginConfig class. As a first step in integration of CDAP plugins and Apache Beam, there should be implemented a mechanism to create an instance of the specific PluginConfig implementation. For this purpose, suggested solution provides a ConfigWrapper<T> class. Apache Beam user creates an instance of this class by passing the PluginConfig specific class as a T parameter. ConfigWrapper<T> object provides simple methods to instantiate the corresponding T object: fromJsonFile(File file) withParams(Map<String, Object> params) setParam(String paramName, Object value) build() Apache Beam user could set required parameters for the CDAP plugin by passing the Map/JSON file or by manually setting each p [3] CDAP context classes for CDAP plugins https://github.com/apache/beam/pull/17104 [https://opengraph.githubassets.com/9260a94844067566a77c159de11543f854fc4c3b1a735395873f5460c9beaeb2/apache/beam/pull/17104]<https://github.com/apache/beam/pull/17104> [BEAM-14081] [CdapIO] Add context classes for CDAP plugins by Krasavinigor · Pull Request #17104 · apache/beam · GitHub<https://github.com/apache/beam/pull/17104> github.com WARNING! This PR depends on PR #17051 Changes: Implements classes for BatchSourceContext, BatchSinkContext and StreamingSourceContext interfaces. Implements BatchContextImpl as part of has-a references for classes. Implements tests. We move all common logic to BatchContextImpl which implements the CDAP interface BatchContext. Each class extends BatchContextImpl and can override it if needed and contains its own logic. After creating the plugin config and an instance of [PluginName]BatchSink/BatchSource/StreamingSource we pass our context through the prepareRun method for in instance. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: Choose reviewer(s) and mention them in a comment (R: @username). Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. Update [4] CDAP plugin wrapper classes https://github.com/apache/beam/pull/17150
