waltage commented on PR #21867: URL: https://github.com/apache/beam/pull/21867#issuecomment-1187593349
@pabloem no worries on the review timeline. I am not the original reporter of the Jira issue, but it looks like this particular feature was requested with an objective of bringing the Python API closer in line with Java API. Since the issue was ported over to github and tagged with first issue, and since Ken in the original issue said it seemed like a good idea, I went ahead and worked on implementing it here. I agree that this particular initializer is quite dense, regardless of whether this option is promoted to a parameter. I think there are a few ways I could go with this: 1. (assuming I can update the other unit tests that use this transform, and provide sufficient integration coverage) Allow this option promotion in light of the already dense parameter set. 2. Develop type and value checking for the ...bq_parameters... dict, which would allow the surface of the initializer to remain consistent. 3. Close the issue/pr, as interest looks like it has subsided. 4. Abstract away a **majority** of the transform's initializer parameters that are BigQuery-specific into a new type/value-checked Class that would allow more flexibility for changes like this (or changes to the BigQuery APIs themselves) without modifying existing surfaces of the transform itself. In a related note, as a tie-breaker for 1 & 2 & 3, and in support of the design of 4 above, does the BEAM team currently analyze any of the public usages of its python libraries? I can look at how this transform is used "in the wild" and try to understand how often parameters are used vs. what keys of the dict are typically set (e.g., at first glance, it looks like partitioning specs are by far the most commonly used keys in the ...bq_parameters... dict). I'm happy to work on any of the above, just let me know how best to proceed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
