rszper commented on code in PR #30187: URL: https://github.com/apache/beam/pull/30187#discussion_r1476618331
########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-bigtable.md: ########## @@ -0,0 +1,55 @@ +--- +title: "Enrichment with Bigtable" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Use Enrichment with Bigtable Review Comment: ```suggestion # Use Bigtable to enrich data ``` ########## website/www/site/content/en/documentation/ml/overview.md: ########## @@ -90,7 +90,8 @@ You can use Apache Beam for data validation and preprocessing by setting up data | Task | Example | | ------- | ---------------| | I want to transform my data for preprocessing| [Preprocess data with MLTransform](/documentation/ml/preprocess-data) | -| I want to explore my data | [Data exploration workflow and example](/documentation/ml/data-processing) |: +| I want to explore my data | [Data exploration workflow and example](/documentation/ml/data-processing) | +| I want to enrich my data | [Data enrichment wth Enrichment transform](/documentation/transforms/python/elementwise/enrichment) |: Review Comment: After we publish the notebook, we should link to that instead. ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment.md: ########## @@ -0,0 +1,50 @@ +--- +title: "Enrichment" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Enrichment transform for data enrichment + +{{< localstorage language language-py >}} + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +Enrichment transform enable users to dynamically enrich their data within a pipeline by doing a key-value lookup to the remote service. It uses [RequestResponeIO](https://beam.apache.org/releases/pydoc/current/apache_beam.io.requestresponseio.html#apache_beam.io.requestresponseio.RequestResponseIO) internally that make sures that the remote service is not overloaded with requests by doing a client-side throttling whenever required. Additionally, in case of service-side errors like TooManyRequests and Timeout exceptions, it retries the requests with exponential backoff. + +Starting Apache Beam 2.54.0, the package provides a built-in enrichment handler for [Cloud Bigtable](https://cloud.google.com/bigtable?hl=en). Review Comment: ```suggestion In Apache Beam 2.54.0 and later versions, the transform includes a built-in enrichment handler for [Cloud Bigtable](https://cloud.google.com/bigtable?hl=en). ``` ########## website/www/site/layouts/partials/section-menu/en/documentation.html: ########## @@ -288,6 +288,7 @@ <span class="section-nav-list-title">Element-wise</span> <ul class="section-nav-list"> + <li><a href="/documentation/transforms/python/elementwise/enrichment/">Enrichment</a></li> Review Comment: If we're going to have the example on a separate page, we need an entry for both pages in the TOC, but I suggest putting all the content in one page. ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-bigtable.md: ########## @@ -0,0 +1,55 @@ +--- +title: "Enrichment with Bigtable" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Use Enrichment with Bigtable + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +The following examples demonstrate how to create pipelines that use the Enrichment transform and BigTableEnrichmentHandler. + +## Example: Enrich data with Bigtable Review Comment: ```suggestion ``` If we only have one example, we don't need this header; we can just show the example. ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment.md: ########## @@ -0,0 +1,50 @@ +--- +title: "Enrichment" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Enrichment transform for data enrichment + +{{< localstorage language language-py >}} + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +Enrichment transform enable users to dynamically enrich their data within a pipeline by doing a key-value lookup to the remote service. It uses [RequestResponeIO](https://beam.apache.org/releases/pydoc/current/apache_beam.io.requestresponseio.html#apache_beam.io.requestresponseio.RequestResponseIO) internally that make sures that the remote service is not overloaded with requests by doing a client-side throttling whenever required. Additionally, in case of service-side errors like TooManyRequests and Timeout exceptions, it retries the requests with exponential backoff. Review Comment: ```suggestion The enrichment transform lets you dynamically enrich data in a pipeline by doing a key-value lookup to a remote service. The transform uses [`RequestResponeIO`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.requestresponseio.html#apache_beam.io.requestresponseio.RequestResponseIO) internally. This feature uses client-side throttling to ensure that the remote service isn't overloaded with requests. If service-side errors occur, like `TooManyRequests` and `Timeout` exceptions, it retries the requests by using exponential backoff. ``` ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-bigtable.md: ########## @@ -0,0 +1,55 @@ +--- +title: "Enrichment with Bigtable" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Use Enrichment with Bigtable + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +The following examples demonstrate how to create pipelines that use the Enrichment transform and BigTableEnrichmentHandler. Review Comment: ```suggestion The following example demonstrates how to create pipelines that use the enrichment transform with `BigTableEnrichmentHandler`. ``` ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-bigtable.md: ########## @@ -0,0 +1,55 @@ +--- +title: "Enrichment with Bigtable" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Use Enrichment with Bigtable + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +The following examples demonstrate how to create pipelines that use the Enrichment transform and BigTableEnrichmentHandler. + +## Example: Enrich data with Bigtable + +In this example, we create a pipeline that uses a BigTableEnrichmentHandler with Enrichment transform. + +The data stored in the Bigtable cluster is of the following format: Review Comment: ```suggestion The data stored in the Bigtable cluster uses the following format: ``` ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment.md: ########## @@ -0,0 +1,50 @@ +--- +title: "Enrichment" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Enrichment transform for data enrichment + +{{< localstorage language language-py >}} + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +Enrichment transform enable users to dynamically enrich their data within a pipeline by doing a key-value lookup to the remote service. It uses [RequestResponeIO](https://beam.apache.org/releases/pydoc/current/apache_beam.io.requestresponseio.html#apache_beam.io.requestresponseio.RequestResponseIO) internally that make sures that the remote service is not overloaded with requests by doing a client-side throttling whenever required. Additionally, in case of service-side errors like TooManyRequests and Timeout exceptions, it retries the requests with exponential backoff. + +Starting Apache Beam 2.54.0, the package provides a built-in enrichment handler for [Cloud Bigtable](https://cloud.google.com/bigtable?hl=en). + +## Examples + +The following examples show how to create pipelines that use the Enrichment transform for data enrichment with built-in enrichment handlers. Review Comment: ```suggestion The following examples show how to create pipelines that use the enrichment transform for data enrichment with built-in enrichment handlers. ``` ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment-bigtable.md: ########## @@ -0,0 +1,55 @@ +--- +title: "Enrichment with Bigtable" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Use Enrichment with Bigtable + +<table> + <tr> + <td> + <a> + {{< button-pydoc path="apache_beam.transforms" class="Enrichment" >}} + </a> + </td> + </tr> +</table> + +The following examples demonstrate how to create pipelines that use the Enrichment transform and BigTableEnrichmentHandler. + +## Example: Enrich data with Bigtable + +In this example, we create a pipeline that uses a BigTableEnrichmentHandler with Enrichment transform. Review Comment: ```suggestion ``` The information in this line is covered in the intro paragraph, so we can just use that. If we add more examples, we can add a line that explains what's unique about each. ########## website/www/site/content/en/documentation/transforms/python/elementwise/enrichment.md: ########## @@ -0,0 +1,50 @@ +--- +title: "Enrichment" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Enrichment transform for data enrichment Review Comment: ```suggestion # Enrichment transform ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
