tvalentyn commented on code in PR #28702: URL: https://github.com/apache/beam/pull/28702#discussion_r1340509627
########## website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md: ########## @@ -0,0 +1,60 @@ +--- +type: languages +title: "Unrecoverable Errors in Beam Python" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Unrecoverable Errors in Beam Python + +## What is an Unrecoverable Error? + +An unrecoverable error is an issue at job start-up time that will +prevent a job from ever running successfully, usually due to some kind +of misconfiguration. Solving these issues when they occur is key to +successfully running a Beam Python pipeline. + +## Common Unrecoverable Errors + +### Job Submission/Runtime Python Version Mismatch + +If the Python version used for job submission does not match the +Python version used to build the worker container, the job will not +execute. Ensure that the Python version being used for job submission +and the container Python version match. + +### PIP Dependency Resolution Failures + +During worker start-up, dependencies are checked and installed in +the worker container before accepting work. If a pipeline requires +additional dependencies not already present in the runtime environment, +they are installed here. If there’s an issue during this process +(e.g. a dependency version cannot be found, or a worker cannot +connect to PyPI) the worker will fail and may try to restart +depending on the runner. Ensure that dependency versions provided in +your requirements.txt file exist and can be installed locally before +submitting jobs. + +### Dependency Verision Mismatches Review Comment: ```suggestion ### Dependency Version Mismatches ``` ########## website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md: ########## @@ -0,0 +1,56 @@ +--- +type: languages +title: "Unrecoverable Errors in Beam Python" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Unrecoverable Errors in Beam Python + +## What is an Unrecoverable Error? + +An unrecoverable error is an issue at job start-up time that will +prevent a job from ever running successfully, usually due to some kind +of misconfiguration. Solving these issues when they occur is key to +successfully running a Beam Python pipeline. + +## Common Unrecoverable Errors + +### Job Submission/Runtime Python Version Mismatch + +If the Python version used for job submission does not match the +Python version used to build the worker container, the job will not +execute. Ensure that the Python version being used for job submission +and the container Python version match. + +### PIP Dependency Resolution Failures + +During worker start-up, dependencies are checked and installed in +the worker container before accepting work. If there’s an issue during +this process (e.g. a dependency version cannot be found) the worker +will restart and try again up to four times before outright failing. +Ensure that dependency versions provided in your requirements.txt file +exist and can be installed locally before submitting jobs. + +### Dependency Verision Mismatches + +When additional dependencies like torch, transformers, etc are not +specified via requirements_file or preinstalled with a custom container +then the worker may go into a restart loop trying to install dependencies Review Comment: Possible suggestion: When additional dependencies like `torch`, `transformers`, etc. are not specified via requirements_file or preinstalled in a custom container then the worker might fail to deserialize (unpickle) the user code. This can result in `ModuleNotFound` errors. If dependencies are installed but their versions don't match the versions in submission environment, pipeline might have `AttributeError` messages. ########## website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md: ########## @@ -0,0 +1,60 @@ +--- +type: languages +title: "Unrecoverable Errors in Beam Python" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Unrecoverable Errors in Beam Python + +## What is an Unrecoverable Error? + +An unrecoverable error is an issue at job start-up time that will +prevent a job from ever running successfully, usually due to some kind +of misconfiguration. Solving these issues when they occur is key to +successfully running a Beam Python pipeline. + +## Common Unrecoverable Errors + +### Job Submission/Runtime Python Version Mismatch Review Comment: If you plan to reference these errors in logs, let's add markdown anchors for better linkability, as titles might change but we can keep the same anchors, so links will be preserved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
