quentin-sommer commented on issue #34270: URL: https://github.com/apache/beam/issues/34270#issuecomment-2792988807
This is the python env: <details> <summary>packages</summary> ``` SecretStorage>=3.2 = 3.3.3 annotated-types>=0.6.0 = 0.7.0 apache-beam==2.63.0 = 2.63.0 attrs>=22.2.0 = 24.3.0 backports.tarfile = 1.2.0 cachetools<6,>=3.1.0 = 5.5.0 certifi>=2017.4.17 = 2024.12.14 cffi>=1.12 = 1.17.1 charset-normalizer<4,>=2 = 3.4.1 cloudpickle~=2.2.1 = 2.2.1 crcmod<2.0,>=1.7 = 1.7 cryptography>=2.0 = 44.0.0 deprecated>=1.2.6 = 1.2.15 dill<0.3.2,>=0.3.1.1 = 0.3.1.1 dnspython<3.0.0,>=1.16.0 = 2.7.0 docopt = 0.6.2 docstring-parser<1 = 0.16 fastavro<2,>=0.23.6 = 1.10.0 fasteners<1.0,>=0.3 = 0.19 google-api-core<3,>=2.0.0 = 2.24.0 google-apitools<0.5.32,>=0.5.31 = 0.5.31 google-auth-httplib2<0.3.0,>=0.1.0 = 0.2.0 google-auth<3,>=1.18.0 = 2.37.0 google-cloud-aiplatform<2.0,>=1.26.0 = 1.77.0 google-cloud-bigquery-storage<3,>=2.6.3 = 2.27.0 google-cloud-bigquery<4,>=2.0.0 = 3.27.0 google-cloud-bigtable<3,>=2.19.0 = 2.28.1 google-cloud-core<3,>=2.0.0 = 2.4.1 google-cloud-datastore<3,>=2.0.0 = 2.20.2 google-cloud-dlp<4,>=3.0.0 = 3.26.0 google-cloud-language<3,>=2.0 = 2.16.0 google-cloud-pubsub<3,>=2.1.0 = 2.27.2 google-cloud-pubsublite<2,>=1.2.0 = 1.11.1 google-cloud-recommendations-ai<0.11.0,>=0.1.0 = 0.10.15 google-cloud-resource-manager<3.0.0dev,>=1.3.3 = 1.14.0 google-cloud-spanner<4,>=3.0.0 = 3.51.0 google-cloud-storage<3,>=2.18.2 = 2.19.0 google-cloud-videointelligence<3,>=2.0 = 2.15.0 google-cloud-vision<4,>=2 = 3.9.0 google-crc32c<2.0dev,>=1.0 = 1.6.0 google-resumable-media<3.0dev,>=2.0.0 = 2.7.2 googleapis-common-protos<2.0.dev0,>=1.56.2 = 1.66.0 grpc-google-iam-v1<1.0.0dev,>=0.12.4 = 0.14.0 grpc-interceptor>=0.15.4 = 0.15.4 grpcio!=1.48.0,!=1.59.*,!=1.60.*,!=1.61.*,!=1.62.0,!=1.62.1,<1.66.0,<2,>=1.33.1 = 1.65.5 grpcio-status>=1.33.2 = 1.65.5 hdfs<3.0.0,>=2.1.0 = 2.7.3 httplib2<0.23.0,>=0.8 = 0.22.0 idna<4,>=2.5 = 3.10 importlib-metadata<=8.5.0,>=6.0 = 8.5.0 jaraco.classes = 3.4.0 jaraco.context = 6.0.1 jaraco.functools = 4.1.0 jeepney>=0.4.2 = 0.8.0 jsonpickle<4.0.0,>=3.0.0 = 3.4.2 jsonschema-specifications>=2023.03.6 = 2024.10.1 jsonschema<5.0.0,>=4.0.0 = 4.23.0 keyring = 25.6.0 keyrings.google-artifactregistry-auth = 1.1.2 more-itertools = 10.6.0 msgpack = 1.1.0 numpy<2.3.0,>=1.14.3 = 2.2.1 oauth2client>=1.4.12 = 4.1.3 objsize<0.8.0,>=0.6.1 = 0.7.0 opentelemetry-api>=1.27.0 = 1.29.0 opentelemetry-sdk>=1.27.0 = 1.29.0 opentelemetry-semantic-conventions==0.50b0 = 0.50b0 orjson<4,>=3.9.7 = 3.10.14 overrides<8.0.0,>=6.0.1 = 7.7.0 packaging>=22.0 = 24.2 pluggy = 1.5.0 proto-plus<2,>=1.7.1 = 1.25.0 protobuf!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.*,<6.0.0.dev0,>=3.20.3 = 5.29.3 pyarrow-hotfix<1 = 0.6 pyarrow<17.0.0,>=3.0.0 = 16.1.0 pyasn1-modules>=0.2.1 = 0.4.1 pyasn1>=0.1.7 = 0.6.1 pycparser = 2.22 pydantic-core==2.27.2 = 2.27.2 pydantic<3 = 2.10.5 pydot<2,>=1.2.0 = 1.4.2 pymongo<5.0.0,>=3.8.0 = 4.10.1 pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 = 3.2.1 python-dateutil = 2.9.0.post0 pytz>=2018.3 = 2024.2 pyyaml<7.0.0,>=3.12 = 6.0.2 redis<6,>=5.0.0 = 5.2.1 referencing>=0.28.4 = 0.36.1 regex>=2020.6.8 = 2024.11.6 requests<3.0.0,>=2.24.0 = 2.32.3 rpds-py>=0.7.1 = 0.22.3 rsa<5,>=3.1.4 = 4.9 shapely<3.0.0dev = 2.0.6 six>=1.5 = 1.17.0 sortedcontainers>=2.4.0 = 2.4.0 sqlparse>=0.4.4 = 0.5.3 typing-extensions>=3.7.0 = 4.12.2 urllib3<3,>=1.21.1 = 2.3.0 wrapt<2,>=1.10 = 1.17.2 zipp>=3.20 = 3.21.0 zstandard<1,>=0.18.0 = 0.23.0 ``` </details> Running with Apache Beam Python 3.11 SDK 2.63.0 in streaming mode > And are these errors transient? Dataflow should always retry these errors. That's the thing that puzzles me. They get retried until they reach the maximum retry time then they are re-thrown. So they're handled by the google api core package (see the `retry_wrapped_func` call in the stacktrace) but it seems after it bubbles up to the beam bigquery error handling its not considered a valid error to catch and it cause a infinite error loop for the bundle -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
