gcarson30 opened a new issue, #34879:
URL: https://github.com/apache/beam/issues/34879
### What happened?
Issue Description:
I am encountering a ValueError when setting the resource_hints parameter in
StandardOptions for an Apache Beam pipeline. Specifically, the min_ram resource
hint is being evaluated as None, even though it is explicitly set to a valid
value (e.g., "16GB" or "1024MB").
The error occurs during pipeline initialization, and the full stack trace is
provided below.
Steps to Reproduce:
Create a Python script with the following minimal reproducible example:
`from apache_beam.options.pipeline_options import PipelineOptions,
StandardOptions
def run_pipeline():
options = PipelineOptions()
standard_options = options.view_as(StandardOptions)
standard_options.resource_hints = {"min_ram": "16GB"} # Explicitly set
min_ram
# Initialize the pipeline
with beam.Pipeline(options=options) as p:
pass # Minimal pipeline logic
if __name__ == "__main__":
run_pipeline()`
Run the script using Python 3.11 and Apache Beam version 2.63.0.
Expected Behavior:
The pipeline should initialize successfully, and the min_ram resource hint
should be correctly interpreted as "16GB".
Actual Behavior:
The pipeline fails to initialize, and the following error is raised:
`Traceback (most recent call last):
File "/path/to/apache_beam/transforms/resources.py", line 194, in
parse_resource_hints
parsed_hints.update(hint_cls.parse(value))
^^^^^^^^^^^^^^^^^^^^^
File "/path/to/apache_beam/transforms/resources.py", line 162, in parse
return {cls.urn: ResourceHint._parse_storage_size_str(value)}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/apache_beam/transforms/resources.py", line 115, in
_parse_storage_size_str
raise ValueError("Input must be a string or integer.")
ValueError: Input must be a string or integer.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "example.py", line 12, in <module>
run_pipeline()
File "example.py", line 8, in run_pipeline
with beam.Pipeline(options=options) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/apache_beam/pipeline.py", line 238, in __init__
self._root_transform().resource_hints =
resource_hints_from_options(options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/apache_beam/transforms/resources.py", line 216, in
resource_hints_from_options
return parse_resource_hints(hints)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/apache_beam/transforms/resources.py", line 196, in
parse_resource_hints
raise ValueError(f"Resource hint {hint} has invalid value {value}.")
ValueError: Resource hint min_ram has invalid value None.`
Environment:
Apache Beam Version: 2.63.0
Python Version: 3.11
Execution Environment: Local machine (Linux)
Additional Context:
I have verified that the min_ram value is explicitly set to "16GB" as a
string.
I have also tested with other valid values (e.g., "1024MB", 16 * 1024**3),
but the issue persists.
The issue appears to originate from the resource_hints_from_options function
in apache_beam/transforms/resources.py.
Suggested Fix:
It seems that the resource_hints dictionary is not being processed
correctly. A potential fix could involve adding additional validation or
debugging the parse_resource_hints function to ensure that valid values are not
incorrectly interpreted as None.
Request:
Please investigate this issue and provide guidance on how to resolve it. Let
me know if additional information or testing is required.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
