Abacn opened a new issue, #36591:
URL: https://github.com/apache/beam/issues/36591
### What happened?
Simple pipeline:
```
pipeline:
transforms:
- type: Create
config:
elements:
- { "col1": 1 }
- { "col1": 2 }
- { "col1": 3 }
- type: WriteToBigQuery
config:
table: "[project]:[dataset].bqyamltest"
schema:
fields:
- name: "col1"
type: "INTEGER"
```
Run with
```
python -m apache_beam.yaml.main --yaml_pipeline_file=bqrow.yaml
--runner=DirectRunner
```
at first, it shows error
```
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File ".../site-packages/apache_beam/yaml/main.py", line 297, in <module>
run()
File ".../site-packages/apache_beam/yaml/main.py", line 148, in run
constructor(p)
File ".../site-packages/apache_beam/yaml/main.py", line 281, in constructor
yaml_transform.expand_pipeline(
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 1408, in
expand_pipeline
providers or {})).expand(root)
^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 1369, in
expand
result = expand_transform(
^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 482, in
expand_transform
return expand_composite_transform(spec, scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 832, in
expand_composite_transform
return transform.expand(None)
^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 815, in
expand
inner_scope.compute_all()
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 209, in
compute_all
self.compute_outputs(transform_id)
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 107, in
wrapper
self._cache[key] = func(self, *args)
^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 245, in
compute_outputs
return expand_transform(self._transforms_by_uuid[transform_id], self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 484, in
expand_transform
return expand_leaf_transform(spec, scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 515, in
expand_leaf_transform
ptransform = scope.create_ptransform(spec, inputs_dict.values())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 395, in
create_ptransform
provider.requires_inputs(spec['type'], config)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_provider.py", line 1499, in
requires_inputs
return self._underlying_provider.requires_inputs(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_provider.py", line 230, in
requires_inputs
if self._urns[typ] in self.schema_transforms():
^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_provider.py", line 207, in
schema_transforms
self._service = self._service()
^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_provider.py", line 383, in
<lambda>
jar_provider(), classpath=classpath),
^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_provider.py", line 344, in
<lambda>
urns, lambda: subprocess_server.JavaJarServer.path_to_beam_jar(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/utils/subprocess_server.py", line 417,
in path_to_beam_jar
raise RuntimeError(
RuntimeError:
...lib/sdks/java/extensions/sql/expansion-service/build/libs/beam-sdks-java-extensions-sql-expansion-service-2.70.0-SNAPSHOT.jar
not found. Please build the server with
```
after building sql expansion service, it now shows error
```
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File ".../site-packages/apache_beam/yaml/main.py", line 297, in <module>
run()
File ".../site-packages/apache_beam/yaml/main.py", line 148, in run
constructor(p)
File ".../site-packages/apache_beam/yaml/main.py", line 281, in constructor
yaml_transform.expand_pipeline(
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 1408, in
expand_pipeline
providers or {})).expand(root)
^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 1369, in
expand
result = expand_transform(
^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 482, in
expand_transform
return expand_composite_transform(spec, scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 832, in
expand_composite_transform
return transform.expand(None)
^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 815, in
expand
inner_scope.compute_all()
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 209, in
compute_all
self.compute_outputs(transform_id)
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 107, in
wrapper
self._cache[key] = func(self, *args)
^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 245, in
compute_outputs
return expand_transform(self._transforms_by_uuid[transform_id], self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 484, in
expand_transform
return expand_leaf_transform(spec, scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 515, in
expand_leaf_transform
ptransform = scope.create_ptransform(spec, inputs_dict.values())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../site-packages/apache_beam/yaml/yaml_transform.py", line 453, in
create_ptransform
raise ValueError(
ValueError: Invalid transform specification at "WriteToBigQuery" at line 9:
.../sdks/java/io/google-cloud-platform/expansion-service/build/libs/beam-sdks-java-io-google-cloud-platform-expansion-service-2.70.0-SNAPSHOT.jar
not found. Please build the server with
cd .../lib; ./gradlew
sdks:java:io:google-cloud-platform:expansion-service:shadowJar
```
we should really only need gcp-io expansion service
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
