James Cho created FLINK-27928:
---------------------------------
Summary: External Resource Framework: 'external-resources' config
delimiter is not working as expected
Key: FLINK-27928
URL: https://issues.apache.org/jira/browse/FLINK-27928
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.15.0
Reporter: James Cho
For flink version 1.15.0, the configuration `external-resources` delimiter
separation(";") functionality between external resource names is not working as
expected.
Reproducing the bug setting up a flink cluster running using docker with two
external resources named `abc` and `efg`:
docker-compose.yml:
{code:java}
services:
jobmanager:
image: flink:1.15.0-scala_2.12
ports:
- "8081:8081"
- "6123:6123"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.15.0-scala_2.12
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 8
external-resources: abc;efg
external-resource.abc.amount: 1
external-resource.efg.amount: 1{code}
Related task manager logs:
{code:java}
root@2873c5ae72c2:/opt/flink# cat log/flink* | grep ExternalResourceUtils
2022-06-07 05:42:34,232 WARN
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - The amount
of the abc;efg should be configured. Will ignore that resource.
2022-06-07 05:42:34,232 INFO
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled
external resources: []
2022-06-07 05:42:34,236 WARN
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not
find driver class name for abc;efg. Please make sure
external-resource.abc;efg.driver-factory.class is configured.
2022-06-07 05:42:34,240 WARN
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - The amount
of the abc;efg should be configured. Will ignore that resource.
{code}
Notice flink recognizes `abc;efg` as a single external resource rather than as
`abc` and `efg`
The same issue does not exist in flink version 1.14.4
docker-compose.yml:
{code:java}
services:
jobmanager:
image: flink:1.14.4-scala_2.12
ports:
- "8081:8081"
- "6123:6123"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.14.4-scala_2.12
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 8
external-resources: abc;efg
external-resource.abc.amount: 1
external-resource.efg.amount: 1 {code}
related tm logs
{code:java}
docker logs ... | grep ExternalResourceUtils
2022-06-07 05:48:13,531 INFO
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled
external resources: [abc, efg]
2022-06-07 05:48:13,534 WARN
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not
find driver class name for abc. Please make sure
external-resource.abc.driver-factory.class is configured.
2022-06-07 05:48:13,534 WARN
org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not
find driver class name for efg. Please make sure
external-resource.efg.driver-factory.class is configured.
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
