Claus Ibsen created CAMEL-26923:
-----------------------------------
Summary: camel-duckdb - copy loads one local file unless options
allow URLs or globs
Key: CAMEL-26923
URL: https://issues.apache.org/jira/browse/CAMEL-26923
Project: Camel
Issue Type: Improvement
Reporter: Claus Ibsen
Assignee: Claus Ibsen
Fix For: 4.24.0, 4.22.3
The {{copy}} operation of the camel-duckdb producer takes the source file from
the message body and loads it into a table with DuckDB's {{read_csv}},
{{read_parquet}} or {{read_json}}.
h3. Behaviour today
{{DuckDbProducer.buildCopySql}} passes the resolved body value to the read
function as it is. DuckDB interprets that value itself: a value with {{*}},
{{?}}, {{[}} or {{\{}} is expanded as a glob pattern over many files, and a
value with a {{scheme://}} prefix (http, https, s3, ...) is read as a remote
source. The route has no way to say that {{copy}} should load one local file,
or only files from a given directory.
h3. Behaviour after the change
By default {{copy}} loads a single local file. Remote sources and glob patterns
become explicit opt-ins, and a base directory can restrict where the source may
be:
* {{allowUrlCopySource}} (default {{false}}): when {{false}}, a source that
starts with a URL scheme ({{^[A-Za-z][A-Za-z0-9+.-]*://}}) is refused.
* {{allowGlobCopySource}} (default {{false}}): when {{false}}, a source that
contains glob characters ({{*}}, {{?}}, {{[}}, {{\{}}) is refused. This applies
to String bodies as well as {{File}} and {{Path}} bodies.
* {{copyBaseDirectory}} (unset by default): when set, a local source must
resolve inside this directory, with symbolic links resolved
({{Path.toRealPath}}). For an allowed glob, the fixed part before the first
glob character is checked.
A refused source fails the exchange with an {{IllegalArgumentException}} whose
message names the option to set.
h3. Classes and code changes
* {{DuckDbProducer}}: a check in {{doCopy}} / {{resolveFilePath}}, before
{{buildCopySql}}, for the URL scheme, glob characters and base directory
containment.
* {{DuckDbEndpoint}}: the three new {{@UriParam}} options (label {{producer}}),
{{allowUrlCopySource}} marked as {{insecure:dev}}.
* {{duckdb-component.adoc}}: document that {{copy}} loads one local file, the
new options, and optionally DuckDB's own settings ({{enable_external_access}},
{{allowed_directories}}, {{autoinstall_known_extensions}}) for deployments that
want DuckDB to enforce the same limits for all operations.
* Regenerated files: component JSON, endpoint configurer and URI factory,
endpoint DSL builder, catalog JSON and doc copy.
* Upgrade guide: 4.24 entry (and a 4.22.3 entry if backported) stating that
{{copy}} no longer accepts a URL or glob pattern unless {{allowUrlCopySource}}
/ {{allowGlobCopySource}} is set.
h3. Tests
In {{DuckDbProducerTest}}, without network access:
* a URL source is refused by default;
* a glob pattern is refused by default, and loads all matching files (two CSV
files in a temporary directory) when {{allowGlobCopySource=true}};
* a {{File}} / {{Path}} body whose name contains glob characters is refused by
default;
* with {{copyBaseDirectory}}: a file inside the directory loads; an absolute
path outside it, a {{../}} path and a symbolic link pointing outside it are
refused;
* the existing {{copyFromCsvFile}}, {{copyFromPathBody}} and
{{copyFromBytesBody}} tests keep passing.
The camel-4.22.x branch has the component without the {{Path}} / {{byte[]}}
body support, so a backport there covers {{String}} and {{File}} bodies and
regenerates the endpoint files on the branch.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)