[
https://issues.apache.org/jira/browse/DRILL-8283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584825#comment-17584825
]
ASF GitHub Bot commented on DRILL-8283:
---------------------------------------
jnturton commented on code in PR #2632:
URL: https://github.com/apache/drill/pull/2632#discussion_r954914617
##########
exec/java-exec/src/main/java/org/apache/drill/exec/util/FileSystemUtil.java:
##########
@@ -42,6 +47,12 @@ public class FileSystemUtil {
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FileSystemUtil.class);
+ private static int recursiveListingMaxSize;
+
+ static {
+ recursiveListingMaxSize =
DrillConfig.create().getInt(ExecConstants.RECURSIVE_FILE_LISTING_MAX_SIZE);
+ }
Review Comment:
That it might be a heavyweight duplication of work was bothering me enough
that I went and timed it. It takes about ~100ms when I start embedded Drill
locally. That's just enough to make me wonder if it's worth trying to redesign
this stuff so that it loads from an existing instance of DrillConfig instead of
constructing its own.
> Add a configurable recursive file listing size limit
> ----------------------------------------------------
>
> Key: DRILL-8283
> URL: https://issues.apache.org/jira/browse/DRILL-8283
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
> Affects Versions: 1.20.2
> Reporter: James Turton
> Assignee: James Turton
> Priority: Minor
> Fix For: 1.20.3
>
>
> Currently a malicious or merely unwitting user can crash their Drill foreman
> by sending
> {code:java}
> select * from dfs.huge_workspace limit 10
> {code}
> causing the query planner to recurse over every file in huge_workspace and
> culminating in
> {code:java}
> 2022-08-09 15:13:22,251 [1d0da29f-e50c-fd51-43d9-8a5086d52c4e:foreman] ERROR
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred,
> exiting. Information message: Unable to handle out of memory condition in
> Foreman.java.lang.OutOfMemoryError: null {code}
> if there are enough files in huge_workspace. A SHOW FILES command can produce
> the same effect. This issue proposes a new BOOT option named
> drill.exec.storage.file.recursive_listing_max_size with a default value of,
> say 10 000. If a file listing task exceeds this limit then the initiating
> operation is terminated with a UserException preventing runaway resource
> usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
