Copilot commented on code in PR #3123:
URL: https://github.com/apache/tika/pull/3123#discussion_r3925846689
##########
tika-eval/tika-eval-app/src/main/java/org/apache/tika/eval/app/ExtractProfileRunner.java:
##########
@@ -124,8 +124,8 @@ private static String getJdbcConnectionString(String
dbPath) {
}
//default to h2
Path p = Paths.get(dbPath);
- return "jdbc:h2:file:" + p.toAbsolutePath();
-
+ // drop the 45s MVStore chunk retention (bloat + growing compaction
cost); CACHE_SIZE is KB
+ return "jdbc:h2:file:" + p.toAbsolutePath() +
";RETENTION_TIME=0;CACHE_SIZE=1048576";
Review Comment:
The default H2 JDBC URL hard-codes a very large page cache
(CACHE_SIZE=1048576 KB = 1 GiB). This can significantly increase memory usage
for users running tika-eval on constrained machines/CI. Consider making
RETENTION_TIME/CACHE_SIZE configurable (e.g., via system properties) rather
than always forcing these values.
##########
tika-eval/tika-eval-app/src/main/java/org/apache/tika/eval/app/ExtractComparerRunner.java:
##########
@@ -183,8 +183,8 @@ private static String getJdbcConnectionString(String
dbPath) {
}
//default to h2
Path p = Paths.get(dbPath);
- return "jdbc:h2:file:" + p.toAbsolutePath();
-
+ // drop the 45s MVStore chunk retention (bloat + growing compaction
cost); CACHE_SIZE is KB
+ return "jdbc:h2:file:" + p.toAbsolutePath() +
";RETENTION_TIME=0;CACHE_SIZE=1048576";
Review Comment:
The default H2 JDBC URL hard-codes a very large page cache
(CACHE_SIZE=1048576 KB = 1 GiB). This can significantly increase memory usage
for users running tika-eval on constrained machines/CI. Consider making
RETENTION_TIME/CACHE_SIZE configurable (e.g., via system properties) rather
than always forcing these values.
##########
tika-langdetect/tika-langdetect-charsoup/src/test/java/org/apache/tika/langdetect/charsoup/core/CharSoupFeatureExtractorTest.java:
##########
@@ -93,6 +98,87 @@ public void testNFCNormalization() {
assertArrayEquals(countsC, countsD);
}
+ @Test
+ public void testUrlMailStrippingMatchesGreedyRegexReference() {
+ // TIKA-4875: the scanners must stay byte-identical to the regexes
they replaced --
+ // the langdetect and junkdetect models were trained on this exact
preprocessing.
+ // Inputs are NFC-stable, so preprocessNoTruncate's NFC step is an
identity here.
+ Pattern greedyUrl =
Pattern.compile("https?://[-_.?&~;+=/#0-9A-Za-z]{10,10000}");
+ Pattern greedyMail =
Pattern.compile("[-_.0-9A-Za-z]{1,100}@[-_0-9A-Za-z]{1,100}[-_.0-9A-Za-z]{1,100}");
+
+ List<String> cases = new ArrayList<>();
+ // middle mail repeat must backtrack to feed the dot-class tail
+ cases.add("a@bb");
+ cases.add("a@b");
+ cases.add("[email protected]");
+ cases.add("[email protected]");
+ cases.add("a@" + "b".repeat(250));
+ cases.add("a".repeat(150) + "@x.y");
+ cases.add("a".repeat(200) + "@");
+ cases.add("local@local@local");
+ cases.add("[email protected] a@bb [email protected]");
+ cases.add("@@@@@@");
+ cases.add("a@a@a@a@a@a@");
+ // mail caps: head {1,100} then tail {1,100} over one long run
+ cases.add("a@" + "b".repeat(100) + "." + "c".repeat(150));
+ cases.add("a@" + "b".repeat(99) + "." + "c".repeat(99));
+ cases.add("a@" + "b".repeat(300));
+ // find() resumes after a match: leftover run chars are not a fresh
local part
+ cases.add("aa@bb cc@dd ee@ff");
+ cases.add("a@bb@cc@dd");
+ // URL length boundaries: min 10 after scheme, cap 10000
+ cases.add("http://"; + "a".repeat(9));
+ cases.add("http://"; + "a".repeat(10));
+ cases.add("https://"; + "a".repeat(10000));
+ cases.add("https://"; + "a".repeat(10001));
+ cases.add("https://"; + "a".repeat(10005) + "@bb");
+ cases.add("http://http://aaaaaaaaaa";);
+ cases.add("http://aaaahttp://bbbbbbbbbb";);
+ cases.add("hhttp://aaaaaaaaaaa";);
+ cases.add("http:/notaurl http//nope https:/x");
+ // the URL pass runs first; its replacement is a barrier for the mail
pass
+ cases.add("http://aaaaaaaaaa@bb";);
+ cases.add("a@http://aaaaaaaaaa";);
+ cases.add("[email protected]");
+ cases.add("[email protected]/http://foobarbazqux";);
+ // non-ASCII neighbors exercise the < 128 guards
+ cases.add("é@bb");
+ cases.add("aé@bb");
+ cases.add("a@büc.d");
+ cases.add("http://aéaaaaaaaaaa";);
+ cases.add("see http://example.com/a/b?q=1#f and mail
bob.smith@sub-domain_x.example.org.");
+
+ long seed = new Random().nextLong();
+ Random random = new Random(seed);
Review Comment:
This test uses a non-deterministic random seed (`new Random().nextLong()`),
which can make CI failures hard to reproduce (reruns likely generate different
cases). Use a fixed default seed (optionally overridable via a system property)
so any failure is reliably repeatable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
