[
https://issues.apache.org/jira/browse/TIKA-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135983#comment-18135983
]
ASF GitHub Bot commented on TIKA-4408:
--------------------------------------
Lynnbb88 opened a new pull request, #3473:
URL: https://github.com/apache/tika/pull/3473
## Issue
[TIKA-4408](https://issues.apache.org/jira/browse/TIKA-4408)
Python scripts starting with `#!/usr/bin/env python` or `#!/usr/bin/env
python3` are detected as `application/x-sh`, including when a `.py` filename
hint is supplied. The existing magic rule looks for `/bin/env python` at offset
1, where these inputs contain `!`.
## Changes
Replace that rule with two literal prefixes anchored at offset 0:
```xml
<match value="#!/usr/bin/env python" type="string" offset="0"/>
<match value="#! /usr/bin/env python" type="string" offset="0"/>
```
This follows the paired no-space/single-space convention of the neighboring
direct-path Python rules and preserves prefix matching for `python`, `python3`,
and version suffixes such as `python3.11`.
Add 18 regression checks covering content-only detection, `.py` filename
hints, direct Python paths, Shell controls, filename-only detection, and
file-start anchoring. Add a changelog entry for the unreleased version. No new
dependencies or public API changes.
General whitespace handling, `env` options before the interpreter (such as
`-S`), and stricter interpreter-token boundaries are outside this change.
## Validation
Fresh local checks on macOS with JDK 21, based on Apache `main` at
`9f8b48425fea3630020924f6ae96e98eabb9ce12`:
- With the new tests and the original production rule: 18 tests, 7 failures,
0 errors.
- With the fix: all 18 Python regression checks passed.
- Full `tika-core` unit suite: 1,418 tests reported, 0 failures, 0 errors, 2
skipped.
- Alternate Xerces provider: 12 tests reported, 0 failures, 0 errors, 2
skipped.
- `TikaIT`: 1 test passed; annotation processor: all 8 tests passed.
- Checkstyle, Spotless, forbidden-API checks, and Apache RAT passed; `BUILD
SUCCESS`.
```sh
./mvnw -B -ntp clean install -pl tika-core -am -Ppedantic \
-Dtika.test.locale=random -Dmaven.repo.local="$PWD/.local_m2_repo"
```
The negative control used the same command with
`-Dtest=PythonShebangDetectionTest -Dsurefire.failIfNoSpecifiedTests=false`.
These checks cover core and its reactor prerequisites; the full repository
build has not been run.
## AI assistance
OpenAI Codex assisted with investigation, implementation, testing, and PR
preparation. The commit includes a `Generated-by` trailer.
> python file identified as application/x-sh under several circumstances
> ----------------------------------------------------------------------
>
> Key: TIKA-4408
> URL: https://issues.apache.org/jira/browse/TIKA-4408
> Project: Tika
> Issue Type: Bug
> Components: core
> Reporter: Carol Alexandru
> Priority: Major
> Labels: match, mime-type
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> The [definition for text/x-python in
> tika-mimetypes.xml|https://github.com/apache/tika/blob/25619272d2f615df4ad87e27e7c8dec576f37627/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml#L8347]
> is missing some matches. In particular
> * #!{+}/usr{+}/bin/env ...
> * all variants using python{+}3{+} instead of python
> For this reason, a file starting with any of the following valid and fairly
> common lines are misidentified as application/x-sh (which matches #!/)
> {{#!/usr/bin/env python3}}
> {{#!/usr/bin/env python}}
> {{{}#!{}}}{{{}/usr/bin/python3{}}}
> {{... etc ...}}
> I might do a pull request if I get around to it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
