hansva opened a new issue, #9433:
URL: https://github.com/apache/hop/issues/9433

   ### What would you like to happen?
   
   The Detect language transform always compares each text against all 75 
spoken languages that Lingua supports, and loads all 75 models into memory when 
the transform is created. There is no way to limit the languages or trade 
accuracy for speed. Most pipelines know which handful of languages their data 
can contain.
   
   Measured with Lingua 1.2.2 on 30,000 short texts (3 to 6 words), one thread:
   
   | Setup | Model load | Rows/s | Heap |
   |---|---|---|---|
   | All spoken languages (today) | 0.8-2.1 s | 5.4k | 1.4 GB |
   | All spoken languages, low accuracy | 0.3 s | 9.7k | 95 MB |
   | 5 languages | 0.3 s | 14.7k | 174 MB |
   | 5 languages, low accuracy | 0.1 s | 23.1k | 39 MB |
   
   ### Proposed changes
   
   1. **Languages to detect.** A new option that lists the languages the 
transform may answer. Empty means all spoken languages, which is today's 
behavior, so existing pipelines give the same output. With 5 languages the 
transform is about 2.7x faster and uses about an eighth of the memory.
   2. **Low accuracy mode.** A new checkbox, off by default. Lingua then uses 
only its trigram models: about 1.8x faster and far less memory, but clearly 
less accurate on short texts. On the test data it changed the detected language 
of most rows, so the tooltip and the documentation must say that it is meant 
for longer texts.
   3. **Build the detector in `init()`.** Today it is a field initializer, so 
every transform copy loads 1.4 GB when it is constructed. Build it in `init()` 
from the two options above, once per copy.
   4. **Keep unloading the models when the transform finishes.** Keeping them 
loaded would save 1 to 2 seconds on the next run in the same JVM, at the cost 
of up to 1.4 GB held for as long as Hop GUI or Hop Server runs. Not worth it.
   
   The plugin does not get smaller: Lingua ships all models in one 80 MB jar. 
The options only change what is loaded and compared.
   
   ### Dialog and metadata
   
   - Both options are `@HopMetadataProperty` fields with `@GuiWidgetElement` 
annotations, in a group, per the dialog rules. Both are available for metadata 
injection.
   - The language list is stored as Lingua's language names (`ENGLISH`, 
`FRENCH`), the same values the transform writes to `detected_language`. An 
unknown name fails the transform at start with a clear message.
   - `check()` warns when exactly one language is listed, since Lingua needs at 
least two to compare.
   
   ### Behavior to document
   
   - With a language list, a text in a language that is not listed gets one of 
the listed languages, or `UNKNOWN` when the two best candidates are too close. 
This is by design.
   - Lingua's models are shared by every Detect language transform in the JVM. 
When one transform finishes and unloads them, another that is still running 
reloads them on demand: results stay correct, it is only slower for a moment.
   
   ### Two existing bugs to fix in the same change
   
   Both are read from the code, not reproduced yet; each needs a test that 
fails first.
   
   - A row whose text field is null or blank is dropped: `process()` returns 
without writing the row. It should pass through with `UNKNOWN` and no 
confidence.
   - With "parallelism" on, a row that fails in a worker is logged, but its job 
counter is never decremented, so the transform waits forever at the end. The 
failure should stop the transform, or go to the error hop when error handling 
is on.
   
   ### Tests
   
   - Default options: same detected language and confidence as before for a 
fixed set of sentences.
   - Language list: only listed languages or `UNKNOWN` come out; an unknown 
name fails at start.
   - Low accuracy mode: the transform runs and fills both output fields.
   - The two bug fixes above.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to