Mario, this link may be useful for “reading large files in chunks” …
https://www.jsoftware.com/help/learning/28.htm
See Section 28.2 Large Files (and associated parameters for Offset;Length).
You would need to “stream through the file" yourself as in:
- Initialise empty buffer
Loop:
- Read next chunk and append to buffer
- parse complete sections in the buffer to output
- leave residual in a buffer
- continue loop
- etc
And manage incomplete XML definitions, balanced sections etc but all the tools
are there.
HTH, Rob
> On 17 Aug 2021, at 8:03 am, Mariusz Grasko <[email protected]> wrote:
>
> I have answered 1 and 2 by myself, it should be doable with !: verb -
> foreigns.
>
> Best regards,
> MG
>
> pon., 16 sie 2021 o 22:22 Mariusz Grasko <[email protected]>
> napisał(a):
>
>> Thank you for some ideas on using external parser.
>> Okay now I have 3 questions:
>> 1. Is it possible to read CSV file streaming-style (for example record by
>> record) without loading everything in memory ? Even if I use some external
>> parsing solution like XSLT or just write something myself in some other
>> language than J, I will end up with large CSV instead of large XML. It
>> makes no difference. The reason that I need to parse it like this, is that
>> there are some rows that I won't need, those would be discarded depending
>> on their field values.
>> If it is not possible I would do more work outside of J in this first
>> parser XML -> CSV.
>> 2. Is there a way to call external program for J script ? If it is
>> possible to wait for it to finish ?
>> If it is not possible, there are definiately ways to run J from other
>> programs.
>> 3. Can someone give a little bit of pointer or on how to use api/expat
>> library ? Do I need to familiarize myself with expat (C library) or just
>> good understanding of J and reading small test in package directory should
>> be enough ?
>> I could send some example file like Devon McCormick suggested.
>>
>> Right now I am working through book "J:The natural language for analytic
>> computing" and playing around with problems like Project Euler, but I could
>> really see myself using J in serious work.
>>
>> Best regards,
>> MG
>>
>>
>> śr., 11 sie 2021 o 09:51 <[email protected]> napisał(a):
>>
>>> In similar situations -but my files are not huge- I extract what I want
>>> into flattened CSV using one or more XQuery scripts, and then load the CSV
>>> files with J. The code is clean, compact and easy to maintain. For
>>> recurrent XQuery patterns, m4 occasionally comes to the rescue. Expect
>>> minor portability issues when using different XQuery processors
>>> (extensions, language level...).
>>>
>>>
>>>
>>> Never got round to SAX parsing beyond tutorials, so I cannot compare.
>>>
>>>
>>> De : Mariusz Grasko <[email protected]>
>>> À : [email protected]
>>> Sujet : [Jprogramming] Is is good idea to use J for reading large XML
>>> files ?
>>> Date : 10/08/2021 18:05:45 Europe/Paris
>>>
>>> Hi,
>>>
>>> We are ecommerce company and have a lot of integrations with suppliers,
>>> products info is nearly always in XML files. I am thinking about using J
>>> as
>>> an analysis tool, do you think that working with large files that need to
>>> be parsed SAX- style without reading everything at once is good idea in J
>>> ?
>>> Also is this even advantageous (as in, would code be terse). Right now XML
>>> parsing is done in Golang, so if parsing in J is not very good we could
>>> try
>>> to rely more on CSV exports. CSV is definiately very good in J.
>>> I am hoping that maybe XML parsing is very good in J and the code would
>>> become much smaller, if this is the case, then I would think about using J
>>> for XMLs with new suppliers.
>>>
>>> Best Regards
>>> M.G.
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>
>>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
