Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2024-10-01 at 09:09:07 +1000, Chris Angelico via Python-list wrote: > On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list > wrote: > > > > On 2024-09-30, Dan Sommers via Python-list wrote: > > > > > In Common Lisp, integers can be written in any integer base from two > > > to thirty s

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list wrote: > > On 2024-09-30, Dan Sommers via Python-list wrote: > > > In Common Lisp, integers can be written in any integer base from two > > to thirty six, inclusive. So knowing the last digit doesn't tell > > you whether an integer is ev

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2024-09-30, Dan Sommers via Python-list wrote: > In Common Lisp, integers can be written in any integer base from two > to thirty six, inclusive. So knowing the last digit doesn't tell > you whether an integer is even or odd until you know the base > anyway. I had to think about that for an

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2024-10-01 at 04:46:35 +1000, Chris Angelico via Python-list wrote: > On Tue, 1 Oct 2024 at 04:30, Dan Sommers via Python-list > wrote: > > > > But why do I need to start with the least > > significant digit? > > If you start from the most significant, you don't know anything about > the num

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 9/30/2024 11:30 AM, Barry via Python-list wrote: On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list wrote: import polars as pl pl.read_json("file.json") This is not going to work unless the computer has a lot more the 60GiB of RAM. As later suggested a streaming par

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

> Streaming won't work because the file is gzipped. You have to receive > the whole thing before you can unzip it. Once unzipped it will be even > larger, and all in memory. GZip is specifically designed to be streamed. So, that's not a problem (in principle), but you would need to have a stream

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 9/30/2024 1:00 PM, Chris Angelico via Python-list wrote: On Tue, 1 Oct 2024 at 02:20, Thomas Passin via Python-list wrote: On 9/30/2024 11:30 AM, Barry via Python-list wrote: On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list wrote: import polars as pl pl.read_json("

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On Tue, 1 Oct 2024 at 04:30, Dan Sommers via Python-list wrote: > > But why do I need to start with the least > significant digit? If you start from the most significant, you don't know anything about the number until you finish parsing it. There's almost nothing you can say about a number given

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2024-09-30, Dan Sommers via Python-list wrote: > On 2024-09-30 at 11:44:50 -0400, > Grant Edwards via Python-list wrote: > >> On 2024-09-30, Left Right via Python-list wrote: >> > [...] >> > Imagine a pathological case of this shape: 1... <60GB of digits>. This >> > is still a valid JSON (it

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2024-09-30 at 11:44:50 -0400, Grant Edwards via Python-list wrote: > On 2024-09-30, Left Right via Python-list wrote: > > Whether and to what degree you can stream JSON depends on JSON > > structure. In general, however, JSON cannot be streamed (but commonly > > it can be). > > > > Imagine a

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On Tue, 1 Oct 2024 at 02:20, Thomas Passin via Python-list wrote: > > On 9/30/2024 11:30 AM, Barry via Python-list wrote: > > > > > >> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list > >> wrote: > >> > >> > >> import polars as pl > >> pl.read_json("file.json") > >> > >> > > >

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 9/30/2024 11:30 AM, Barry via Python-list wrote: On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list wrote: import polars as pl pl.read_json("file.json") This is not going to work unless the computer has a lot more the 60GiB of RAM. As later suggested a streaming par

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2024-09-30, Left Right via Python-list wrote: > Whether and to what degree you can stream JSON depends on JSON > structure. In general, however, JSON cannot be streamed (but commonly > it can be). > > Imagine a pathological case of this shape: 1... <60GB of digits>. This > is still a valid JSON

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list > wrote: > > > import polars as pl > pl.read_json("file.json") > > This is not going to work unless the computer has a lot more the 60GiB of RAM. As later suggested a streaming parser is required. Barry -- https://m

ANN: Python Meeting Düsseldorf - 02.10.2024

/This announcement is in German since it targets a local user group//meeting in Düsseldorf, Germany/ Ankündigung Python Meeting Düsseldorf - Oktober 2024 Ein Treffen von Python Enthusiasten und Interessi

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

Whether and to what degree you can stream JSON depends on JSON structure. In general, however, JSON cannot be streamed (but commonly it can be). Imagine a pathological case of this shape: 1... <60GB of digits>. This is still a valid JSON (it doesn't have any limits on how many digits a number can