On Fri, Aug 9, 2019 at 3:32 PM Jonathan Hall <fli...@flimzy.com> wrote: > > I think you're right that most people are frustrated by the encoder, but as I > mentioned in another message just a bit ago, the same fundamental problem > exists with decoder, and for certain workloads, I believe it should be solved. > > Having said that, I think tackling the writer first definitely makes the most > sense. > > With the decoder, at least it's possible, although terribly cumbersome, to > cobble together a solution with the tokenizer interface of json.Decoder. And > truth be told, my proposal wouldn't really elminate the use of the tokenizer > interface--but it would make it possible to use it on a burried type (i.e. > within a UnmarshalJSONStream() method), to achieve the benefit of stream > reading.
It may not be terribly cumbersome, and may not need the tokenizer interface. I have a json streaming library (streaming in the sense that multiple json docs one after the other, not one large doc) based on the std encoder/decoder, and something like that can be developed to deal with large json docs. https://github.com/bserdar/jsonstream > > > > On Friday, August 9, 2019 at 10:17:41 PM UTC+2, burak serdar wrote: >> >> On Fri, Aug 9, 2019 at 2:10 PM Robert Engels <ren...@ix.netcom.com> wrote: >> > >> > I'm sorry, maybe I didn't understand your original concern. There is an >> > example of doing stream based processing in the json package (using >> > Decoder). >> > >> > How is this not sufficient? >> > >> > The only problem I see with it is that it doesn't allow error >> > correction/continuation, but in the modern world that seems rather rare >> > (or very difficult to do well). >> >> I was thinking similarly and after reading those github issues, it >> looks like the main problem is with Encoder, and not with Decoder. >> Encoder's problem can be solved by providing an unbuffered output >> option that directly writes to the io.Writer. >> >> I like the idea of stream-friendly marshaler/unmarshaler interfaces. >> >> > >> > -----Original Message----- >> > From: Jonathan Hall >> > Sent: Aug 9, 2019 11:00 AM >> > To: golang-nuts >> > Subject: Re: [go-nuts] RFC for opt-in streaming support in encoding/json >> > package >> > >> > An interesting observation. >> > >> > Although in a sense, we already have the decoding half of that low-level >> > streaming API exposed by way of the `json.Decoder` type. >> > >> > Would you be suggesting that a separate, stream-based API makes sense even >> > within the stdlib? >> > >> > I'm not really sure what that separate API would look like, or do >> > differently than my proposal (I'm open to new ideas, though). >> > >> > Given that the "Go way" of handling streams is with io.Reader/io.Writer >> > (as opposed to events, for example), and the internal implementation of >> > `json/encoding` is already so close to that, I wonder if the APIs would >> > end up looking very much the same, anyway. >> > >> > Jonathan >> > >> > >> > On Friday, August 9, 2019 at 5:53:41 PM UTC+2, Robert Engels wrote: >> >> >> >> In other environments (e.g. Java), the streaming processors are distinct >> >> from the instance oriented - usually for good reason as the APIs are very >> >> different - the former is usually event based, and the latter returns >> >> realized instances as a whole or an error. The streaming processors can >> >> often skip ill-formed entities, and/or have them manipulated during >> >> decoding. >> >> >> >> -----Original Message----- >> >> From: Jonathan Hall >> >> Sent: Aug 9, 2019 10:38 AM >> >> To: golang-nuts >> >> Subject: Re: [go-nuts] RFC for opt-in streaming support in encoding/json >> >> package >> >> >> >> Thanks for the reply. My responses inline below. >> >> >> >> On Friday, August 9, 2019 at 5:14:52 PM UTC+2, burak serdar wrote: >> >>> >> >>> On Fri, Aug 9, 2019 at 8:53 AM Jonathan Hall <fli...@flimzy.com> wrote: >> >>> > >> >>> > Can you say more? Better in which way? >> >>> >> >>> Better in the way that it wouldn't change existing code. >> >> >> >> >> >> That doesn't seem like a benefit, in its own right >> >> >> >> I understand the desire not to just change code for its own sake, or add >> >> extra features nobody needs. But people have been asking for these types >> >> of features for several years. This doesn't seem like a frivolous code >> >> change to me. >> >> >> >>> >> >>> Also, I think >> >>> the use cases for existing and proposed json encoders/decoders are >> >>> different enough to justify a separate implementation. >> >> >> >> >> >> I don't think I agree with this. >> >> >> >> The proposal deals with a subset of current use cases, but not, strictly >> >> speaking, a _different set_ of use cases. And the number of commentators >> >> on the issues linked above, I think lends weight to the idea that the use >> >> cases this proposal addresses are not insignificant, or fundamentally >> >> "different". >> >> >> >> If I were to fork the standard `encoding/json` library, and add my >> >> proposed functionality, the code would still be 95% the same. Standard >> >> reasons for sharing code apply, as far as I can tell. >> >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> >> "golang-nuts" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an >> >> email to golan...@googlegroups.com. >> >> To view this discussion on the web visit >> >> https://groups.google.com/d/msgid/golang-nuts/df688733-a2e9-4bc8-aa7b-09267827007a%40googlegroups.com. >> >> >> >> >> >> >> >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "golang-nuts" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> > email to golan...@googlegroups.com. >> > To view this discussion on the web visit >> > https://groups.google.com/d/msgid/golang-nuts/718e2b9e-39e2-44f2-9308-8c94e31afbff%40googlegroups.com. >> > >> > >> > >> > >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "golang-nuts" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> > email to golan...@googlegroups.com. >> > To view this discussion on the web visit >> > https://groups.google.com/d/msgid/golang-nuts/924142684.7377.1565381399326%40wamui-agami.atl.sa.earthlink.net. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/4d901ba9-5724-45cd-b7f7-24fccd9ef5db%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMV2Rqp%3D8Ughztq4cn714J5prsoAx57GOevBJOqUqV4sMyMkGg%40mail.gmail.com.
