Re: [sqlite] json() number value parsing

[Rolf Ade]

Am 04/09/2017 10:34 AM, Olivier Mascia wrote:

Le 9 avr. 2017 à 03:08, Jens Alfke  a écrit :


On Apr 7, 2017, at 5:26 PM, Rolf Ade  wrote:
./sqlite3
SQLite version 3.19.0 2017-04-07 20:20:08
[...]
sqlite> select json(' { "this" : 000.23 } ');
{"this":000.23}

If I read RFC 7159 (http://www.rfc-editor.org/rfc/rfc7159.txt) correct
this should return: "Error: malformed JSON".


In this case I would go with Postel’s Law, paraphrased as “Be
strict in what you write, but lenient in what you read.” I don’t see
a point in disallowing something as trivial as redundant leading
zeroes.


Mr. Hipp has already fixed this:
https://www.sqlite.org/src/info/204e72f0080e8f08

If you think, that Postel's law should applied here, then Olivier
already pointed out rightfully:


If you'd go with Postal's Law, you would make it so:

sqlite> select json(' { "this" : 000.23 } ');  // be lenient in what you 
read
{"this":0.23} // be strict in what you write


I think, you do your users no good on the long run, if you accept not
recommendation compliant input (without explict request to do that by
the user). After all, JSON isn't a very complex standard and the
specification does not let much room (if ever) to argue if a certain
input string is valid or not. And JSON isn't an internet protocol, but
a data interchange format.

That all said I'm far from being religious about this. Even a "won't
fix" or a "works as designed" would have been OK with me.

This even wasn't a case I god bitten by this in the wild. For another
project I'm currently writing my 'own' JSON parser. To do that, I took
a look at the sqlite JSON parser implementation (just because I knew
it's on my hard disk and Mr. Hipps code is high quality). I stumbled
over this just by studying the sqlite json1.c code and wanted to make
sure, this implemenation detail is known and decided deliberately.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Jens Alfke]

> On Apr 10, 2017, at 6:37 AM, Dominique Devienne  wrote:
> 
> I ran into http://json5.org/  which is an interesting 
> "for-humans" extension to JSON.

I love JSON5. It’s an extension of JSON that adds extra syntax from JavaScript, 
like single quotes, trailing commas, and unquoted alphanumeric keys. I find it 
much more readable and much easier to write. A key benefit is that, by using 
single quotes, you can embed JSON5 in a C/C++/Swift/etc. string literal without 
having to escape all those pesky double-quotes:

const char *json  = "{\"foo\": \"bar\"}";
const char *json5 = "{foo: 'bar'}";

FYI, I wrote a converter in C++ that translates JSON5 to JSON. I use it a lot 
in unit tests in my code. It's also useful when parsing configuration files 
that are likely to be human-written or -edited.
https://github.com/couchbaselabs/fleece/blob/master/Fleece/JSON5.hh
https://github.com/couchbaselabs/fleece/blob/master/Fleece/JSON5.cc

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Dominique Devienne]

On Mon, Apr 10, 2017 at 2:27 PM, Richard Hipp  wrote:

> On 4/10/17, Richard Hipp  wrote:
> > SQLite returns true from json_valid() for the following cases which
> > should allegedly be false:
> >
> >   n_multidigit_number_then_00.json
> >   n_string_unescaped_newline.json
> >   n_string_unescaped_tab.json
> >
>
> The second and third are now fixed on trunk.
>
> The first test case consists of a valid JSON with an extra 0x00
> terminator.  As SQLite interprets the 0x00 as an end-of-string marker,
> I do not see this one as an error.


Great! Thanks Richard.

FWIW, looking for whether trailing zeros for the fractional part of reals
was OK,
(seems like they are), I ran into http://json5.org/ which is an interesting
"for-humans" extension to JSON. --DD
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Richard Hipp]

On 4/10/17, Richard Hipp  wrote:
> SQLite returns true from json_valid() for the following cases which
> should allegedly be false:
>
>   n_multidigit_number_then_00.json
>   n_string_unescaped_newline.json
>   n_string_unescaped_tab.json
>

The second and third are now fixed on trunk.

The first test case consists of a valid JSON with an extra 0x00
terminator.  As SQLite interprets the 0x00 as an end-of-string marker,
I do not see this one as an error.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Richard Hipp]

On 4/10/17, Dominique Pellé  wrote:
>
> SQLite json could be added to https://github.com/nst/JSONTestSuite
> (in the parsers/ directory).
>

Thanks for the pointer.  I did not know about that project.

SQLite returns true from json_valid() for the following cases which
should allegedly be false:

  n_multidigit_number_then_00.json
  n_string_unescaped_newline.json
  n_string_unescaped_tab.json

This is the script that I ran to produce the results above:

CREATE TABLE files(name);
.mode csv
.import '| ls *.json' files

-- mal-formed JSON that json_valid shows as correct.
SELECT name FROM files
 WHERE name LIKE 'n_%'
  AND json_valid(readfile(name));

-- well-formed JSON that json_valid shows as incorrect.
SELECT name FROM files
 WHERE name LIKE 'y_%'
  AND NOT json_valid(readfile(name));

-- implementation-defined JSON does not cause crashes
SELECT name FROM files
 WHERE name LIKE 'i_%'
   AND json_valid(readfile(name)) NOT IN (0,1);

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Dominique Pellé]

Dominique Devienne  wrote:

> On Sun, Apr 9, 2017 at 10:34 AM, Olivier Mascia  wrote:
>
>> > Le 9 avr. 2017 à 03:08, Jens Alfke  a écrit :
>> >
>> >> On Apr 7, 2017, at 5:26 PM, Rolf Ade  wrote:
>> >> ./sqlite3
>> >> SQLite version 3.19.0 2017-04-07 20:20:08
>> >> [...]
>> >> sqlite> select json(' { "this" : 000.23 } ');
>> >> {"this":000.23}
>> >> If I read RFC 7159 (http://www.rfc-editor.org/rfc/rfc7159.txt <
>> http://www.rfc-editor.org/rfc/rfc7159.txt>) correct
>> >> this should return: "Error: malformed JSON".
>> >
>> > In this case I would go with Postel’s Law, paraphrased as “Be strict in
>> what you write, but lenient in what you read.” I don’t see a point in
>> disallowing something as trivial as redundant leading zeroes.
>>
>> If you'd go with Postal's Law, you would make it so:
>>
>> sqlite> select json(' { "this" : 000.23 } ');   // be lenient in what you
>> read
>> {"this":0.23}   // be strict in what you
>> write
>>
>
> I disagree. There's a spec. it should be followed, by default.
>
> A separate, explicitly enabled "lenient mode" could be added,
> for leading zeros, NaNs, C-comments, binary strings, etc...
> but it should not be the default.
>
> Otherwise you end up forever having to support non-conformity,
> possibly simply because of an oversights.
>
> Plus json_valid() is then lying about checking well-formed'ness. --DD
>
> sqlite> select json_valid('');
> 0
> sqlite> select json_valid('00.1');
> 1
> sqlite> select json_valid('0.1');
> 1
> sqlite> select json_valid('[]');
> 1
> sqlite> select json_valid('[00]');
> 1
> sqlite> select json_valid('[00.00]');
> 1


SQLite json could be added to https://github.com/nst/JSONTestSuite
(in the parsers/ directory).

This JSON test suite checks many kinds of invalid and valid JSON
inputs to make sure that they are rejected or accepted as expected
by RFC 7159. It surprisingly finds issues in many JSON parsers.

Dominique
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Dominique Devienne]

On Sun, Apr 9, 2017 at 10:34 AM, Olivier Mascia  wrote:

> > Le 9 avr. 2017 à 03:08, Jens Alfke  a écrit :
> >
> >> On Apr 7, 2017, at 5:26 PM, Rolf Ade  wrote:
> >> ./sqlite3
> >> SQLite version 3.19.0 2017-04-07 20:20:08
> >> [...]
> >> sqlite> select json(' { "this" : 000.23 } ');
> >> {"this":000.23}
> >> If I read RFC 7159 (http://www.rfc-editor.org/rfc/rfc7159.txt <
> http://www.rfc-editor.org/rfc/rfc7159.txt>) correct
> >> this should return: "Error: malformed JSON".
> >
> > In this case I would go with Postel’s Law, paraphrased as “Be strict in
> what you write, but lenient in what you read.” I don’t see a point in
> disallowing something as trivial as redundant leading zeroes.
>
> If you'd go with Postal's Law, you would make it so:
>
> sqlite> select json(' { "this" : 000.23 } ');   // be lenient in what you
> read
> {"this":0.23}   // be strict in what you
> write
>

I disagree. There's a spec. it should be followed, by default.

A separate, explicitly enabled "lenient mode" could be added,
for leading zeros, NaNs, C-comments, binary strings, etc...
but it should not be the default.

Otherwise you end up forever having to support non-conformity,
possibly simply because of an oversights.

Plus json_valid() is then lying about checking well-formed'ness. --DD

sqlite> select json_valid('');
0
sqlite> select json_valid('00.1');
1
sqlite> select json_valid('0.1');
1
sqlite> select json_valid('[]');
1
sqlite> select json_valid('[00]');
1
sqlite> select json_valid('[00.00]');
1
sqlite>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Olivier Mascia]

> Le 9 avr. 2017 à 03:08, Jens Alfke  a écrit :
> 
>> On Apr 7, 2017, at 5:26 PM, Rolf Ade  wrote:
>> ./sqlite3 
>> SQLite version 3.19.0 2017-04-07 20:20:08
>> [...]
>> sqlite> select json(' { "this" : 000.23 } ');
>> {"this":000.23}
>> If I read RFC 7159 (http://www.rfc-editor.org/rfc/rfc7159.txt 
>> ) correct
>> this should return: "Error: malformed JSON".
> 
> In this case I would go with Postel’s Law, paraphrased as “Be strict in what 
> you write, but lenient in what you read.” I don’t see a point in disallowing 
> something as trivial as redundant leading zeroes.

If you'd go with Postal's Law, you would make it so:

sqlite> select json(' { "this" : 000.23 } ');   // be lenient in what you read
{"this":0.23}   // be strict in what you write

:)

-- 
Best Regards, Meilleures salutations, Met vriendelijke groeten,
Olivier Mascia, http://integral.software


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] json() number value parsing

[Jens Alfke]

> On Apr 7, 2017, at 5:26 PM, Rolf Ade  wrote:
> 
> If I read RFC 7159 (http://www.rfc-editor.org/rfc/rfc7159.txt 
> ) correct
> this should return: "Error: malformed JSON".

In this case I would go with Postel’s Law, paraphrased as “Be strict in what 
you write, but lenient in what you read.” I don’t see a point in disallowing 
something as trivial as redundant leading zeroes.

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users