[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2020-02-16 Thread Xqt
Xqt added a comment.


  > PS: I add @Xqt because I guess you can help here
  
  Like matej_suchanek suggests please submit Patches to gerrit for reviewing. 
You may use Gerrit patch uploader 
 if you don't want your 
own account there.
  
  Anyway `'amount': '{:f}'.format(Decimal(self._fromdecimal(self.amount)))` 
looks a bit odd to me.` self._fromdecimal` seems to be a method converting 
`decimal.Decimal` to something else and `Decimal()` wents this 
back. The result sounds like `self._todecimal()` method. Anyway I haven't 
investigated deeper into this matter currently.

TASK DETAIL
  https://phabricator.wikimedia.org/T119226

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: DD063520, Xqt
Cc: Addshore, matej_suchanek, Xqt, Liuxinyu970226, DD063520, thiemowmde, 
Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, 
Zkhalido, darthmon_wmde, Viztor, Nandana, Wenyi, Lahi, Gq86, GoranSMilovanovic, 
QZanden, Tbscho, MayS, LawExplorer, Mdupont, JJMC89, Dvorapa, _jensen, 
rosalieper, Altostratus, Avicennasis, Scott_WUaS, mys_721tx, Wikidata-bugs, 
aude, jayvdb, Ricordisamoa, Masti, Alchimista, Mbch331, Rxy
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2020-02-15 Thread matej_suchanek
matej_suchanek added a comment.


  My T204331  is probably a 
duplicate.
  
  > Should I submit this?
  
  Sure, reviews are better in Gerrit.
  
  There are two problems:
  
  - Pywikibot can loose precision when manipulating data
  - Pywikibot submits values that are not valid according to Wikibase (eg. 
`"+1.9e-09"`)

TASK DETAIL
  https://phabricator.wikimedia.org/T119226

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: DD063520, matej_suchanek
Cc: matej_suchanek, Xqt, Liuxinyu970226, DD063520, thiemowmde, Tobias1984, 
Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Zkhalido, 
darthmon_wmde, Viztor, Nandana, Wenyi, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Tbscho, MayS, LawExplorer, Mdupont, JJMC89, Dvorapa, _jensen, rosalieper, 
Altostratus, Avicennasis, Scott_WUaS, mys_721tx, Wikidata-bugs, aude, jayvdb, 
Ricordisamoa, Masti, Alchimista, Mbch331, Rxy
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2020-02-14 Thread ArthurPSmith
ArthurPSmith added a comment.


  Sorry I never got around to looking at this further. @DD063520 do you 
understand the above comment from @thiemowmde about using the wbparsevalue api 
rather than python internals?

TASK DETAIL
  https://phabricator.wikimedia.org/T119226

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArthurPSmith
Cc: Xqt, Liuxinyu970226, DD063520, thiemowmde, Tobias1984, Aklapper, 
StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Zkhalido, darthmon_wmde, 
Viztor, Nandana, Wenyi, Lahi, Gq86, GoranSMilovanovic, QZanden, Tbscho, MayS, 
LawExplorer, Mdupont, JJMC89, Dvorapa, _jensen, rosalieper, Altostratus, 
Avicennasis, Scott_WUaS, mys_721tx, Wikidata-bugs, aude, jayvdb, Ricordisamoa, 
Masti, Alchimista, Mbch331, Rxy
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2020-02-14 Thread DD063520
DD063520 added a comment.


  Ok,
  
  I think I found a patch. We can changes this 
https://github.com/wikimedia/pywikibot/blob/2dfe67426c22c3c11cf9be0eabcf538f8848bd48/pywikibot/__init__.py#L847:
  
def toWikibase(self):
"""
Convert the data to a JSON object for the Wikibase API.
@return: Wikibase JSON
@rtype: dict
"""
json = {'amount': self._fromdecimal(self.amount),
'upperBound': self._fromdecimal(self.upperBound),
'lowerBound': self._fromdecimal(self.lowerBound),
'unit': self.unit
}
return json
  
  to:
  
def toWikibase(self):
"""
Convert the data to a JSON object for the Wikibase API.

@return: Wikibase JSON
@rtype: dict
"""
json = {'amount': 
'{:f}'.format(Decimal(self._fromdecimal(self.amount))),
'upperBound': 
{:f}'.format(Decimal(self._fromdecimal(self.upperBound)),
'lowerBound': 
{:f}'.format(Decimal(self._fromdecimal(self.lowerBound)),
'unit': self.unit
}
return json
  
  Should I submit this? Or do you see any problems?
  
  Salut
  D063520

TASK DETAIL
  https://phabricator.wikimedia.org/T119226

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: DD063520
Cc: Liuxinyu970226, DD063520, thiemowmde, Tobias1984, Aklapper, StudiesWorld, 
pywikibot-bugs-list, ArthurPSmith, Zkhalido, darthmon_wmde, Viztor, Nandana, 
Wenyi, Lahi, Gq86, GoranSMilovanovic, QZanden, Tbscho, MayS, LawExplorer, 
Mdupont, JJMC89, Dvorapa, _jensen, rosalieper, Altostratus, Avicennasis, 
Scott_WUaS, mys_721tx, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Masti, 
Alchimista, Mbch331, Rxy
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-07-05 Thread ArthurPSmith
ArthurPSmith added a comment.
Ok, that echoes something Tobias has said also about using strings and avoiding IEEE fp. I'm going to look at getting T112140 working first and then see if I can bring that implementation to bear on this.TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmithCc: thiemowmde, Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-07-05 Thread thiemowmde
thiemowmde added a comment.
Pywikibot should not assume all QuantityValues can be casted to IEEE numbers. For example, a QuantityValue can be "100.054321". Depending on the data types you have in your programming language (if it's single, double or something else) converting this to a number and back to a string will result in something like "100.06" or worse:

100.054321.toFixed( 40 )
// Output: "100.0582076609134674072265625000"

That should be avoided, obviously. Never cast the elements from a QuantityValue to IEEE numbers when not necessary for actual calculations.

When dealing with user inputs you can (and should) either use wbparsevalue, or come up with your own parser that converts this to a string allowed in a QuantityValue. Again, make sure you are converting "100054321e-14" to "100.054321" without loosing precision.TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: thiemowmdeCc: thiemowmde, Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-07-04 Thread thiemowmde
thiemowmde added a comment.
This limitation is not on input values. Use the wbparsevalue API, which is what the wikidata.org UI does and what https://www.wikidata.org/wiki/Help:Statements#Quantitative_values describes, and you can have all kinds of inputs.

The resulting QuantityValue data structure must be in the format the Wikibase code base specifies in https://github.com/DataValues/Number/blob/master/src/DataValues/DecimalValue.php#L43 (a QuantityValue is constructed of 3 DecimalValues and the unit). This format does not use the numbers from the JSON spec, but strings, to avoid all kinds of rounding issues and data loss that happens when you enter the IEEE world. Pywikibot should do the same and avoid doing math on IEEE floating point numbers.

So yes, I'm afraid this is an issue in Pywikibot not following the Wikibase specifications.TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: thiemowmdeCc: thiemowmde, Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-07-02 Thread ArthurPSmith
ArthurPSmith added a comment.
That restriction is NOT in the JSON spec: http://tools.ietf.org/html/rfc7159.html#section-6 - also the leading plus is not required by JSON. Is there some other reason for the limitation in the wikidata code? DataValues is a wikidata-specific PHP library right? I can't think of any good reason to keep this limitation on input values.TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmithCc: thiemowmde, Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-07-01 Thread ArthurPSmith
ArthurPSmith added a comment.
Hmm. So is it a pywikibot problem or a wikibase API problem? Is pywikibot sending in JSON format?TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmithCc: thiemowmde, Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-07-01 Thread ArthurPSmith
ArthurPSmith added a comment.
As far as testing goes, I have (in my own copy) added the following to the pywikibot tests/wikibase_edit_tests.py file (within the class TestWikibaseMakeClaim):

def _check_quantity_claim(self, value, uncertainty):
"""Helper function to add and check quantity claims"""
testsite = self.get_repo()
item = self._clean_item(testsite, 'P64')

# set new claim
claim = pywikibot.page.Claim(testsite, 'P64', datatype='quantity')
target = pywikibot.WbQuantity(value, error=uncertainty)
claim.setTarget(target)
item.addClaim(claim)
item.get(force=True)
claim = item.claims['P64'][0]
self.assertEqual(claim.getTarget(), target)


def test_medium_quantity_edit(self):
"""Attempt to add medium-size quantity claim."""
self._check_quantity_claim(1.5, 0.1)


def test_small_quantity_edit(self):
"""Attempt to add very small quantity claim."""
self._check_quantity_claim(1.0e-7, 2.0e-8)


def test_large_quantity_edit(self):
"""Attempt to add large quantity claim."""
self._check_quantity_claim(1.935e35, 1e32)


def test_negative_quantity_edit(self):
"""Attempt to add negative quantity claims."""
self._check_quantity_claim(-1.5, 0.1)

When these tests are run via
 python pwb.py tests/wikibase_edit_tests.py -v
both test_large_quantity_edit() and test_small_quantity_edit fail, with messages:

Attempt to add large quantity claim. ... WARNING: API error invalid-snak: Invalid snack (Value must match the pattern for decimal values.)
 Attempt to add very small quantity claim. ... WARNING: API error invalid-snak: Invalid snak. (Value must match the pattern for decimal values.)TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmithCc: Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-06-28 Thread Tobias1984
Tobias1984 added a comment.
Here is some information of the range of values the API accepts: https://www.wikidata.org/wiki/Help:Statements#Quantitative_values

We probably need more testing so things like 1e-123 arrives as:

0.001

and 1.0e-123 arrives as:

0.0010

Those are some rather important edge cases. The API works more like a metrologist or engineering software where trainling zeros are preserved. But I am also not an expert in this topic.TASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Tobias1984Cc: Tobias1984, Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T119226: Very small (or very large) quantity values (represented in scientific notation) result in error in add/update via pywikibot/wikidata API

2016-06-27 Thread ArthurPSmith
ArthurPSmith added a comment.
Please note this is still an issue with the latest pywikibot code and current wikidata release - as of June 23, 2016. The following is the fix I have in the pywikibot core pywikibot/__init__.py file:

instead of

format(value, "+g")

we need:

if math.fabs(value) < 0.001:
num_str = float_fix.convert_sc_to_str(float(value))
if value >= 0:
num_str = '+{0}'.format(num_str)
else:
num_str = format(value, "+g")
return num_strTASK DETAILhttps://phabricator.wikimedia.org/T119226EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArthurPSmithCc: Aklapper, StudiesWorld, pywikibot-bugs-list, ArthurPSmith, Mdupont, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs