[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[daniel]

daniel added a comment.
The above patch implements this for quantities. Datetime is already always normalized to ISO (Gregorian).

Shall we close this, or wait until the subtasks are closed?TASK DETAILhttps://phabricator.wikimedia.org/T117031EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: mkroetzsch, gerritbot, Jc3s5h, aude, daniel, Aklapper, Smalyshev, mschwarzer, MelodyKramer, Avner, Lewizho99, Maathavan, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[gerritbot]

gerritbot added a comment.
Change 296962 merged by jenkins-bot:
Unit conversion support for full statements

https://gerrit.wikimedia.org/r/296962TASK DETAILhttps://phabricator.wikimedia.org/T117031EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: mkroetzsch, gerritbot, Jc3s5h, aude, daniel, Aklapper, Smalyshev, mschwarzer, MelodyKramer, Avner, Lewizho99, Maathavan, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[gerritbot]

gerritbot added a comment.
Change 296962 had a related patch set uploaded (by Smalyshev):
Unit conversion support for full statements

https://gerrit.wikimedia.org/r/296962TASK DETAILhttps://phabricator.wikimedia.org/T117031EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Jc3s5h, aude, daniel, Aklapper, Smalyshev, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.


  In https://phabricator.wikimedia.org/T117031#2050075, @Smalyshev wrote:
  
  > We are currently using XSD 1.1 format in both cases, and in both cases the 
date is (proleptic) Gregorian.
  
  
  I think Smalyshev didn't use the word "currently" clearly in this sentence. 
Maybe he means "the proposal as currently envisioned". But he can't mean the 
current version of WikiData, because the current version has no Julian to 
Gregorian conversion software, and quite a few dates in the database are 
currently stored in the Julian calendar.
  
  A different question: what programming language would the future Julian to 
Gregorian routine be written in. Maybe I can find something, or translate 
something.

TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, debt, Gehel, Izno, jkroll, 
Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Smalyshev]

Smalyshev added a comment.


  We are currently using XSD 1.1 format in both cases, and in both cases the 
date is (proleptic) Gregorian.

TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, debt, Gehel, Izno, jkroll, 
Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.


  For the [[mw:Wikibase/Indexing/RDF Dump Format]] I note that time already has 
a simple value and a full value. The simple value is already specified to be in 
the XSD 1.1 format if the full value can be converted to Gregorian. Would this 
simple value be sufficient?

TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, debt, Gehel, Izno, jkroll, 
Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.


  Let's discuss which version of the schema to use a a starting point. You 
suggesthttps://www.w3.org/TR/2001/REC-xmlschema-2-20010502/#dateTime .  But the 
linked version of the schema does not allow year zero, the linked version of 
the schema purports to support leap seconds, and the linked version of the 
schema has a different representation for time zones than a more recent version.
  
  A later version, https://www.w3.org/TR/xmlschema11-2/#d-t-values defines  
as 1 BC, does not support leap seconds (but nevertheless purports to use UTC), 
and represents time zones as an offset measured in minutes.
  
  I don't think we can proceed unless we decide which version of the schema to 
use as a starting point.

TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, debt, Gehel, Izno, jkroll, 
Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Smalyshev]

Smalyshev added a comment.


  @Jc3s5h Basic ISO 8601 is not perfect for our goals, since we need to 
represent dates that non-extended ISO format can not. Thus, we are extending 
it, in accordance with W3C guidelines 
:
  
  > To accommodate year values greater than  additional digits can be added 
to the left of this representation.
  
  Note that this standard does not require fixed digits.
  
  Now, we know that some data stores are unable to accommodate extended date 
ranges, especially one that are implemented in Java and take shortcut of 
representing xsd:dateTime as Java calendar/datetime values (which can not 
accommodate the full range of dates Wikidata has). These solutions will have to 
be either modified or be unable to process certain data. There's no way around 
it - if they can not read our dates directly, they'd have to do something extra 
to read them. So far we've chosen the way that would be as close to the W3C 
standard as possible and not require to big a departure from ISO.

TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, debt, Gehel, Izno, jkroll, 
Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.


  An additional issue with borrowing the format from ISO 8601 (without using 
the definitions from ISO 8601) is that that standard requires a fixed number of 
digits for the years. So if the age of the universe is to be representable, 
today would have to be written 0002016-02-26. Doing otherwise would force 
data consumers to check their ISO 8601 parsing software to see if it can handle 
our violations.

TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, debt, Gehel, Izno, jkroll, 
Wikidata-bugs, Jdouglas, Deskana, Manybubbles, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.

xsd:dateTime reiterates the necessity of using the Gregorian calendar. The 
Gregorian calendar cannot possibly exist before the Earth, and Wikidata has a 
need to express dates that occurred before the Earth was formed. So either the 
TimeValue data type must be redefined to limit values to those that fall within 
the range of validity of the Gregorian calendar (the exact limits are not 
obvious) and a different data type created for dates in the far distant past or 
future, or we should stop naming or implying any external standard and 
acknowledge its something we made up ourselves, unrelated to ISO or the World 
Wide Web Consortium. It is not honest to knowingly ascribe to others statements 
that we know they did not make, such as the possibility of expressing the age 
of the universe in a notation that conforms to an ISO or W3C standard.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[daniel]

daniel added a comment.

@Smalyshev I only see one potential use case for precise dates in the distant 
past: astronomical events. But it seems unlikely we'll ever have items about 
individual eclipses or conjugations of planets in the far past.

@Jc3s5h The ISO date would be used as the "normalized" value, intended for 
indexing/searching. Even if it does not quite work for distant dates, it should 
do the trick in almost all use cases we currently have. The only alternative I 
see would be to use an integer to represent the time, e.g. a 64 bit unix epoch, 
that would be good for a range of half a trillion years. But it would make it 
hard to compare these values with dates from other sources.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.

I would add that the question of whether, in the future, the transition from 
one day to the next should be determined by the rotation of the earth,  or by 
atomic clocks which disregard the earth's rotation, will be the subject of 
(according to Rachel Courtland 
)
 a "fierce debate" at the World Radiocommunication Conference in November. 
Treating an international organization's standard as atomic-based when it is 
really rotation-based becomes more serious during a time when a "fierce debate" 
is raging.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Aklapper]

Aklapper added a comment.

@Jc3s5h: Please read the Phabricator etiquette 
. Thanks 
for your understanding and your help to keep this discussion technical instead 
of personal.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Aklapper
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.

If our documentation or source code comments contain falsehoods then the 
Wikidata team is a pack of liars. ISO requires the Gregorian calendar. If you 
use an ISO-like notation to represent something other than the Gregorian 
calendar we are liars and Wikidata would deserve to be defunded. **STOP WRITING 
ISO UNLESS YOU REALLY MEAN IT AND INTEND TO ABIDE BY EVERY SINGLE RULE IN THE 
SPEC.**


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[daniel]

daniel added a comment.

@Jc3s5h Please calm down. Documentation is often outdated or inconsistent or 
unclear. We can work to improve it, but it will never be perfect. But that is 
not the issue here, since I did mean ISO - well, technically, I mean 
xsd:datetime. I did not mean "iso-ish string maybe using some other calendar".

We would use an xsd date (ISO) for the *normalized* value, for indexing and 
comparison. Internally, the value would be stored in a calendar specific way - 
for gregorian and julian, in something that uses a similar syntax as ISO. That 
"original" value would also be visible in JSON output, along with the 
normalized form (currently, the normalized form is not there yet). In RDF, it's 
still unclear whether we'll include the original value at all, or only use the 
normalized form.

The same kind of normalization would be applied to make sure that all values of 
the "length" property will be comparable by converting them all to meters 
(while still making the original value available in rdf). You would have two 
values associated with the statement, one "original" (in miles or whatever), 
one "normalized" (in meter).


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Smalyshev]

Smalyshev added a comment.

It means ISO 8601 compliant value.

> The difference between these two approaches grows to a full day about the 
> year 3400 BC. Thus ISO 8601 is not fit for use in prehistoric times.


Do we have any dates around or before year 3400 BC that are know with day or 
less accuracy? In fact, in years 3000 BC and below, there are only two dates 
with precision of 9 (year) and one with precision of decade, and none with more 
precise dates. In fact, I do not think it is possible to have dates with higher 
precision for such years in either of the calendars we are currently converting 
(Gregorian and Julian), since I can not see where such precise dates would be 
coming from.

So I think we'll be fine.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.

The phrase "Gregorian ISO value" requires further consideration. For example, 
does it mean an ISO 8601 compliant value, or the strings currently in use which 
resemble ISO 8601 but permit a number of violations, such as 
"2015-**00**-**00**T00:00:00Z"?

Also ISO 8601 requires the Gregorian calendar, which is a solar calendar based 
on counting actual sunrises and sunsets. For events in the distant past, most 
astronomy and other science is based on constant-length seconds such as 
produced by an atomic clock (see the Wikipedia article "Delta T". The 
difference between these two approaches grows to a full day about the year 3400 
BC. Thus ISO 8601 is not fit for use in prehistoric times.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117031: Represent normalized unit values in full values RDF

[Jc3s5h]

Jc3s5h added a comment.

We have "universe" (Q1) which displays the start date as "
13798 million years BCE Gregorian" (Gregorian is a superscript). But the 
furthest back one could possibly extrapolate the proleptic Gregorian calendar 
was the first time the Earth rotated on its axis, which was much later (around 
4,540 million years ago, measured in constant length seconds). So the date 
stated for the start of the universe in wikidata is false because the calendar 
was undefined and beyond any plausible extrapolation procedure at the time of 
the event.


TASK DETAIL
  https://phabricator.wikimedia.org/T117031

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, aude, daniel, Aklapper, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, 
Deskana, Manybubbles



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs