[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment. Thanks for confirming!TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Addshore added a comment. All looks good > addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="SHOW TABLES;" stdClass Object ( [Tables_in_cognate_wiktionary] => cognate_pages ) stdClass Object ( [Tables_in_cognate_wiktionary] => cognate_sites ) stdClass Object ( [Tables_in_cognate_wiktionary] => cognate_titles ) addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="DESCRIBE cognate_pages;" stdClass Object ( [Field] => cgpa_site [Type] => bigint(20) [Null] => NO [Key] => PRI [Default] => [Extra] => ) stdClass Object ( [Field] => cgpa_namespace [Type] => int(11) [Null] => NO [Key] => PRI [Default] => [Extra] => ) stdClass Object ( [Field] => cgpa_title [Type] => bigint(20) [Null] => NO [Key] => PRI [Default] => [Extra] => ) addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="DESCRIBE cognate_sites;" stdClass Object ( [Field] => cgsi_key [Type] => bigint(20) [Null] => NO [Key] => PRI [Default] => [Extra] => ) stdClass Object ( [Field] => cgsi_dbname [Type] => varbinary(32) [Null] => NO [Key] => [Default] => [Extra] => ) stdClass Object ( [Field] => cgsi_interwiki [Type] => varbinary(32) [Null] => NO [Key] => [Default] => [Extra] => ) addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="DESCRIBE cognate_titles;" stdClass Object ( [Field] => cgti_raw [Type] => varbinary(255) [Null] => NO [Key] => [Default] => [Extra] => ) stdClass Object ( [Field] => cgti_raw_key [Type] => bigint(20) [Null] => NO [Key] => PRI [Default] => [Extra] => ) stdClass Object ( [Field] => cgti_normalized_key [Type] => bigint(20) [Null] => NO [Key] => MUL [Default] => [Extra] => ) addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="SELECT * FROM cognate_titles;" Query OK, 0 row(s) affected addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="SELECT * FROM cognate_pages;" Query OK, 0 row(s) affected addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=extension1 --replicadb=any --wikidb=cognate_wiktionary --query="SELECT * FROM cognate_sites;" Query OK, 0 row(s) affectedTASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, AddshoreCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment. In T162252#3174575, @jcrespo wrote: I would ask Addshore to confirm by running SELECT on the empty tables from terbium/tin, etc, using mediawiki scripts. @Addshore can you confirm? ThanksTASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
jcrespo added a comment. I would ask Addshore to confirm by running SELECT on the empty tables from terbium/tin, etc, using mediawiki scripts.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment. In T162252#3167753, @Addshore wrote: So although replication to labs is not a blocker, It is something that we would like to happen for these tables. We have been talking about it and we are not really comfortable replicating x1 to labs, as if something goes wrong we would be leaking private data. We have some filtering in place, but it is not a silver bullet and if they don't work as expected, the impact can be quite big, as I said, private information being leaked. So for now we'd prefer to leave it out from labs. There are some discussion about some services in the future might allow x1 split and replication to labs, if those finally go on, we could include these tables there.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Addshore added a comment. In T162252#3167440, @Marostegui wrote: I will discuss with Jaime what to do with dbstore servers and labs and if we really need to do it there, as he mentioned, x1 is not being replicated to labs as well. If not, I will remove the database and the tables from dbstore1001 and dbstore1002. So although replication to labs is not a blocker, It is something that we would like to happen for these tables.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment.
Hello,
So I have created the db and the 3 tables in the following production hosts (without replication):
root@neodymium:/home/marostegui/git/software/dbtools# cat x1.hosts
dbstore2001.codfw.wmnet 3306
tempdb2001.codfw.wmnet 3306
db2033.codfw.wmnet 3306
dbstore1001.eqiad.wmnet 3306
dbstore1002.eqiad.wmnet 3306
db1029.eqiad.wmnet 3306
db1031.eqiad.wmnet 3306
The dbstore servers, for the x1 replication thread will need to be modified, if we really think it is necessary to replicate this DB and its data as they have:
Replicate_Wild_Do_Table: flowdb.%,wikishared.%,heartbeat.%
root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat x1.hosts | awk -F " " '{print $1}'`; do echo $i; mysql --skip-ssl -h$i cognate_wiktionary -e "show tables;";done
dbstore2001.codfw.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
tempdb2001.codfw.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
db2033.codfw.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
dbstore1001.eqiad.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
dbstore1002.eqiad.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
db1029.eqiad.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
db1031.eqiad.wmnet
+--+
| Tables_in_cognate_wiktionary |
+--+
| cognate_pages|
| cognate_sites|
| cognate_titles |
+--+
I will discuss with Jaime what to do with dbstore servers and labs and if we really need to do it there, as he mentioned, x1 is not being replicated to labs as well.
If not, I will remove the database and the tables from dbstore1001 and dbstore1002.
Will update the ticket.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Addshore added a comment. In T162252#3164085, @jcrespo wrote: @Addshore I hope labs access is not a blocker for this, that can be done at a later date. Nope, not a blocker for this.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
jcrespo added a comment. production hosts Was thinking on dbstore (backup) hosts, which were problematic (remember you where the ones to set up those last time) + private table filtering. x1 has been traditionally not replicated to labs, this can be challenging (I would start by not replicationg at all, but setting the filters accordingly). Also reviewing the grants for the wikiuser and wikiadmin users.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
jcrespo added a comment. @Addshore I hope labs access is not a blocker for this, that can be done at a later date.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment. Yup, we can leave labs aside for the "first stage" and then study those without blocking this task I would say.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment. A quick look on the x1 production hosts (thanks Jaime for the clarification) shows no replication filters so for those, we should be fine to create all on the master and let it replicate. As Jaime mentioned, for labs we might need to manually intervene there, if creating the database+tables does not replicate well to labs because of all the filtering.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Liuxinyu970226, jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
jcrespo added a comment. This sentence actually confused me x1 == extension1 :-)TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
jcrespo added a comment. To clarify- it may be blocked on us right now to create the database and because labs filtering is not well managed, but the general idea stays for normal table creations.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Marostegui added a comment. In T162252#3163937, @jcrespo wrote: So that we are not a blocker- creation of tables in production, specially if we have already given the OK to the plans, is not considered a schema change, so anyone with production rights can do it- you just need to mark it on the deployments calendar. With this we (DBAs) are not saying we do not want to do it, but you will have to wait for us to have a free slot, while many other people with development production rights may be able to do it better and earlier. @Marostegui This is going to x1 as it is a shared thing between wikis- we may need to check replication filters. Ah @jcrespo thanks for the clarification! This sentence actually confused me The extension is already running on beta, a database named cognate_wiktionary has been created on the extension1 cluster. and forgot to ask you past Thursday in our meeting, I had it noted down for Monday actually :)TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
jcrespo added a comment. So that we are not a blocker- creation of tables in production, specially if we have areadly given the OK to the plans, is not considered a schema change, so anyone with production rights can do it- you just need to mark it on the deployments calendar. Wis this we (DBAs) are not saying we do not want to do it, but you will have to wait for us to have a free slot, while many other people with develoment production rights may be able to do it better and earlier. @Marostegui This is going to x1 as it is a shared thing between wikis- we may need to check replication filters.TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T162252: Create SQL database and Tables for Cognate extension to be used on Wiktionaries
Addshore added a comment. In T162252#3157135, @Marostegui wrote: As per T148988#2742029 I assume this is all public info and no further filtering is required when replicating to labs. Indeed, everything in these tables is public information and can be replicated to labs, no filtering required. Most of the *wiktionary databases currently live in s3 so I would assume we need to create the new database and the 3 tables there too? Well, the database can live anywhere / on any 'cluster' configured on mediawiki and the extension will work just fine. If you feel that s3 is the best place for it then we can put it there!TASK DETAILhttps://phabricator.wikimedia.org/T162252EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Marostegui, Lydia_Pintscher, Aklapper, Lea_Lacroix_WMDE, Addshore, QZanden, Salgo60, Minhnv-2809, Urbanecm, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GPHemsley, Darkdadaah, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
