[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic closed this task as "Resolved".
GoranSMilovanovic added a comment.


  Ok. In any case the fix to this script is easy if anything similar happens 
again. I will begin to monitor the Sqoop runs more closely.
  Thank you @elukey !

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread elukey
elukey added a comment.


  I think we should be fine from now on, I wouldn't add more complexity to what 
we have :)

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread GoranSMilovanovic
GoranSMilovanovic added a comment.


  @elukey Thank you. I was thinking along the following lines:
  
  - if due to any updates, upgrades, or other changes, this turns out to be a 
persistent problem,
  - then is there a way to ask from some Analytics Client if 
`org.mariadb.jdbc.Driver`  should or should not be specified,
  - e.g. by issuing a query, or checking something, directly from R/Python 
during the script runtime?
  
  Because if something like that is possible, than I could re-factor the script 
a bit to include one additional decision step to formulate the Sqoop call 
correctly.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread elukey
elukey added a comment.


  @GoranSMilovanovic sure! During the migration of the hosts where Hive 
Server/Metastore runs to Debian Buster, we encountered a lot of problems with 
the only available java lib for mysql, namely the one containing the 
`org.mariadb.jdbc.Driver` JDBC driver. We have now reverted back to the old 
mysql driver, manually porting the missing debian packages from Stretch to 
Buster, and now sqoop needs to run without the extra --driver option. So this 
option caused problems due to us trying to figure out how to upgrade our 
systems following Debian best practices, but hopefully now we should be good 
(at least until Debian Bullseye, the new version, will be out).
  
  I have opened https://issues.apache.org/jira/browse/HIVE-25020 to Hive 
upstream to investigate further, but it will be a longer topic to solve I am 
afraid.
  
  Lemme know if you want to know more.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread Maintenance_bot
Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, Maintenance_bot
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb, Lalamarie69, 
Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, 
Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread gerritbot
gerritbot added a comment.


  Change 683801 **merged** by GoranSMilovanovic:
  
  [analytics/wmde/WD/WikidataAnalytics@master] T281316
  
  https://gerrit.wikimedia.org/r/683801

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, gerritbot
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, Lalamarie69, maantietaja, Alter-paule, Beast1978, 
Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, 
Scott_WUaS, JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread gerritbot
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, gerritbot
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, Lalamarie69, maantietaja, Alter-paule, Beast1978, 
Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, 
Scott_WUaS, JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-30 Thread gerritbot
gerritbot added a comment.


  Change 683801 had a related patch set uploaded (by GoranSMilovanovic; author: 
GoranSMilovanovic):
  
  [analytics/wmde/WD/WikidataAnalytics@master] T281316
  
  https://gerrit.wikimedia.org/r/683801

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, gerritbot
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic added a comment.


  @WMDE-leszek @elukey I would like to learn from this.
  
  The following argument to `/usr/bin/sqoop`
  
  > --driver org.mariadb.jdbc.Driver
  
  seems to have been causing us trouble for some time already in 
WDCM_Sqoop_Clients.R 

  
  I am now uncertain whether should I deploy the script to the respective 
Analytics Client (stat1004 in this particular case)  with this argument present 
or not.
  
  It would  help me if I could understand the cause of the recent failure in 
relation to `driver org.mariadb.jdbc.Driver`, similarly to what @elukey  asked 
for in T281316#7040737 .
  
  Thanks.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-29 Thread GoranSMilovanovic
GoranSMilovanovic reopened this task as "Open".
GoranSMilovanovic added a comment.


  @elukey Let's take a close look at this, if you agree.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey, GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-29 Thread elukey
elukey closed this task as "Resolved".
elukey claimed this task.
elukey added a comment.


  No issues from our side, going to close, please reopen if necessary!

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment.


  @elukey No worries. Let me know if you need any external tests performed.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread elukey
elukey added a comment.


  Nice! Let's keep it open since I want to understand if we need to use 
`--driver com.mysql.jdbc.Driver` or not, it will have some impact also for 
Analytics, thanks a lot for bringing this up and sorry for the  trouble!

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic lowered the priority of this task from "High" to "Low".

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment.


  @elukey
  
  > As a quick workaround you should be able to unblock your queries just 
removing --driver org.mariadb.jdbc.Driver, can you try to see if it works?
  
  That would do, thank you Luca! I will keep the ticket open just in case, 
until the update script is done, but the `enwiki` Sqoop test was already a 
success.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread GoranSMilovanovic
GoranSMilovanovic added a comment.


  @elukey Thank your for a prompt response, Luca!
  
  > The first error is unrelated, it is due to the fact that you have created 
the /tmp/wmde directory as analytics-privatedata (and by default permissions 
allow only user + group, not others) and your username can't read files in 
there.
  
  I thought so.
  
  > We moved the hive server nodes to Debian Buster recently (T231067 
) and we had a problem with Hive and 
Mariadb...
  
  I suspected so : )
  
  > As a quick workaround you should be able to unblock your queries just 
removing --driver org.mariadb.jdbc.Driver, can you try to see if it works?
  
  Testing now and getting back to you then. Thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-28 Thread elukey
elukey added a comment.


  Hi Goran!
  
  We moved the hive server nodes to Debian Buster recently (T231067 
) and we had a problem with Hive and 
Mariadb, highlights in https://issues.apache.org/jira/browse/HIVE-25020.
  As a quick workaround you should be able to unblock your queries just 
removing `--driver org.mariadb.jdbc.Driver`, can you try to see if it works?
  
  The first error is unrelated, it is due to the fact that you have created the 
/tmp/wmde directory as `analytics-privatedata` (and by default permissions 
allow only user + group, not others) and your username can't read files in 
there.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: elukey
Cc: elukey, Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T281316: WDCM_Sqoop_Clients.R fails from stat1004 (again)

2021-04-27 Thread GoranSMilovanovic
GoranSMilovanovic triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T281316

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: Lydia_Pintscher, MisterSynergy, Aklapper, Manuel, WMDE-leszek, 
GoranSMilovanovic, Invadibot, maantietaja, Akuckartz, 4748kitoko, Nandana, 
Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
JAllemandou, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs