[Cloud] Code of Conduct Committee -- call for new members

[Martin Urbanec]

Hello all,

The Code of Conduct Committee is a team of five trusted individuals (plus
five auxiliary members) with diverse affiliations responsible for general
enforcement of the Code of conduct for Wikimedia technical spaces.
Committee members are in charge of processing complaints, discussing with
the parties affected, agreeing on resolutions, and following up on their
enforcement. For more on their duties and roles, see
https://www.mediawiki.org/wiki/Code_of_Conduct/Committee.

This is a call for community members interested in volunteering for
appointment to this committee. Volunteers serving in this role should be
experienced Wikimedians or have had experience serving in a similar
position before.

The current committee is doing the selection and will research and discuss
candidates. Six weeks before the beginning of the next Committee term,
meaning July 16, 2023, they will publish their candidate slate (a list of
candidates) on-wiki. The community can provide feedback on these
candidates, via private email to the group choosing the next Committee. The
feedback period will be two weeks. The current Committee will then either
finalize the slate, or update the candidate slate in response to concerns
raised. If the candidate slate changes, there will be another two week
feedback period covering the newly proposed members. After the selections
are finalized, there will be a training period, after which the new
Committee is appointed. The current Committee continues to serve until the
feedback, selection, and training process is complete.

If you are interested in serving on this committee or like to nominate a
candidate, please write an email to techconductcandidates AT wikimedia.org
with details of your experience on the projects, your thoughts on the code
of conduct and the committee and what you hope to bring to the role and
whether you have a preference in being auxiliary or main member of the
committee. The committee consists of five main members plus five auxiliary
members and they will serve for a year; all applications are appreciated
and will be carefully considered. The deadline for applications is *the end
of day on May 31 2023*.

Please feel free to pass this invitation along to any users who you think
may be qualified and interested.

Best,
Martin Urbanec, on behalf of the Code of Conduct Committee
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information: 
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

[Cloud] Re: [Cloud-announce] Re: Two toolforge outages coming next week, Monday and Thursday

[Martin Urbanec]

Hi,

Is there an easy way to stop jsub's "failed to redirect job output" error
messages I receive in my mailbox during the NFS outage, ideally, for all of
the ~50 jobs I have scheduled for my tools?

Unfortunately, currently I receive dozens of mails every hours.

Martin

On Mon, Apr 3, 2023, 1:24 AM Andrew Bogott  wrote:

> Reminder:  The first of these outages will start in about 30 minutes.
> Toolforge NFS will be read-only for as long as 18-19 hours.
>
>
>
> On 3/29/23 2:17 PM, Andrew Bogott wrote:
>
> There will be two major Toolforge outages this coming week. Each outage
> will cause tool downtime and may require manual restarts afterwards.
>
> The first outage is an NFS migration [0] and will take place on Monday,
> beginning at around 0:00 UTC and lasting well into the day, possibly as
> late as 19:00 UTC. During this long period, Toolforge NFS will be
> read-only. This will cause most tools (for example, anything that writes a
> log file) to fail.
>
> The second outage will be a database migration [1] and will take place on
> Thursday at 17:00UTC. During this window ToolsDB will be read-only. This
> migration should take about an hour but unexpected side-effects may extend
> the downtime.
>
> We try very hard to avoid outages of this magnitude, but at this point we
> need to choose downtime over the increasing risk of data loss.
>
> More details can be found below.
>
>
> [0] NFS Outage and system reboots Monday: The existing toolforge NFS
> server is running on aging hardware and lacks a straightforward path for
> maintenance or upgrading. To improve this we are moving NFS to a cinder+VM
> platform which should support easier upgrades, migrations, and expansions
> in the future. In order to maintain data integrity during the migration,
> the old server will need to be made read-only while the last set of file
> changes is synchronized with the new server. Because the NFS service is
> actively used, it will take many hours to complete the final sync.
>
> To ensure stable mounts of the new server, every node in Toolforge will
> be rebooted as part of this migration. That means that even tools which do
> not use NFS will be affected, although most tools should restart gracefully.
>
> This task is documented as https://phabricator.wikimedia.org/T333477.
>
>
> [1] DB outage Thursday: As part of the ongoing effort to upgrade
> user-created Toolforge databases, we will migrate ToolsDB to a new VM
> that will have a more recent version of Debian and MariaDB and will use a
> more resilient storage solution.
>
> The new VM is ready, and we plan to point all tools to use it on *Apr, 6
> 2023 at 17:00 UTC*.
>
> This will involve about *1 hour of read-only time* for the database. Any
> existing database connection will be terminated, and if your tool does not
> reconnect automatically you might have to restart it manually.
>
> An email will be sent shortly before starting the migration, and when it's
> finished.
>
> Please also make sure your tool is connecting to the database using the
> canonical hostname *tools.db.svc.wikimedia.cloud* and not any other
> hostname or IP address.
>
> For more details, and to report any issue, you can read or leave a comment
> at https://phabricator.wikimedia.org/T333471
>
> For more context you can also check out the parent task
> https://phabricator.wikimedia.org/T301949
>
>
> ___
> Cloud-announce mailing list -- cloud-annou...@lists.wikimedia.org
> List information:
> https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/
> ___
> Cloud mailing list -- cloud@lists.wikimedia.org
> List information:
> https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
>
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information: 
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

[Cloud] [reminder] Code of Conduct Committee – call for new members

[Martin Urbanec]

Hello!

Just a reminder that there is only a week left for nominations to the
technical Code of Conduct Committee (CoCC). As you can read below in my
previous email, the Code of Conduct Committee is a term of five trusted
individuals (plus five auxiliary members) who are responsible for
enforcement of the Code of Conduct for Wikimedia technical spaces.

If you are interested in serving on the committee, please write an email to
techconductcandidates AT wikimedia.org with details of your experience on
the projects, your thoughts on the code of conduct and the committee and
what you hope to bring to the role and whether you have a preference in
being auxiliary or main member of the committee. Appointed members will
serve on the committee for a year.  The deadline for applications is *the
end of day on 30 April 2022*.

Please feel free to pass this invitation along to any users who you think
may be qualified and interested. I am also happy to answer any questions
about the CoCC – if you have any, please feel free to let me know via email.

Best regards,

Martin Urbanec, on behalf of the Code of Conduct Committee

pá 15. 4. 2022 v 10:44 odesílatel Martin Urbanec <
martin.urba...@wikimedia.cz> napsal:

> Hello all,
>
> It's coming close to the time for annual appointments of community members
> to serve on the Code of Conduct committee (CoCC). The Code of Conduct
> Committee is a team of five trusted individuals (plus five auxiliary
> members) with diverse affiliations responsible for general enforcement of
> the Code of conduct for Wikimedia technical spaces. Committee members are
> in charge of processing complaints, discussing with the parties affected,
> agreeing on resolutions, and following up on their enforcement. For more on
> their duties and roles, see
> https://www.mediawiki.org/wiki/Code_of_Conduct/Committee.
>
> This is a call for community members interested in volunteering for
> appointment to this committee. Volunteers serving in this role should be
> experienced Wikimedians or have had experience serving in a similar
> position before.
>
> The current committee is doing the selection and will research and discuss
> candidates. Six weeks before the beginning of the next Committee term,
> meaning 07 May 2022, they will publish their candidate slate (a list of
> candidates) on-wiki. The community can provide feedback on these
> candidates, via private email to the group choosing the next Committee. The
> feedback period will be two weeks. The current Committee will then either
> finalize the slate, or update the candidate slate in response to concerns
> raised. If the candidate slate changes, there will be another two week
> feedback period covering the newly proposed members. After the selections
> are finalized, there will be a training period, after which the new
> Committee is appointed. The current Committee continues to serve until the
> feedback, selection, and training process is complete.
>
> If you are interested in serving on this committee or like to nominate a
> candidate, please write an email to techconductcandidates AT wikimedia.org
> with details of your experience on the projects, your thoughts on the code
> of conduct and the committee and what you hope to bring to the role and
> whether you have a preference in being auxiliary or main member of the
> committee. The committee consists of five main members plus five auxiliary
> members and they will serve for a year; all applications are appreciated
> and will be carefully considered. The deadline for applications is *the
> end of day on 30 April 2022*.
>
> Please feel free to pass this invitation along to any users who you think
> may be qualified and interested.
>
> Best,
>
> Martin Urbanec, on behalf of the Code of Conduct Committee
>
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information: 
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

[Cloud] Code of Conduct Committee – call for new members

[Martin Urbanec]

 Hello all,

It's coming close to the time for annual appointments of community members
to serve on the Code of Conduct committee (CoCC). The Code of Conduct
Committee is a team of five trusted individuals (plus five auxiliary
members) with diverse affiliations responsible for general enforcement of
the Code of conduct for Wikimedia technical spaces. Committee members are
in charge of processing complaints, discussing with the parties affected,
agreeing on resolutions, and following up on their enforcement. For more on
their duties and roles, see
https://www.mediawiki.org/wiki/Code_of_Conduct/Committee.

This is a call for community members interested in volunteering for
appointment to this committee. Volunteers serving in this role should be
experienced Wikimedians or have had experience serving in a similar
position before.

The current committee is doing the selection and will research and discuss
candidates. Six weeks before the beginning of the next Committee term,
meaning 07 May 2022, they will publish their candidate slate (a list of
candidates) on-wiki. The community can provide feedback on these
candidates, via private email to the group choosing the next Committee. The
feedback period will be two weeks. The current Committee will then either
finalize the slate, or update the candidate slate in response to concerns
raised. If the candidate slate changes, there will be another two week
feedback period covering the newly proposed members. After the selections
are finalized, there will be a training period, after which the new
Committee is appointed. The current Committee continues to serve until the
feedback, selection, and training process is complete.

If you are interested in serving on this committee or like to nominate a
candidate, please write an email to techconductcandidates AT wikimedia.org
with details of your experience on the projects, your thoughts on the code
of conduct and the committee and what you hope to bring to the role and
whether you have a preference in being auxiliary or main member of the
committee. The committee consists of five main members plus five auxiliary
members and they will serve for a year; all applications are appreciated
and will be carefully considered. The deadline for applications is *the end
of day on 30 April 2022*.

Please feel free to pass this invitation along to any users who you think
may be qualified and interested.

Best,

Martin Urbanec, on behalf of the Code of Conduct Committee
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information: 
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

[Cloud] Re: Kubernetes-based jobs engine -- how to use Python virtual environments?

[Martin Urbanec]

Hello everyone,

Thanks for the suggestion. Does this mean that I have to maintain 2 virtual
environments (one for bastion and the other one for k8s pod my code will
actually run in)?

Martin

po 4. 4. 2022 v 10:06 odesílatel Arturo Borrero Gonzalez <
aborr...@wikimedia.org> napsal:

>
>
> On 4/2/22 14:53, Martin Urbanec wrote:
> > Hello,
> >
> > I just received a dozen emails about grid engine migration. I tried to
> > migrate my personal tool (tool.martin-urbanec) first. This tool
> > currently generates a Jupyter-notebook based report daily.
> >
> > I do that by calling jupyter nbconvert --to html --execute
> > community_configuration_usage.ipynb from a virtual environment where
> > Jupyter is installed, together with a couple of other Python modules.
> >
> > I managed to create new virtual environment that works from the new
> > Buster bastion, and it works when executed directly from the bastion,
> > but I can't get it to execute via the k8s-based engine:
>
> Your problem may be related to bootstrapping the venv. See if this
> information can help you:
>
>
> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_jobs
>
> This is very similar to what JMC89 replied in the other email.
>
> --
> Arturo Borrero Gonzalez
> Site Reliability Engineer
> Wikimedia Cloud Services
> Wikimedia Foundation
>
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information: 
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

[Cloud] Re: [Cloud-announce] Re: Do you use /mnt/nfs/secondary-scratch? Expect downtime tomorrow.

[Martin Urbanec]

Hello Andrew,

Will there be any impact on Toolforge users using scratch?

Martin

út 18. 1. 2022 v 18:00 odesílatel Andrew Bogott 
napsal:

> Since no one expressed concerns about this, I'm going to go ahead and
> roll this out tomorrow morning at 16:00 UTC.  Here's what to expect:
>
> 1) If your VM mounts secondary-scratch but doesn't actually use it,
> nothing much will happen
> 2) If your VM or tool has an open file on that volume when the
> switchover happens, it will probably freeze up.  I will reboot VMs that
> this happens to.
> 3) If you had files on the scratch volume before this change, they will
> be gone after the change. Precious files will be recoverable after the
> fact for a few weeks.
>
> -Andrew
>
>
> On 1/14/22 2:06 PM, Andrew Bogott wrote:
> > Hello, all!
> >
> > We are in the process of re-engineering and virtualizing[0] the NFS
> > service provided to Toolforge and VMs. The transition will be rocky
> > and involve some service interruption... I'm still running tests to
> > determine exactly host much disruption will be required.
> >
> > The first volume that I'd like to replace is 'scratch,' typically
> > mounted as /mnt/nfs/secondary-scratch. I'm seeking feedback about how
> > vital scratch uptime is to your current workflow, and how disruptive
> > it would be to lose data there.
> >
> > If you have a project or tool that uses scratch, please respond with
> > your thoughts! My preference would be to wipe out all existing data on
> > scratch and also subject users to several unannounced periods of
> > downtime, but I also don't want anyone to suffer. If you have
> > important/persistent data on that volume then the WMCS team will work
> > with you to migrate that data somewhere safer, and if you have an
> > important workflow that will break due to Scratch downtime then I'll
> > work harder on devising a low-impact roll-out.
> >
> > Thank you!
> >
> > -Andrew
> >
> > [0] https://phabricator.wikimedia.org/T291405
> >
> ___
> Cloud-announce mailing list -- cloud-annou...@lists.wikimedia.org
> List information:
> https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/
> ___
> Cloud mailing list -- cloud@lists.wikimedia.org
> List information:
> https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
>
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information: 
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

Re: [Cloud] [Ops] Change to how Cloud VPS and Toolforge contact Wikis

[Martin Urbanec]

Hi Arturo,

a quick question: MediaWIki has a strict limit on bad logins. If all of
WMCS will be NATed, that would mean that *any* bot having too many bad
login attempts could block all other bots from logging in. Is that
prevented through technical measures, somehow?

Thanks,

Martin

po 25. 1. 2021 v 11:56 odesílatel Arturo Borrero Gonzalez <
aborr...@wikimedia.org> napsal:

> Hello,
>
> we are planning to change how Cloud VPS instances and Toolforge tools
> contact
> WMF-hosted wikis, in particular the source IP address for the network
> connection.
> The new IP address that wikis will see is 185.15.56.1.
>
> The change is scheduled to go live on 2021-02-08.
>
> More detailed information in wikitech:
>
>   https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis
>
> If you are a Cloud VPS user or Toolforge developer, check your tools after
> that
> date to make sure they are properly running. If you detect a block, a
> rate-limit
> or similar, please let us know.
>
> If you are a WMF SRE or engineer involved with the wikis, be informed that
> this
> address could generate a significant traffic volume, perhaps about 30%-40%
> total
> wiki edits. We are trying to smooth the change as much as possible, so
> please
> send your feedback if you think there is something we didn't account for
> yet.
>
> Thanks, best regards.
> --
> Arturo Borrero Gonzalez
> SRE / Wikimedia Cloud Services
> Wikimedia Foundation
>
> ___
> Ops mailing list
> o...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/ops
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

[Martin Urbanec]

MusikAnimal is right, however, Wikidata and Commons either have a sui
generis slice, or they share it with a few very large wikis. Tools that do
any kind of crosswiki analysis would instantly break, as most of them
utilise joining by Wikidata items at the very least.

I second Maarten here. This would mean a lot of things that currently
require a (relatively simple) SQL query would need a full script, which
would do the join at the application level.

I fully understand the reasoning, but there needs to be some replacement.
Intentionally introduce breaking changes while providing no "new standard"
is a bad pattern in a community environment.

Martin

On Wed, Nov 11, 2020, 10:31 PM MusikAnimal  wrote:

> Technically, cross-wiki joins aren't completely disallowed, you just have
> to make sure each of the db names are on the same slice/section, right?
>
> ~ MA
>
> On Wed, Nov 11, 2020 at 4:11 PM Maarten Dammers 
> wrote:
>
>> Hi Joaquin,
>> On 10-11-2020 21:26, Joaquin Oltra Hernandez wrote:
>>
>> TLDR: Wiki Replicas' architecture is being redesigned for stability and
>> performance. Cross database JOINs will not be available and a host
>> connection will only allow querying its associated DB. See [1]
>> 
>> for more details.
>>
>> If you only think of Wikipedia, not a lot will break probably, but if you
>> take into account Commons and Wikidata a lot will break. A quick grep in my
>> folder with Commons queries returns 123 lines with cross database joins. So
>> yes, stuff will break and tools will be abandoned. This follows the
>> practice that seems to have become standard for the WMF these days:
>> Decisions are made with a small group within the WMF without any community
>> involved. Only after the decision has been made, it's announced.
>>
>> Unhappy and disappointed,
>>
>> Maarten
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] [Cloud-announce] Fwd: Phasing out the .wmflabs tld on September 8th

[Martin Urbanec]

Thanks @Huji Lee ! TIL openssh has super simple
ProxyJump directive :).

po 7. 9. 2020 v 21:32 odesílatel Huji Lee  napsal:

> This is also a good time remind everyone to update their ProxyJump
> configuration [1] to include the new TLDs. At least I would have forgotten
> it if I had not seen this remidner.
>
>   [1]
> https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#ProxyJump_(recommended)
>
> On Mon, Sep 7, 2020 at 2:31 PM Andrew Bogott 
> wrote:
>
>> Reminder: This naming change is happening tomorrow, less than 24 hours
>> from now.  Update your .ssh/config now so you aren't confused tomorrow!
>>
>> -Andrew
>>
>>
>>  Forwarded Message 
>> Subject: Phasing out the .wmflabs tld on September 8th
>> Date: Wed, 19 Aug 2020 08:55:18 -0500
>> From: Andrew Bogott  
>> Reply-To: andrewbog...@gmail.com
>> To: cloud-annou...@lists.wikimedia.org
>>
>> tl;dr:
>>
>> VMs created on or after September 8th will stop having .eqiad.wmflabs
>> domains, and be found only under .eqiad1.wikimedia.cloud
>>
>>
>> The whole story:
>>
>> Currently cloud-vps VMs stand astride two worlds: wmflabs and
>> wikimedia.cloud. Here's the status quo:
>>
>>
>> - New VMs get three different DNS entries:
>> hostname.project.eqiad1.wikimedia.cloud, hostname.project.eqiad.wmflabs,
>> and hostname.eqiad.wmflabs [0]
>>
>> - Reverse DNS lookups return hostnames under eqiad1.wikimedia.cloud
>>
>> - VMs themselves believe (e.g. via hostname -f) that they're still under
>> eqiad.wmflabs
>>
>> That hybrid system has done a good job maintaining backwards
>> compatibility, but it's a bit of a mess. In the interest of simplifying,
>> standardizing, and eliminating ever more uses of the term 'labs', we're
>> going to start phasing out the wmflabs domain name. Beginning on September
>> 8th, new VMs will no longer receive any naming associated with .wmflabs [1].
>>
>> - New VMs will get one DNS entry: hostname.project.eqiad1.wikimedia.cloud
>>
>> - New VMs will continue to have a pointer DNS entry that refers to the
>> .wikimedia.cloud name
>>
>> - New VMs will be assigned an internal hostname under .wikimedia.cloud
>>
>> In order to avoid breaking existing systems, these changes will NOT be
>> applied retroactively to existing VMs. Old DNS entries will live on until
>> the VM is deleted and should be largely harmless.  If, however, you find
>> yourself rewriting code in order to deal with VMs under both domains (due
>> to the change in hostname -f behavior), don't worry -- adjusting an old VM
>> to identify as part of .wikimedia.cloud only requires a simple change to
>> /etc/hosts. I'll be available to make that change for any project that
>> chooses consistency over backwards-compatibility.
>>
>>
>> [0]
>> https://phabricator.wikimedia.org/phame/post/view/191/new_names_for_everyone
>>
>> [1] https://phabricator.wikimedia.org/T260614
>>
>>
>> ___
>> Wikimedia Cloud Services announce mailing list
>> cloud-annou...@lists.wikimedia.org (formerly
>> labs-annou...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Toolforge: new domain toolforge.org

[Martin Urbanec]

Yes, you need to add the host to the allowed hosts.

Martin

On Tue, Jul 7, 2020, 3:13 PM Roy Smith  wrote:

> I've started getting:
>
> > Invalid HTTP_HOST header: 'spi-tools-dev.toolforge.org'. You may need
> to add 'spi-tools-dev.toolforge.org' to ALLOWED_HOSTS.
>
> I assume that's related to the change?
>
>
>
> > On Jul 7, 2020, at 6:50 AM, Arturo Borrero Gonzalez <
> aborr...@wikimedia.org> wrote:
> >
> > On 2020-07-06 17:59, Arturo Borrero Gonzalez wrote:
> >> Hi there!
> >>
> >> Tomorrow 2020-07-06 at about 10:00 UTC we will enable the legacy
> redirector and
> >> this migration will be completed.
> >>
> >> All requests to tools.wmflabs.org/ will be permanently
> redirected to
> >> .toolforge.org.
> >>
> >> If you need additional context about this, please read:
> >>
> >> https://wikitech.wikimedia.org/wiki/News/Toolforge.org
> >>
> >> Please reach out if you need help or have doubts.
> >>
> >
> > This has been done!
> >
> > regards.
> >
> > --
> > Arturo Borrero Gonzalez
> > SRE / Wikimedia Cloud Services
> > Wikimedia Foundation
> >
> > ___
> > Wikimedia Cloud Services mailing list
> > Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Toolforge: new domain toolforge.org

[Martin Urbanec]

Hmm, perhaps we should update the interwiki map as well?

Martin

po 6. 7. 2020 v 18:00 odesílatel Arturo Borrero Gonzalez <
aborr...@wikimedia.org> napsal:

> Hi there!
>
> Tomorrow 2020-07-06 at about 10:00 UTC we will enable the legacy
> redirector and
> this migration will be completed.
>
> All requests to tools.wmflabs.org/ will be permanently
> redirected to
> .toolforge.org.
>
> If you need additional context about this, please read:
>
> https://wikitech.wikimedia.org/wiki/News/Toolforge.org
>
> Please reach out if you need help or have doubts.
>
> regards.
>
> --
> Arturo Borrero Gonzalez
> SRE / Wikimedia Cloud Services
> Wikimedia Foundation
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud-announce] [reminder] Pywikibot will end Python 2 support in July!

[Martin Urbanec]

As the last release of Python 2 is finally out, the July release of
Pywikibot is going to be the **last release that supports Python 2**.
Support of Python 3.4 and MediaWiki older than 1.19 is also going to be
dropped. After this release, Pywikibot is not going to receive any further
patches and bug fixes related to those Python and MediaWiki versions.
Functions and other stuff specific to Python 3.4, Python 2.x or MediaWiki
older than 1.19 will be removed.

For your convenience, this release is marked with a "python2"
git tag and it is also the last 3.0.x release. In case you really need it,
the Pywikibot team created /shared/pywikibot/core_python2 repository in
Toolforge and a python2-pywikibot package in software repositories of some
operating systems.

The Pywikibot team strongly recommends that you migrate your scripts from
Python 2 to Python 3. The migration steps were described in the previous
message, which can be found here:
https://lists.wikimedia.org/pipermail/pywikibot/2020-January/009976.html
Detailed plan of Python 2 deprecation with dates is described here:
https://www.mediawiki.org/wiki/Manual:Pywikibot/Compatibility

If you encounter any problems with the migration, you can always ask us
here: https://phabricator.wikimedia.org/T242120

Best regards,

Pywikibot team
___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud] [Cloud-announce] [Cloud VPS] Puppet labs/private.git data loss incident affecting some projects

[Martin Urbanec]

Thank you very much, Andrew and company.

čt 4. 6. 2020 v 18:58 odesílatel Andrew Bogott 
napsal:

> Followup on this:
>
> The WMCS team is pretty sure that all user-facing services have been
> restored. If you encounter any current unexpected breakage, please email
> me directly or use !help on IRC.
>
> There's still a fair bit of less-urgent cleanup left to do.  Puppet will
> remain disabled on most VMs until that's finished, which may take a day
> or two.
>
> -Andrew + the WMCS team.
>
>
> On 6/4/20 10:18 AM, Bryan Davis wrote:
> > At 2020-06-04T11:12 UTC a change was merged to the
> > operations/puppet.git repository which resulted in data loss for Cloud
> > VPS projects using a local Puppetmaster
> > (role::puppetmaster::standalone). The specific data loss is removal of
> > any local to the Puppetmaster instance commits overlaid on the
> > upstream labs/private.git repository. These patches would have
> > contained passwords, ssh keys, TLS certificates, and similar
> > authentication information for Puppet managed configuration.
> >
> > The majority of Cloud VPS projects are not affected by this
> > configuration data loss. Several highly used and visible projects,
> > including Toolforge (tools) and Beta Cluster (deployment-prep), have
> > some impact. We have disabled Puppet across all Cloud VPS instances
> > that were reachable by our central command and control service (cumin)
> > and are currently evaluating impact and recovering data from
> > /var/logs/puppet.log change logs where available.
> >
> > More information will be collected at
> >  and an incident report
> > will also be prepared once the initial response is complete.
> >
> > Bryan
>
>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] User tables

[Martin Urbanec]

Hmm, does create temporary table work? If not, can it?

Martin

On Fri, May 8, 2020, 11:04 PM Huji Lee  wrote:

> I was thinking of running a query, temporarily store its output on
> ToolsDB. Join it with some other query and then throw it away. All in small
> scale and for short-term.
>
> On Fri, May 8, 2020 at 5:01 PM Brooke Storm  wrote:
>
>> No, the wikireplicas are read-only.
>>
>> That said, ToolsDB (Toolforge user DB) is not equipped to handle becoming
>> an aggregate datamart for the wiki replicas at any thing close to large
>> (wiki) scale--just small chunks that are ideally regularly cleaned up. Are
>> we talking about a database on a Cloud VPS instance or ToolsDB?
>>
>> Brooke Storm
>> SRE
>> Wikimedia Cloud servicesbst...@wikimedia.org
>> IRC: bstorm_
>>
>> On 5/8/20 12:31 PM, Huji Lee wrote:
>>
>> Hi all,
>> Is it possible to store data into user tables through queries on
>> Wikireplica DBs? Or is it only possible by mysqldump'ing from the replica
>> DB and loading into the user table in a separate step?
>> I am thinking of aggregate data.
>> Thanks!
>>
>> ___
>> Wikimedia Cloud Services mailing listcl...@lists.wikimedia.org (formerly 
>> lab...@lists.wikimedia.org)https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] [Cloud-announce] Pywikibot is ending Python 2 support!

[Martin Urbanec]

Together with the last Python 2 release from April, 2020, Pywikibot team
will release the **last version that supports Python 2**. We created a
**python2" tag** marking the version, so you can continue running your
Python 2 scripts using this tag, if you really need to.

After that version, Pywikibot is not going to receive any further patches
and bug fixes related to Python 2. Its code is going to be cleaned from
Python 2 specific functions, patches, deprecations and other stuff, so make
sure you'll use this tag if you still want to run Pywikibot using Python 2.

Pywikibot team strongly recommends to migrate your scripts to Python 3. To
make it happen, you can use Python 2to3 script installed by default with
Python 2.6+, see https://docs.python.org/2/library/2to3.html. You can also
just try to run your script using Python 3 (the "-simulate" parameter could
be handy) and fix all the issues. If you encounter problems with the
migration, you can always ask us here:
https://phabricator.wikimedia.org/T242120

Best regards,

Martin Urbanec and Dvorapa
___
Wikimedia Cloud Services announce mailing list
cloud-annou...@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud-announce] Pywikibot is ending Python 2 support!

[Martin Urbanec]

Together with the last Python 2 release from April, 2020, Pywikibot team
will release the **last version that supports Python 2**. We created a
**python2" tag** marking the version, so you can continue running your
Python 2 scripts using this tag, if you really need to.

After that version, Pywikibot is not going to receive any further patches
and bug fixes related to Python 2. Its code is going to be cleaned from
Python 2 specific functions, patches, deprecations and other stuff, so make
sure you'll use this tag if you still want to run Pywikibot using Python 2.

Pywikibot team strongly recommends to migrate your scripts to Python 3. To
make it happen, you can use Python 2to3 script installed by default with
Python 2.6+, see https://docs.python.org/2/library/2to3.html. You can also
just try to run your script using Python 3 (the "-simulate" parameter could
be handy) and fix all the issues. If you encounter problems with the
migration, you can always ask us here:
https://phabricator.wikimedia.org/T242120

Best regards,

Martin Urbanec and Dvorapa
___
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

Re: [Cloud] Please help with optimizing a SQL query

[Martin Urbanec]

Sorry, just realized. I want list of users who have such a first edit. Ie
if user A makes an upload that's not tagged, the result shouldn't contain
user A.

Anyway, I feel the query should work that way - given it should first
compute the subquery, and then limit it to rows having matching entry in
change_tag.

Martin

ne 24. 11. 2019 v 21:28 odesílatel Martin Urbanec <
martin.urba...@wikimedia.cz> napsal:

> Ah, yes, first *upload* to Commons tagged with that tag. Thanks a lot,
> John!
>
> Martin
>
> ne 24. 11. 2019 v 20:56 odesílatel John 
> napsal:
>
>> Thats not what that query is getting. that is getting their first upload
>> that is tagged with that change id. If you want to discount any non-upload
>> edits I can look at optimizing it.
>>
>> On Sun, Nov 24, 2019 at 2:39 PM Martin Urbanec <
>> martin.urba...@wikimedia.cz> wrote:
>>
>>> Hello,
>>>
>>> could someone please help me with optimizing the following query?
>>>
>>> USE commonswiki_p;
>>>
>>> SELECT first_upload, uploads, username FROM
>>> (
>>>   SELECT MIN(log_timestamp) AS first_upload, MIN(log_id) AS
>>> first_upload_id, COUNT(log_timestamp) AS uploads, log_user_text AS username
>>>   FROM logging_compat
>>>   LEFT JOIN user ON user_id = log_user
>>>   JOIN page ON log_page = page_id
>>>   WHERE log_type = "upload" AND (log_action = "upload" OR log_action =
>>> "overwrite") AND user_registration > "2019010100"
>>>   GROUP BY log_user
>>> ) AS first_uploads
>>> JOIN change_tag ON ct_log_id = first_upload_id
>>> WHERE ct_tag_id=21;
>>>
>>> It takes over 30 minutes :/. I want to have a list of users whose first
>>> contrib to Wikimedia Commons is tagged with tag number 21.
>>>
>>> Thanks!
>>>
>>> Martin
>>> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Please help with optimizing a SQL query

[Martin Urbanec]

Ah, yes, first *upload* to Commons tagged with that tag. Thanks a lot, John!

Martin

ne 24. 11. 2019 v 20:56 odesílatel John  napsal:

> Thats not what that query is getting. that is getting their first upload
> that is tagged with that change id. If you want to discount any non-upload
> edits I can look at optimizing it.
>
> On Sun, Nov 24, 2019 at 2:39 PM Martin Urbanec <
> martin.urba...@wikimedia.cz> wrote:
>
>> Hello,
>>
>> could someone please help me with optimizing the following query?
>>
>> USE commonswiki_p;
>>
>> SELECT first_upload, uploads, username FROM
>> (
>>   SELECT MIN(log_timestamp) AS first_upload, MIN(log_id) AS
>> first_upload_id, COUNT(log_timestamp) AS uploads, log_user_text AS username
>>   FROM logging_compat
>>   LEFT JOIN user ON user_id = log_user
>>   JOIN page ON log_page = page_id
>>   WHERE log_type = "upload" AND (log_action = "upload" OR log_action =
>> "overwrite") AND user_registration > "2019010100"
>>   GROUP BY log_user
>> ) AS first_uploads
>> JOIN change_tag ON ct_log_id = first_upload_id
>> WHERE ct_tag_id=21;
>>
>> It takes over 30 minutes :/. I want to have a list of users whose first
>> contrib to Wikimedia Commons is tagged with tag number 21.
>>
>> Thanks!
>>
>> Martin
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] Please help with optimizing a SQL query

[Martin Urbanec]

Hello,

could someone please help me with optimizing the following query?

USE commonswiki_p;

SELECT first_upload, uploads, username FROM
(
  SELECT MIN(log_timestamp) AS first_upload, MIN(log_id) AS
first_upload_id, COUNT(log_timestamp) AS uploads, log_user_text AS username
  FROM logging_compat
  LEFT JOIN user ON user_id = log_user
  JOIN page ON log_page = page_id
  WHERE log_type = "upload" AND (log_action = "upload" OR log_action =
"overwrite") AND user_registration > "2019010100"
  GROUP BY log_user
) AS first_uploads
JOIN change_tag ON ct_log_id = first_upload_id
WHERE ct_tag_id=21;

It takes over 30 minutes :/. I want to have a list of users whose first
contrib to Wikimedia Commons is tagged with tag number 21.

Thanks!

Martin
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] [Toolforge] Proxy maintenance operation next Monday 2019-10-28 @ 14:30 UTC

[Martin Urbanec]

Is there something you missed to say?

"operation which is migrating data stored in Redis which can be tricky. The
o"

Martin

po 21. 10. 2019 v 19:29 odesílatel Arturo Borrero Gonzalez <
aborr...@wikimedia.org> napsal:

> Hi there!
>
> Next Monday 2019-10-28 @ 14:30 UTC we will do a maintenance operation on
> Toolforge which consists in rebuilding the main front proxy [0] used to
> serve
> webservices. We expect this to be done within a 30 minutes window.
>
> The operation consists on replacing the old virtual machines supporting the
> proxy (currently running Debian Jessie) with more modern instances running
> Debian Buster. Both Grid/Kubernetes backends are affected by this change.
> We
> don't expect a lot of service downtime, but there is a key point in the
> operation which is migrating data stored in Redis which can be tricky. The
> o
>
> Examples of things affected by this change:
>
> * Browsing Toolforge webservices
> * Browsing to https://tools.wmflabs.org/
> * Browsing to https://tools.wmflabs.org/admin/ (Toolforge landing page)
> * Browsing PAWS (to some extent, since it shares part of the toolforge
> proxy)
>
> Example of things not affected by this change:
>
> * webservices backend operations
> * SSH bastions
> * grid queues, grid jobs
> * wiki-replicas, toolsdb
> * other CloudVPS projects
>
> regards.
>
> [0] https://phabricator.wikimedia.org/T235627
>
> --
> Arturo Borrero Gonzalez
> SRE / Wikimedia Cloud Services
> Wikimedia Foundation
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] Optimalizing a SQL query

[Martin Urbanec]

Hello,

I'm trying to run this query more fastly. Can you help me how to optimalize
it?

MariaDB [commonswiki_p]> select count(*) from logging_logindex where
log_type="thanks" and log_title="Martin_Urbanec";
+--+
| count(*) |
+--+
|6 |
+--+
1 row in set (8 min 42.87 sec)

MariaDB [commonswiki_p]>

It's supposed to provide web-accessible information, and it's not possible
to wait 8 mins for completion.

https://tools.wmflabs.org/sql-optimizer/  didn't offer any solution.

Any suggestions?

Martin
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] Fwd: [Cloud VPS alert] Puppet failure on tools-sgeexec-0905.tools.eqiad.wmflabs

[Martin Urbanec]

I'm not sure I should be the intended recipient of this email.What happened
there?

Best,
Martin

-- Forwarded message -
From: 
Date: Mon, 11 Mar 2019 at 09:40
Subject: [Cloud VPS alert] Puppet failure on
tools-sgeexec-0905.tools.eqiad.wmflabs
To: 



Puppet is failing to run on the "tools-sgeexec-0905.tools.eqiad.wmflabs"
instance in Wikimedia Cloud VPS.

Working Puppet runs are needed to maintain instance security and logins.
As long as Puppet continues to fail, this system is in danger of becoming
unreachable.

You are receiving this email because you are listed as member for the
project that contains this instance.  Please take steps to repair
this instance or contact a Cloud VPS admin for assistance.

For further support, visit #wikimedia-cloud on freenode or

___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] How to Setup Cron Job for Toolsforge's tool

[Martin Urbanec]

I'm afraid you have to create a small wrapper script, activate venv in it
and run that shell script from crontab.

Martin

út 25. pro 2018 0:30 odesílatel Jay prakash <0freerunn...@gmail.com> napsal:

> Hello, I am getting the following problem on the test. I have installed
> the module in venv as per
> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_(Python3_+_Kubernetes).
> and I have also test
>
> /data/project/jayprakashbot/www/python/venv/bin/python instread of python
> in cron file. same result
>
>
> ```error
>
> @tools-bastion-03:~$ cat cron-29.err
>
> Traceback (most recent call last):
>
>   File "/data/project/jayprakashbot/www/python/src/gen_stats.py", line 2,
> in 
>
> import toolforge
>
> ImportError: No module named 'toolforge'
>
> ```
>
> On Tue, Dec 25, 2018 at 3:30 AM Giovanni Tirloni 
> wrote:
>
>> On 12/24/18 7:50 PM, Jay prakash wrote:
>> > Hello,
>> >  I want to set up a cron job for my tool on tools forge.
>> >
>> >  From (venv) tools.indic-wsstats@tools-bastion-03:$
>> > ```
>> > 1. $ crontab -e
>> > 2. paste the 0 */6 * * * python
>> > /data/project/indic-wsstats/www/python/src/gen_stats.py
>> > 3. Ctrl O as default name like (tmp/XX)
>> > 4. Ctrl X
>> > ```
>> > Is this enough? Am I missing something?
>>
>> Hello,
>>
>>   That should be it. You can read more about it here:
>>
>>
>> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Creating_a_crontab
>>
>>   And here's a handy tool if you need help figuring out the schedule:
>>
>>https://crontab-generator.org
>>
>> Regards,
>>
>>
>> --
>> Giovanni Tirloni
>> Operations Engineer
>> Wikimedia Cloud Services
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Autopromoted groups aren't listed in user_groups table

[Martin Urbanec]

Depending on your usecase, you might find this useful as well:

Groups that are set because of wgAutopromote are stored in *logging*, so if
you want to know since when the user fulfils the condition for
autopromoting, it's available. See the following query made on enwiki.

MariaDB [enwiki_p]> select * from logging where log_type="rights" order by
log_timestamp desc limit 1;
+--+--+-++--+---+---+---++-+-+-+---+--+
| log_id   | log_type | log_action  | log_timestamp  | log_user | log_actor
| log_namespace | log_title | log_comment_id | log_comment | log_params
  |
log_deleted | log_user_text | log_page |
+--+--+-++--+---+---+---++-+-+-+---+--+
| 94360005 | rights   | autopromote | 20181030172205 | 16997303 | 0
| 2 | Kubaski   | 10 | |
a:2:{s:12:"4::oldgroups";a:0:{}s:12:"5::newgroups";a:1:{i:0;s:17:"extendedconfirmed";}}
|   0 | Kubaski   | 58637813 |
+--+--+-++--+---+---+---++-+-+-+---+--+
1 row in set (0.00 sec)

This row represents 17:22, 30 October 2018 Kubaski (talk | contribs) was
automatically updated from (none) to extended confirmed user (thank) on
https://en.wikipedia.org/wiki/Special:Log/rights .

Best,
Martin

út 30. 10. 2018 v 13:53 odesílatel Alex Monk  napsal:

> Hi max,
>
> I think the implicit groups don't get stored in the database by MediaWiki,
> they are determined at runtime.
> You might be able to go through
> https://www.mediawiki.org/w/api.php?action=help=query%2Busers
> with usprop=implicitgroups and filter them yourself.
>
> Many thanks
> Alex
>
> On Tue, 30 Oct 2018 at 09:40, max  wrote:
>
>> Hello. I'm a ruwiki user MBH.
>>
>> I want to ask, how can I get (from Quarry or Toollabs database replica)
>> a list of users having one of autopromoted groups (i.e. "autoconfirmed",
>> or "uploader" in ruwiki)? DB table user_groups contains only manually
>> assigned groups, such as "sysop".
>>
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Python Flask app - localization

[Martin Urbanec]

Hey everyone,

thank you very much for your helpful advices, both at IRC and in this list.
I created a little library that's satisfying my current needs. Its
sourcecode is on https://github.com/urbanecm/flask-jsonlocale and it can be
installed via pip install flask-JSONLocale.

Best,

Martin

st 19. 9. 2018 v 0:08 odesílatel Kunal Mehta 
napsal:

> Hi,
>
> On 9/18/18 7:13 AM, Martin Urbanec wrote:
> > I'm sometimes use Flask for my Toolforge tools. I'd like to localize
> > them, ideally with messages directory in similar format like MediaWiki
> > has in i18n directory.
>
> There's the pywikibot.i18n component[1][2], though I've never actually
> tried using it with a Flask tool.
>
> [1]
> https://gerrit.wikimedia.org/g/pywikibot/core/+/master/pywikibot/i18n.py
> [2]
>
> https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.html#module-pywikibot.i18n
>
> -- Legoktm
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Python Flask app - localization

[Martin Urbanec]

Hi Guilherme,

well, maybe I should have explained what I'm trying to do in the original
email. Fixing this mistake now.

I have a tool called Wikinity <https://tools.wmflabs.org/wikinity>. The
tool takes data from Wikidata and displays items with no image uploaded. It
became to be used by Czech Commons users in the preparation step for their
photo trip. The source
<https://phabricator.wikimedia.org/source/tool-wikinity/> is awful, to say
the most important parts from how its written:

   - Frontend is in public/*.php. Those files used to be HTML files while
   the tool was in one language only. I switched them to PHP only to be able
   to use Intuition.
   - Backend logic is in public/*.py. for the main purpose of tool,
   public/map.py is important, which selects appropriate Wikidata query
   (stored in queries/*.txt) based on GET parameters and creates a map.
   - JavaScript just displays the map generated by map.py.
   - Other files are used for other things like storing queries.

Yeah, I know its awful (but working :D). I plan to rewrite it. As I already
use Translatewiki, which generates locales
<https://phabricator.wikimedia.org/source/tool-wikinity/browse/master/messages/>
in
certain format, I want to keep it. The fornat is straightforward to me and
not like gettext which I managed to understood, but I think this is more
clear. Of course, you might not agree, that's my current point of view.

My plan is to convert this into a simple Flask app, which should
incorporate several backend CGI scripts written in Python into one file
with much more clear structure. This Flask app should ofc do the same
things like the current tool does. I know how to do everything from the
function side. The only one thing I'm thinking about is the localization -
more certainly, what way I should use to use locales in this format
<https://phabricator.wikimedia.org/source/tool-wikinity/browse/master/messages/cs.json>
in
a Flask app (more precisely, in its Jinja2 templates, I do not have
anything to localize in the Python part).

BTW, I tried to use Flask-Babel and I manged to have working localization
<https://tools.wmflabs.org/commons-mass-description>. The only one thing I
don't like is that it uses gettext, which I don't like and its not like
"modify this json file and the locales will change automatically", but
"modify this .po file, then compile it it and maybe it will work".

Martin

út 18. 9. 2018 v 19:10 odesílatel Guilherme Gonçalves <
guilherme.p.g...@gmail.com> napsal:

> Hi Martin,
>
> I know of a few approaches, depending on the complexity of your tool:
>
>- You can get pretty far with having your strings in, say, JSON files
>that you load and format yourself in code based on placeholders.
>- The Python standard library has a built-in gettext
><https://docs.python.org/2.7/library/gettext.html> module if you
>prefer a more standard format.
>- For a more comprehensive solution that adds date formatting, I heard
>good things about Flask-Babel <https://pythonhosted.org/Flask-Babel/>,
>but never used it myself.
>
> Whichever way you choose, I'd definitely recommend managing the
> translations through TranslateWiki <https://translatewiki.net/>, which
> you might already know about.
>
> Em ter, 18 de set de 2018 às 15:14, Martin Urbanec <
> martin.urba...@wikimedia.cz> escreveu:
>
>> Hi,
>>
>> I'm sometimes use Flask for my Toolforge tools. I'd like to localize
>> them, ideally with messages directory in similar format like MediaWiki has
>> in i18n directory.
>>
>> I know about Intuition for PHP, does anybody know about similar thing for
>> Flask application?
>>
>> Best,
>> Martin
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
>
> --
> Guilherme P. Gonçalves
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] Python Flask app - localization

[Martin Urbanec]

Hi,

I'm sometimes use Flask for my Toolforge tools. I'd like to localize them,
ideally with messages directory in similar format like MediaWiki has in
i18n directory.

I know about Intuition for PHP, does anybody know about similar thing for
Flask application?

Best,
Martin
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Looking into running job's output

[Martin Urbanec]

Thank you very much for your reply and explanation. It's PWB bot actually,
which is based on Python.

And yeah, I should send such mails to the list instead of Bryan. So I'm
CCing the list, so the reply is preserved forever.

Thank you again,
Martin

Dne po 17. zář 2018 22:39 uživatel Bryan Davis 
napsal:

> On Sun, Sep 16, 2018 at 11:50 PM, Martin Urbanec
>  wrote:
> > Hi, STDOUT/STDERR from jobs is being added to jobname.{out,err} files
> when
> > the job is ended. Is it possible to monitor the output while the job is
> > running?
>
> I believe that the logging of stdout and stderr is actually continuous
> from the point of view of Grid Engine, but there are a couple of
> things that may make it seem to be batched from the point of view of
> you observing the files:
>
> * NFS server write aggregation and NFS client inode status updates
> cause a several seconds to several minutes delay in a write by Grid
> Engine to a log file being visible from a Toolforge bastion where you
> are trying to read it.
>
> * Your application may actually be adding additional buffering of
> writes to stdout/stderr that are not flushed through to Grid Engine to
> send on to the NFS hosted log files until program completion (or the
> buffer filling up). One common place that I have personally seen this
> is with Python scripts. See
> https://stackoverflow.com/questions/107705/disable-output-buffering
> for some ideas on how to work around this if it is a problem for you.
> If you do have a python script that does a lot of stdout/stderr output
> completely disabling write buffering may have negative impacts on
> shared NFS server performance. Finding a way to line buffer (hang on
> to writes until you see a newline) will probably be much less
> impactful on our limited shared resources than trying to write each
> byte independently.
>
> Direct inspection of the raw stdout/stderr streams of a process
> running under Grid Engine's control is technically possible, but would
> require root permissions typically. You would have to know what exec
> host the job is running on, ssh to that host, and then use `strace` or
> a similar debugging tool to inspect the stream. This is not something
> that a normal Toolforge user can accomplish themselves.
>
> Questions like this make great topics for the
> cloud@lists.wikimedia.org mailing list. You can get help there not
> just from me, but also from the larger Toolforge user community. It
> also makes the responses easier for others to find when they have
> similar issues.
>
> Bryan
> --
> Bryan Davis  Wikimedia Foundation
> [[m:User:BDavis_(WMF)]] Manager, Technical EngagementBoise, ID USA
> irc: bd808v:415.839.6885 x6855
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] pip

[Martin Urbanec]

The issue is there is no PyPI package called webcite :). Compare result of
https://pypi.org/project/pywikibot/ (a page) and
https://pypi.org/project/webcite (404 page). Maybe
https://pypi.org/project/archivenow can help you?

Martin

po 13. 8. 2018 v 3:39 odesílatel Huji Lee  napsal:

> I plan to do so, but I want to figure out all the nuances first.
>
> For instance, the method I discovered and mentioned above does not seem to
> support all pip packages (i tried `pip install webcite` and it couldn't
> find a compatible version). Any thoughts?
>
> On Sun, Aug 12, 2018 at 7:18 PM Nick Wilson (Quiddity) <
> nwil...@wikimedia.org> wrote:
>
>> Hmm, there are two sub-sections related to pip within
>> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_(Python3_+_Kubernetes)
>> but nothing within
>> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Developing#Pywikibot
>> Perhaps you could add some details of what you got to work, in the
>> appropriate place? (I assume the latter?) Thanks! :)
>>
>>
>> On Sat, Aug 11, 2018 at 7:28 AM Huji Lee  wrote:
>>
>>> I think I found the solution actually. I found it on
>>> https://wikitech.wikimedia.org/wiki/ORES/Deployment#Update_wheels and
>>> am pasting it below as well:
>>>
>>> virtualenv -p python3 venv
>>> source venv/bin/activate
>>> pip install --upgrade pip
>>> pip install whatever
>>>
>>>
>>> On Sat, Aug 11, 2018 at 10:20 AM Huji Lee  wrote:
>>>
 Actually, I should have tested it before sending that email. When I run 
 python3
 -m venv $HOME/www/python/venv I get the following error. What am I
 missing here?

 The virtual environment was not created successfully because ensurepip
 is not
 available.  On Debian/Ubuntu systems, you need to install the
 python3-venv
 package using the following command.

 apt-get install python3-venv

 You may need to use sudo with that command.  After installing the
 python3-venv
 package, recreate your virtual environment.

 On Sat, Aug 11, 2018 at 8:57 AM Huji Lee  wrote:

> Perfect, thanks!
>
> On Sat, Aug 11, 2018 at 6:01 AM Michael Schönitzer <
> michael.schoenit...@wikimedia.de> wrote:
>
>> It's available – and you should use venv. Something like:
>>
>> $ python3 -m venv $HOME/www/python/venv$ source 
>> $HOME/www/python/venv/bin/activate$ pip install --upgrade pip*$* pip 
>> install whatever
>>
>> Cheers,
>>  M
>>
>>
>> 2018-08-11 2:51 GMT+02:00 Huji Lee :
>>
>>> Hi all,
>>>
>>> What is the best way to install pip on the Clouds servers? I have a
>>> pywikibot program that depends on a pip-based package, and would like 
>>> to be
>>> able to install that dependency using *pip install*.
>>>
>>> Thanks,
>>>
>>> Huji
>>>
>>> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>
>>
>>
>>
>> --
>> Michael F. Schönitzer
>>
>>
>>
>> Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>> Tel. (030) 219 158 26-0
>> http://wikimedia.de
>>
>> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge
>> allen Wissens frei teilhaben kann. Helfen Sie uns dabei!
>> http://spenden.wikimedia.de/
>>
>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens
>> e.V. Eingetragen im Vereinsregister des Amtsgerichts 
>> Berlin-Charlottenburg
>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt
>> für Körperschaften I Berlin, Steuernummer 27/681/51985.
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>>
>>
>> --
>> Nick Wilson (Quiddity)
>> Wikimedia Foundation
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Collect user's mail

[Martin Urbanec]

Ad IFTTT) Super idea, didn't know that Wikipedia is an trigger in this
service :D. They want article that is marked as article of the week
(cs.wiki do not have article of the day).

Martin

so 10. 2. 2018 v 19:10 odesílatel Nick Wilson (Quiddity) <
nwil...@wikimedia.org> napsal:

> On Sat, Feb 10, 2018 at 3:26 AM, Martin Urbanec
> <martin.urba...@wikimedia.cz> wrote:
> > Hello,
> >
> > upon request at Czech Wikipedia's helpdesk I created a tool that sends
> > notifications about new weekly articles by e-mail together with its first
> > paragraph and link to it.
>
>
> You could also potentially use IFTTT ?
> E.g. advise everyone to use
>
> https://ifttt.com/applets/306821p-read-a-weekly-digest-of-featured-wikipedia-articles
> or for people who want it daily
>
> https://ifttt.com/applets/64419059d-receive-an-email-with-wikipedia-s-article-of-the-day
>
> I use
> https://ifttt.com/applets/64362538d-automatically-get-a-weekly-vocabulary-list-of-wikipedia-words-of-the-day-in-your-email
> and thus get this once a week:
>
> http://storage9.static.itmages.com/i/18/0210/h_1518284828_5973154_9b0eebf21b.jpeg
> (ignore the incorrect branding. Naming things (perfectly accurately)
> is hard.)
>
> docs / notes are at
>
> http://blog.hatnote.com/post/124069724187/wikipedia-and-ifttt-a-technical-guide
>
>
> Alternatively, if the editors/readers wanted a list of "the most
> edited articles of the week", then there's http://weekly.hatnote.com/
> - There is not currently a Czech version, but adding that would be
> easier than reinventing the tool! :)
>
> HTH
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

-- 
Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Collect user's mail

[Martin Urbanec]

Repeating myself: A) having an email address in your account isn't required
for editing B) what if I don't have an account?

M.

so 10. 2. 2018 v 16:57 odesílatel Maximilian Doerr <
maximilian.do...@gmail.com> napsal:

> Why not just make use of the Special:EmailUser function?
>
> Cyberpower678
> English Wikipedia Account Creation Team
> English Wikipedia Administrator
> Global User Renamer
>
> On Feb 10, 2018, at 06:26, Martin Urbanec <martin.urba...@wikimedia.cz>
> wrote:
>
> Hello,
>
> upon request at Czech Wikipedia's helpdesk I created a tool that sends
> notifications about new weekly articles by e-mail together with its first
> paragraph and link to it. To accomplish such a thing I need user's email,
> so I firstly decided to store it in a database. To prevent this tool from
> spamming I of course require its confirmation by accessing an URL with a
> random string (MD5 hash of user's email *and* random number from 1 to
> 100; I mean, those two things are in one hash). You can have a look at this
> tool at tools.wmflabs.org/wiki2email/.
>
> My question is: Is this okay? Should I add some kind of formal information
> to the tool? If so, is there some help page? Should I stop with collecting
> mails at all and use some WMF-maintained service for mass-emailing (mailman
> at lists.wikimedia.org maybe?) and make the tool to just send an email to
> the list itself?
>
> This question came to my mind before creating, so I do appologize for
> asking after programming.
>
> Best regards,
> Martin Urbanec
> --
> Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
>
> ___
>
>
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

-- 
Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Collect user's mail

[Martin Urbanec]

Well, username is *also* a private information according the page
MarcoAurelio linked, so I would need to solve it anyway. I don't need to
have the emails but what if somebody w/o account at WP would like to use
this?

M.

so 10. 2. 2018 v 13:37 odesílatel Francisco Venancio <
fvenan...@wikimedia.org> napsal:

> Martin,
> Do you need the users' emails or sending them email messages enough?
>
> MediaWiki api could be used to send emails to users that have email set in
> their preference.
>
> See https://www.mediawiki.org/wiki/API:Emailuser
>
> Chico Venancio
> Cloud Services Technical Support
>
> Em 10 de fev de 2018 09:27, "Martin Urbanec" <martin.urba...@wikimedia.cz>
> escreveu:
>
>
>
> so 10. 2. 2018 v 13:23 odesílatel Guilherme Gonçalves <
> guilherme.p.g...@gmail.com> napsal:
>
>> Hi Martin,
>>
>> I'm not authoritative on PII policies at all, but here's a couple of
>> things that came to mind as I read your question.
>>
>> 2018-02-10 11:26 GMT+00:00 Martin Urbanec <martin.urba...@wikimedia.cz>:
>>
>>> To prevent this tool from spamming I of course require its confirmation
>>> by accessing an URL with a random string (MD5 hash of user's email *and* 
>>> random
>>> number from 1 to 100; I mean, those two things are in one hash).
>>>
>>
>> Does this mean the URL for a given email address can be guessed in at
>> most 100 attempts by someone who doesn't control the address? I think you'd
>> typically want to draw your random numbers from a much larger range, or use
>> as token something that was encrypted or signed with a secret only your
>> server knows. It would probably also make sense to make your URLs valid for
>> only a certain time.
>>
>
> *1000, but increased to 10 000 000, which should be big enough. I also can
> use more qualit hash than MD5 which will slow it down even more.
>
>>
>> However...
>>
>>
>>> Should I stop with collecting mails at all and use some WMF-maintained
>>> service for mass-emailing (mailman at lists.wikimedia.org maybe?) and
>>> make the tool to just send an email to the list itself?
>>>
>>
>> If creating a single mailing list is an option (for instance, you don't
>> plan on customizing the emails per user), this seems like a very good way
>> to go.
>>
>
> It is, this just was the easiest way for me when I was writing the tool.
>
>>
>>
>>>
>>> This question came to my mind before creating, so I do appologize for
>>> asking after programming.
>>>
>>> Best regards,
>>> Martin Urbanec
>>> --
>>> Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
>>>
>>> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>
>>
>>
>>
>> --
>> Guilherme P. Gonçalves
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> --
> Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

-- 
Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Collect user's mail

[Martin Urbanec]

Warning added, thanks for link.

Martin

so 10. 2. 2018 v 12:36 odesílatel MarcoAurelio <strig...@gmail.com> napsal:

> I'd say that you should display <
> https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#If_my_tools_collect_Private_Information...>
> *before* you collect that private information. It's true that email
> addresses are not listed in that page as private information, but EU-wide
> email addresses are private information so I'd display the warning
> nonetheless. Thanks.
>
> 2018-02-10 12:26 GMT+01:00 Martin Urbanec <martin.urba...@wikimedia.cz>:
>
>> Hello,
>>
>> upon request at Czech Wikipedia's helpdesk I created a tool that sends
>> notifications about new weekly articles by e-mail together with its first
>> paragraph and link to it. To accomplish such a thing I need user's email,
>> so I firstly decided to store it in a database. To prevent this tool from
>> spamming I of course require its confirmation by accessing an URL with a
>> random string (MD5 hash of user's email *and* random number from 1 to
>> 100; I mean, those two things are in one hash). You can have a look at this
>> tool at tools.wmflabs.org/wiki2email/.
>>
>> My question is: Is this okay? Should I add some kind of formal
>> information to the tool? If so, is there some help page? Should I stop with
>> collecting mails at all and use some WMF-maintained service for
>> mass-emailing (mailman at lists.wikimedia.org maybe?) and make the tool
>> to just send an email to the list itself?
>>
>> This question came to my mind before creating, so I do appologize for
>> asking after programming.
>>
>> Best regards,
>> Martin Urbanec
>> --
>> Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

-- 
Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

[Cloud] Collect user's mail

[Martin Urbanec]

Hello,

upon request at Czech Wikipedia's helpdesk I created a tool that sends
notifications about new weekly articles by e-mail together with its first
paragraph and link to it. To accomplish such a thing I need user's email,
so I firstly decided to store it in a database. To prevent this tool from
spamming I of course require its confirmation by accessing an URL with a
random string (MD5 hash of user's email *and* random number from 1 to 100;
I mean, those two things are in one hash). You can have a look at this tool
at tools.wmflabs.org/wiki2email/.

My question is: Is this okay? Should I add some kind of formal information
to the tool? If so, is there some help page? Should I stop with collecting
mails at all and use some WMF-maintained service for mass-emailing (mailman
at lists.wikimedia.org maybe?) and make the tool to just send an email to
the list itself?

This question came to my mind before creating, so I do appologize for
asking after programming.

Best regards,
Martin Urbanec
-- 
Můj kalendář najdete na https://martin.urbanec.cz/calendar.html
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Changing my shell preference

[Martin Urbanec]

At dev.tools.wmflabs.org: command not found. At bastion.wmflabs.org the
command exists, ask for LDAP password but do nothing for both bastion and
dev.tools.wmflabs.org.

Martin

ne 31. 12. 2017 v 16:35 odesílatel Chad Horohoe <choro...@wikimedia.org>
napsal:

> It can be set in LDAP. Does `chsh.ldap` work for you? It'll prompt for
> your LDAP password, I believe.
>
> -Chad
>
>
> On Sun, Dec 31, 2017 at 3:42 AM Martin Urbanec <
> martin.urba...@wikimedia.cz> wrote:
>
>> Hello,
>>
>> your password on tools is your Wikitech password. Anyway, what you tried
>> won't work because it works with /etc/passwd and tools authorize you
>> using LDAP. The only one think that comes to my mind is to add
>> /usr/bin/zsh to the .bashrc file (and probably delete the rest of its
>> content as it won't be needed). But this is a hack, maybe there is some
>> LDAP field that can be used for the same. I have no way how I can examine
>> the LDAP thing because I'm not an admin :).
>>
>> Martin
>>
>> ne 31. 12. 2017 v 5:00 odesílatel Huji Lee <huji.h...@gmail.com> napsal:
>>
>>> Is there a way to change my preferred shell on the Clouds to zsh? When I
>>> try   chsh -s `which zsh`  it asks for a password, which I don't have
>>> of course.
>>>
>>> Thanks,
>>> Huji
>>> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Changing my shell preference

[Martin Urbanec]

Hello,

your password on tools is your Wikitech password. Anyway, what you tried
won't work because it works with /etc/passwd and tools authorize you using
LDAP. The only one think that comes to my mind is to add /usr/bin/zsh to
the .bashrc file (and probably delete the rest of its content as it won't
be needed). But this is a hack, maybe there is some LDAP field that can be
used for the same. I have no way how I can examine the LDAP thing because
I'm not an admin :).

Martin

ne 31. 12. 2017 v 5:00 odesílatel Huji Lee  napsal:

> Is there a way to change my preferred shell on the Clouds to zsh? When I
> try   chsh -s `which zsh`  it asks for a password, which I don't have of
> course.
>
> Thanks,
> Huji
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Tools that need user tables joins

[Martin Urbanec]

You *can* join them, but not by using database logic. You must join them in
your application.

Martin

so 23. 12. 2017 v 19:03 odesílatel Maximilian Doerr <
maximilian.do...@gmail.com> napsal:

> Ah.
>
> Cyberpower678
> English Wikipedia Account Creation Team
> English Wikipedia Administrator
> Global User Renamer
>
> On Dec 23, 2017, at 12:58, John  wrote:
>
> You can create user databases, however those are hosted on a different
> server than the replicas, and we cannot join them.
>
> On Sat, Dec 23, 2017 at 12:55 PM, Maximilian Doerr <
> maximilian.do...@gmail.com> wrote:
>
>> So users can no longer create their own DBs now?  Or am I missing
>> something?
>>
>> Cyberpower678
>> English Wikipedia Account Creation Team
>> English Wikipedia Administrator
>> Global User Renamer
>>
>> On Dec 23, 2017, at 12:44, Martin Domdey  wrote:
>>
>> Hi Maarten!
>>
>> I didn't got it by -announce, but I need it too in close future.
>>
>> Cheers,
>> Martin ...
>>
>>
>> --
>> Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail
>> gesendet.
>> Am 23.12.2017, 14:29, Maarten Dammers  schrieb:
>>>
>>> Hi everyone,
>>>
>>> In the new database setup user databases are no longer possible on the
>>> same servers as where the production databases are. I noticed on
>>> https://phabricator.wikimedia.org/T142807 Daniel saying "Death blow for
>>> GHEL coordinate extraction and WikiMiniAtlas." and on
>>> https://phabricator.wikimedia.org/T183066 several tools broke down.
>>>
>>> Do we have an overview of tools that are now broken? Did the database
>>> admins actually contact the tool maintainers about the loss of
>>> functionality or was this just send to the -announce list?
>>>
>>> Maarten
>>>
>>>
>>>
>>> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>>
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud