Re: Future of fedora-packages

[Clement Verna]

On Wed, 27 Feb 2019 at 22:11, Jim Perrin  wrote:
>
> How much heresy is involved in us using Amazon's elasticsearch service
> for this, so that we don't have yet-another-thing to maintain?

If we want to proceed with elasticsearch I think that this is indeed a
good way forward. What would it take to use our current AWS account to
setup a test cluster ?

>
> On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> > On Tue, 26 Feb 2019 at 14:39, Clement Verna  
> > wrote:
> >>
> >> Hi all,
> >>
> >> fedora-packages [0] code base is showing its age. The code base and
> >> the technology stack  (Turbogears2 [1] web framework and the Moksha
> >> [2] middleware) is currently not ready for Python3 and I am not
> >> planning to do the work required to make it Python3 compatible, so the
> >> application will stop working when Fedora 29 is EOL.
> >>
> >> In order to keep the service running, I have started a Proof Of
> >> Concept (fedora-search [3]) to replace the backend of the application.
> >> Fedora-search would be a REST API service offering full test search
> >> API. Such a service would then be available for other application to
> >> use, fedora-packages would then become a frontend only application
> >> using the service provided by fedora-search.
> >>
> >> While the POC shows that this is a viable solution, I don't think that
> >> we should be proceeding that way, for the simple reason that this add
> >> yet another code base to maintain, I think we should use this
> >> opportunity to consider using Elasticsearch instead of maintaining our
> >> own "search engine".
> >>
> >
> > The main issues to getting elasticsearch working in the past was the 
> > following:
> >
> > 1 The number of systems needed to make it work. There is a large
> > difference from their 'proof-of-concept see how great this is' to 'ok
> > you want to do anything with load' setups in everything from storage
> > to number of search nodes to network speeds. [The number of hardware
> > for the data we have was to start with 5-8 dedicated Dell systems,
> > some amount of shared fast storage, and N virtual machines with a
> > 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> > into the cloud.. because the feed it from PHX2 to the cloud is
> > expensive.]
> >
> > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > that all packages needed to be packaged in Fedora and follow Fedora
> > packaging rules. [This one has been relaxed.]
> >
> > 3. Running of elasticsearch was a large service in itself. It doesn't
> > take care of itself and we would need one or more people who know it
> > well to keep it running. [This goes down the ladder.. the logstash
> > backends are also full services.. ] Most of that was written in Java
> > which no one on the team at the time had good experiences with.
> >
> > 4. A kibana/elasticsearch query expert. Just like any database, most
> > of the queries you can make are the worse kind. They will take a lot
> > more CPU/memory/time than they should making just grepping for data
> > faster.
> >
> > However that is 3-5 years ago.. so a lot has changed since then.
> >
> >
> >> I think that Elasticsearch offers quite a few advantages :
> >>   - Powerful Query language
> >>   - Python bindings
> >>   - Javascript bindings
> >>   - Can be deployed in our infrastructure or used as a service
> >>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >>
> >> So what is the general feeling about using Elasticsearch in our
> >> infrastructure ? Should we look at deploying a cluster in our infra /
> >> Should we approach the Council to see if we can get founding to have
> >> this service hosted by Elastic ?
> >>
> >> Thanks
> >> Clément
> >>
> >> [0] - https://apps.fedoraproject.org/packages/
> >> [1] - http://www.turbogears.org/
> >> [2] - https://mokshaproject.github.io/mokshaproject.net/
> >> [3] - https://github.com/fedora-infra/fedora-search
> >> ___
> >> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> >> To unsubscribe send an email to 
> >> infrastructure-le...@lists.fedoraproject.org
> >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives: 
> >> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> >
> >
> >
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To

Re: RFR: Message-Tagging-Service

[Chenxiong Qi]

On Thursday, February 28, 2019 1:24:04 PM CST Chenxiong Qi wrote:
> On Tuesday, February 26, 2019 3:32:01 AM CST Mikolaj Izdebski wrote:
> > On Thu, Feb 21, 2019 at 6:42 AM Chenxiong Qi  wrote:
> > > This mail is for a new micro-service called Message-Tagging-Service (aka
> > > MTS). It serves to tag module build triggered by specific MBS event.
> > > More detailed information is provided inside RFR ticket[1].
> > 
> > Thanks for working on this. In the ticket I agreed to be a sponsor for
> > this
> > RFR.
> > 
> > > MTS works with a series of predefined rules to see if a module build
> > > should be tagged with one or more tags. There is requirement coming from
> > > module maintainers to ensure a module build is tagged into correct
> > > platforms to fulfill the dependencies of module metadata. Comment[2] has
> > > a specific use case for that.
> > 
> > As a packager and module maintainer I agree that currently there are
> > problems with tagging modules into appropriate tags. From what I heard
> > there are no plans for MBS to fix this and we are expected to use MTS
> > instead.
> > 
> > > So far, MTS has been containerized and deployed in internal. The image
> > > is available from quay.io[3]. We would love to run MTS in Fedora as well
> > > in order to make it easier to manage module build tag for module
> > > maintainers and rel-eng.
> > 
> > I believe that using containers is allowed and expected these days and
> > that the part of RFR process that relates to having the software
> > packaged for EPEL 7 can be skipped.
> > 
> > > If anything is missed for this mail thread, please point out. Questions
> > > welcome! Thanks for your time.
> > 
> > I have a couple of questions:
> > 
> > 1. As I understand, MTS is driven by a configuration file
> > (mts-rules.yaml) that specifies which modules should be tagged with
> > which Koji tags. Where is this configuration going to be stored?
> > Upstream image on quay.io? Fedora ansible.git? A different git
> > repository?
> 
> Technically, the rule file could be anywhere that is accessible by a HTTP
> GET operation to get the content. In practice to deploy MTS to Fedora, from
> my point of view, it would be good for rule maintainers to use a git
> repository so that they can review every changes to the rules.
> 
> @infra and @rel-eng guys, which way do you prefer to maintain the rule file,
> and what is your opinion of which git repository should be used for storing
> the rule file?
> 
> > 2. Who is going to maintain the above rules configuration? MTS
> > maintainers listed in the ticket? Release engineering?
> 
> I have the same question actually. My understand of "Maintainership
> contacts" is just for the service maintenance. I think rel-eng could be
> able to determine which tag(s) should be applied to a specific module
> build. Hopefully, rel-eng could help to maintain the content of rule file.
> @rel-eng, what do you think?
> 
> > 3. MTS requires a Koji user (and corresponding Kerberos keytab) with
> > no special permissions, which it would use to tag module builds. Koji
> > users are managed by release engineering. Does release engineering
> > agree to have MTS used in Fedora? I think it would be good to open a
> > releng ticket for them to explicitly agree on the proposal.
> 
> I wasn't aware of this yet. I'll file an ticket to ask for agreement.

Ticket is created https://pagure.io/releng/issue/8176

> 
> > 4. Do the people listed as MTS maintainers agree to sign-up for its
> > maintenance? It would be good if they at least acknowledged this in
> > the ticket.
> 
> @Luzi @Valerij Can you please ack in the ticket?
> 
> > --
> > Mikolaj
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to
> > infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct:
> > https://getfedora.org/code-of-conduct.html List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorap
> > r
> > oject.org



___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: RFR: Message-Tagging-Service

[Chenxiong Qi]

On Tuesday, February 26, 2019 3:32:01 AM CST Mikolaj Izdebski wrote:
> On Thu, Feb 21, 2019 at 6:42 AM Chenxiong Qi  wrote:
> 
> > This mail is for a new micro-service called Message-Tagging-Service (aka
> > MTS). It serves to tag module build triggered by specific MBS event.
> > More detailed information is provided inside RFR ticket[1].
> 
> 
> Thanks for working on this. In the ticket I agreed to be a sponsor for this
> RFR.
 
> 
> > MTS works with a series of predefined rules to see if a module build
> > should be tagged with one or more tags. There is requirement coming from
> > module maintainers to ensure a module build is tagged into correct
> > platforms to fulfill the dependencies of module metadata. Comment[2] has
> > a specific use case for that.
> 
> 
> As a packager and module maintainer I agree that currently there are
> problems with tagging modules into appropriate tags. From what I heard
> there are no plans for MBS to fix this and we are expected to use MTS
> instead.
> 
> 
> > So far, MTS has been containerized and deployed in internal. The image
> > is available from quay.io[3]. We would love to run MTS in Fedora as well
> > in order to make it easier to manage module build tag for module
> > maintainers and rel-eng.
> 
> 
> I believe that using containers is allowed and expected these days and
> that the part of RFR process that relates to having the software
> packaged for EPEL 7 can be skipped.
> 
> 
> > If anything is missed for this mail thread, please point out. Questions
> > welcome! Thanks for your time.
> 
> 
> I have a couple of questions:
> 
> 1. As I understand, MTS is driven by a configuration file
> (mts-rules.yaml) that specifies which modules should be tagged with
> which Koji tags. Where is this configuration going to be stored?
> Upstream image on quay.io? Fedora ansible.git? A different git
> repository?

Technically, the rule file could be anywhere that is accessible by a HTTP GET 
operation to get the content. In practice to deploy MTS to Fedora, from my 
point of view, it would be good for rule maintainers to use a git repository 
so that they can review every changes to the rules.

@infra and @rel-eng guys, which way do you prefer to maintain the rule file, 
and what is your opinion of which git repository should be used for storing 
the rule file?

> 
> 2. Who is going to maintain the above rules configuration? MTS
> maintainers listed in the ticket? Release engineering?

I have the same question actually. My understand of "Maintainership contacts" 
is just for the service maintenance. I think rel-eng could be able to 
determine which tag(s) should be applied to a specific module build. 
Hopefully, rel-eng could help to maintain the content of rule file. @rel-eng, 
what do you think?

> 
> 3. MTS requires a Koji user (and corresponding Kerberos keytab) with
> no special permissions, which it would use to tag module builds. Koji
> users are managed by release engineering. Does release engineering
> agree to have MTS used in Fedora? I think it would be good to open a
> releng ticket for them to explicitly agree on the proposal.

I wasn't aware of this yet. I'll file an ticket to ask for agreement.

> 
> 4. Do the people listed as MTS maintainers agree to sign-up for its
> maintenance? It would be good if they at least acknowledged this in
> the ticket.

@Luzi @Valerij Can you please ack in the ticket?

> 
> --
> Mikolaj
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List
> Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
> Archives:
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr
> oject.org



___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Antonette Caldwell]

I would like to take on maintaining the package, however, I do not have 
adequate experience in creating and maintaining as such. I have looked at the 
rpm packaging guidelines, and I can see what to do, but actually doing it is a 
little bit different. I understand everyone has different methods in completing 
the same outcome, which is creating and maintaining packages.

I am learning, however, within a mock environment, but I do not sufficient 
permissions to push the package anyway. If it is possible to go forward with 
Elasticsearch, I would be interested in maintaining but I would need a 
co-maintainer for assistance.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Todd Zullinger]

Neal Gompa wrote:
> On Wed, Feb 27, 2019 at 5:56 PM Stephen John Smoogen  wrote:
>>
>> On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski  wrote:
>>>
>>> On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  
>>> wrote:
 2. Packaging of elasticsearch was a mess. At the time we had rules
 that all packages needed to be packaged in Fedora and follow Fedora
 packaging rules. [This one has been relaxed.]
>>>
>>> I just want to point out that Elasticsearch has been packaged [1] in
>>> Fedora for more than 4 years.
>>>
>>> https://src.fedoraproject.org/rpms/elasticsearch
>>
>> The version I am seeing there is 1.7.5 and the version on github is
>> 6.6.1 .. i didn't know if that was a 'dont even think about using
>> that'
> 
> Upstream skipped a bunch of versions. They went from 1.x to 2.x to 5.x
> and now 6.x with 7.x in development.
> 
> I forgot the exact reason for this, but it had something to do with
> aligning the versions across all of Elastic's software.

Though it's still worth noting that 1.7.5 has been EOL for 2
years (2017-01-16, per https://www.elastic.co/support/eol).
There's around 15 CVE's since that release (whether 1.7.x
is vulnerable to any/all of them is another matter).

It doesn't seem like the Fedora packages are being actively
maintained.  There's been no substanative commits in the
past 3 years.

-- 
Todd


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Stephen John Smoogen]

On Wed, 27 Feb 2019 at 16:05, Jim Perrin  wrote:
>
> How much heresy is involved in us using Amazon's elasticsearch service
> for this, so that we don't have yet-another-thing to maintain?
>

I was wondering how much data are we looking to shove there, does that
data need to be 'protected', and how fast do we need it to be for us
to talk back and forth to the cloud. The heresy side I don't have any
say in..


> On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> > On Tue, 26 Feb 2019 at 14:39, Clement Verna  
> > wrote:
> >>
> >> Hi all,
> >>
> >> fedora-packages [0] code base is showing its age. The code base and
> >> the technology stack  (Turbogears2 [1] web framework and the Moksha
> >> [2] middleware) is currently not ready for Python3 and I am not
> >> planning to do the work required to make it Python3 compatible, so the
> >> application will stop working when Fedora 29 is EOL.
> >>
> >> In order to keep the service running, I have started a Proof Of
> >> Concept (fedora-search [3]) to replace the backend of the application.
> >> Fedora-search would be a REST API service offering full test search
> >> API. Such a service would then be available for other application to
> >> use, fedora-packages would then become a frontend only application
> >> using the service provided by fedora-search.
> >>
> >> While the POC shows that this is a viable solution, I don't think that
> >> we should be proceeding that way, for the simple reason that this add
> >> yet another code base to maintain, I think we should use this
> >> opportunity to consider using Elasticsearch instead of maintaining our
> >> own "search engine".
> >>
> >
> > The main issues to getting elasticsearch working in the past was the 
> > following:
> >
> > 1 The number of systems needed to make it work. There is a large
> > difference from their 'proof-of-concept see how great this is' to 'ok
> > you want to do anything with load' setups in everything from storage
> > to number of search nodes to network speeds. [The number of hardware
> > for the data we have was to start with 5-8 dedicated Dell systems,
> > some amount of shared fast storage, and N virtual machines with a
> > 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> > into the cloud.. because the feed it from PHX2 to the cloud is
> > expensive.]
> >
> > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > that all packages needed to be packaged in Fedora and follow Fedora
> > packaging rules. [This one has been relaxed.]
> >
> > 3. Running of elasticsearch was a large service in itself. It doesn't
> > take care of itself and we would need one or more people who know it
> > well to keep it running. [This goes down the ladder.. the logstash
> > backends are also full services.. ] Most of that was written in Java
> > which no one on the team at the time had good experiences with.
> >
> > 4. A kibana/elasticsearch query expert. Just like any database, most
> > of the queries you can make are the worse kind. They will take a lot
> > more CPU/memory/time than they should making just grepping for data
> > faster.
> >
> > However that is 3-5 years ago.. so a lot has changed since then.
> >
> >
> >> I think that Elasticsearch offers quite a few advantages :
> >>   - Powerful Query language
> >>   - Python bindings
> >>   - Javascript bindings
> >>   - Can be deployed in our infrastructure or used as a service
> >>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >>
> >> So what is the general feeling about using Elasticsearch in our
> >> infrastructure ? Should we look at deploying a cluster in our infra /
> >> Should we approach the Council to see if we can get founding to have
> >> this service hosted by Elastic ?
> >>
> >> Thanks
> >> Clément
> >>
> >> [0] - https://apps.fedoraproject.org/packages/
> >> [1] - http://www.turbogears.org/
> >> [2] - https://mokshaproject.github.io/mokshaproject.net/
> >> [3] - https://github.com/fedora-infra/fedora-search
> >> ___
> >> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> >> To unsubscribe send an email to 
> >> infrastructure-le...@lists.fedoraproject.org
> >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives: 
> >> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> >
> >
> >
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___

Re: Future of fedora-packages

[Neal Gompa]

On Wed, Feb 27, 2019 at 5:56 PM Stephen John Smoogen  wrote:
>
> On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski  wrote:
> >
> > On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  
> > wrote:
> > > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > > that all packages needed to be packaged in Fedora and follow Fedora
> > > packaging rules. [This one has been relaxed.]
> >
> > I just want to point out that Elasticsearch has been packaged [1] in
> > Fedora for more than 4 years.
> >
> > https://src.fedoraproject.org/rpms/elasticsearch
>
> The version I am seeing there is 1.7.5 and the version on github is
> 6.6.1 .. i didn't know if that was a 'dont even think about using
> that'
>

Upstream skipped a bunch of versions. They went from 1.x to 2.x to 5.x
and now 6.x with 7.x in development.

I forgot the exact reason for this, but it had something to do with
aligning the versions across all of Elastic's software.



-- 
真実はいつも一つ！/ Always, there's only one truth!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Stephen John Smoogen]

On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski  wrote:
>
> On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  wrote:
> > 2. Packaging of elasticsearch was a mess. At the time we had rules
> > that all packages needed to be packaged in Fedora and follow Fedora
> > packaging rules. [This one has been relaxed.]
>
> I just want to point out that Elasticsearch has been packaged [1] in
> Fedora for more than 4 years.
>
> https://src.fedoraproject.org/rpms/elasticsearch

The version I am seeing there is 1.7.5 and the version on github is
6.6.1 .. i didn't know if that was a 'dont even think about using
that'




-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Meeting Agenda for 2019-02-28 Infrastructure meeting

[Stephen John Smoogen]

= Preamble =
The infrastructure team will be having its weekly meeting tomorrow,
2019-02-27 at 15:00 UTC in #fedora-meeting-1 on the freenode network.

We have a gobby document at
https://infinote.fedoraproject.org/cgit/infinote/tree/fedora-infrastructure-meeting-next
which can be edited for the agenda (see: https://fedoraproject.org/wiki/Gobby )

Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.

If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.

= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.

= Meeting start stuff =

#startmeeting Infrastructure (2019-02-27)
#meetingname infrastructure
#topic aloha
#chair nirik pingou puiterwijk relrod smooge tflink threebean cverna
mkonecny mizdebsk

= Let new people say hello =

#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
#info Getting Started Guide:
https://fedoraproject.org/wiki/Infrastructure/GettingStarted


= Status / Information / Trivia / Announcements =

(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info  - your name)

#topic announcements and information
#info nirik will have sparse hours due to house move
#info Mass update/reboots planned for 2019-02-26 -> 2019-02-28
#info Beta Freeze Begins 2019-03-05
#info Pagure 3.2.0/3.2.1 was deployed to stg/prod/pkgs.
#info Taskotron Staging re-deployed on F29, mostly working
#info

= Things we should discuss =

We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)

#topic Oncall
#info smooge is on call from 2019-02-14 -> 2019-02-21
#info puiterwijk is on call from 2019-02-21 -> 2019-02-28
#info smooge is on call from 2019-02-28 -> 2019-03-07
#info ?? is on call from 2019-03-07 -> 2019-03-14
#info ?? is on call from 2019-03-14 -> 2019-03-21
#info Summary of last week: (from smooge )

#topic Monitoring discussion
#info https://nagios.fedoraproject.org/nagios
#info Go over existing out items and fix

#topic Tickets discussion
#info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket


Go thru each ticket one by one

#topic Priorities for next week?
#info please put tickets needing to be focused on here


#topic Discuss: Is the Fedora pastebin still useful? - relrod
#info how many users are using it?
#info should we look at converging with CentOS one so simpler setup?

= Apprentice office hours =

#topic Apprentice Open office minutes
#info A time where apprentices may ask for help or look at problems.

Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.

= Learn about some application or setup in infrastructure =

(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)

#info

= Meeting end stuff =

#topic Open Floor

#endmeeting

-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [ansible] [distgit/pagure] Drop --autoreload from our systemd service file

[Todd Zullinger]

Hi,

Pierre-YvesChibon wrote:
> [distgit/pagure] Drop --autoreload from our systemd service file

I noticed a few recent commits using this bracket prefix
style and I wanted to give a heads-up about how this can
cause problems, in case it's not well-known.

(If I've mentioned it before, please forgive my memory
lapse.)

Using brackets to prefix the commit message will cause grief
for any patches which are applied via 'git am', as the
contents within the brackets will be stripped.

Even after the ansible repo is available via pagure, there
are likely still times when someone will attach a patch to a
bug or send it with git send-email.

If this [prefix/area] convention is more widely adopted in
the ansible repo, it's worth keeping in mind the potential
for surprise (and subtle loss of commit message details) it
will likely cause down the road.

The prefix stripping can be avoided by passing the
'--keep-non-patch' option to git am.  However, there's no
config variable to allow that to be easily set per repo, so
it'd be very easy to forget.

This has been discussed on the git list in the past.  One of
the more recent discussions was from 2015:

https://public-inbox.org/git/5655d3da.1050...@informatik.uni-hamburg.de/T/#u

I had a recent bug report filed about this as well:

https://bugzilla.redhat.com/show_bug.cgi?id=1671640

(as evidence that it surprises others still).

Hope this is helpful,

-- 
Todd


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: cloud retirement

[Peter Robinson]

> Hey everyone.
>
> As you know, we currently have a RHOSP5 ancient cloud. After a bunch of
> work last year, we got a RHOSP13 cloud up and mostly working, but it was
> a ton of work. After hearing from the Fedora Council and our various
> management chains we determined that it wouldn't really be a good use of
> our time moving forward to keep maintaining a OpenStack cloud.
>
> We have not yet determined what we want to do with the hardware that we
> had allocated to this, but are weighing our options. We may want to
> setup OpenShift bare nodes so we can do kubevirt, we may want to just
> setup a normal virthost setup managed by ansible.
>
> For the items currently in our cloud, we will be looking at options for
> them, we are definitely not shutting things off until we have plans in
> place.
>
> Happy to answer any questions and will make sure everything is properly
> migrated.

Do we have a documented list of services running on that cloud instance?

Do we have a timeline?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Jim Perrin]

How much heresy is involved in us using Amazon's elasticsearch service
for this, so that we don't have yet-another-thing to maintain?

On 2/27/19 4:19 AM, Stephen John Smoogen wrote:
> On Tue, 26 Feb 2019 at 14:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
> 
> The main issues to getting elasticsearch working in the past was the 
> following:
> 
> 1 The number of systems needed to make it work. There is a large
> difference from their 'proof-of-concept see how great this is' to 'ok
> you want to do anything with load' setups in everything from storage
> to number of search nodes to network speeds. [The number of hardware
> for the data we have was to start with 5-8 dedicated Dell systems,
> some amount of shared fast storage, and N virtual machines with a
> 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> into the cloud.. because the feed it from PHX2 to the cloud is
> expensive.]
> 
> 2. Packaging of elasticsearch was a mess. At the time we had rules
> that all packages needed to be packaged in Fedora and follow Fedora
> packaging rules. [This one has been relaxed.]
> 
> 3. Running of elasticsearch was a large service in itself. It doesn't
> take care of itself and we would need one or more people who know it
> well to keep it running. [This goes down the ladder.. the logstash
> backends are also full services.. ] Most of that was written in Java
> which no one on the team at the time had good experiences with.
> 
> 4. A kibana/elasticsearch query expert. Just like any database, most
> of the queries you can make are the worse kind. They will take a lot
> more CPU/memory/time than they should making just grepping for data
> faster.
> 
> However that is 3-5 years ago.. so a lot has changed since then.
> 
> 
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>>
>> [0] - https://apps.fedoraproject.org/packages/
>> [1] - http://www.turbogears.org/
>> [2] - https://mokshaproject.github.io/mokshaproject.net/
>> [3] - https://github.com/fedora-infra/fedora-search
>> ___
>> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
>> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> 
> 
> 
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Antonette Caldwell]

I looked at the version of Elasticsearch via sources and spec file, and I know 
that Elasticsearch is on 6.0 right now. It is true that we do need an expert in 
elasticsearch in order to move forward on using elasticsearch as an api if 
agreed on using elasticsearch.

I am currently working on Elasticsearch on another project and learning more 
about Elasticsearch, but this is my first time actually using it. Are there any 
plans to upgrade Elasticsearch to an updated version?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Mikolaj Izdebski]

On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen  wrote:
> 2. Packaging of elasticsearch was a mess. At the time we had rules
> that all packages needed to be packaged in Fedora and follow Fedora
> packaging rules. [This one has been relaxed.]

I just want to point out that Elasticsearch has been packaged [1] in
Fedora for more than 4 years.

https://src.fedoraproject.org/rpms/elasticsearch

--
Mikolaj Izdebski
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Wed, 27 Feb 2019 at 13:27, Stephen John Smoogen  wrote:
>
> On Tue, 26 Feb 2019 at 14:39, Clement Verna  wrote:
> >
> > Hi all,
> >
> > fedora-packages [0] code base is showing its age. The code base and
> > the technology stack  (Turbogears2 [1] web framework and the Moksha
> > [2] middleware) is currently not ready for Python3 and I am not
> > planning to do the work required to make it Python3 compatible, so the
> > application will stop working when Fedora 29 is EOL.
> >
> > In order to keep the service running, I have started a Proof Of
> > Concept (fedora-search [3]) to replace the backend of the application.
> > Fedora-search would be a REST API service offering full test search
> > API. Such a service would then be available for other application to
> > use, fedora-packages would then become a frontend only application
> > using the service provided by fedora-search.
> >
> > While the POC shows that this is a viable solution, I don't think that
> > we should be proceeding that way, for the simple reason that this add
> > yet another code base to maintain, I think we should use this
> > opportunity to consider using Elasticsearch instead of maintaining our
> > own "search engine".
> >
>
> The main issues to getting elasticsearch working in the past was the 
> following:
>
> 1 The number of systems needed to make it work. There is a large
> difference from their 'proof-of-concept see how great this is' to 'ok
> you want to do anything with load' setups in everything from storage
> to number of search nodes to network speeds. [The number of hardware
> for the data we have was to start with 5-8 dedicated Dell systems,
> some amount of shared fast storage, and N virtual machines with a
> 10-40GB backbone.. or throwing all of Fedora Infrastructure at once
> into the cloud.. because the feed it from PHX2 to the cloud is
> expensive.]
>
> 2. Packaging of elasticsearch was a mess. At the time we had rules
> that all packages needed to be packaged in Fedora and follow Fedora
> packaging rules. [This one has been relaxed.]
>
> 3. Running of elasticsearch was a large service in itself. It doesn't
> take care of itself and we would need one or more people who know it
> well to keep it running. [This goes down the ladder.. the logstash
> backends are also full services.. ] Most of that was written in Java
> which no one on the team at the time had good experiences with.
>
> 4. A kibana/elasticsearch query expert. Just like any database, most
> of the queries you can make are the worse kind. They will take a lot
> more CPU/memory/time than they should making just grepping for data
> faster.
>
> However that is 3-5 years ago.. so a lot has changed since then.

Thanks smooge for feeling my knowledge gap on what was looked at
previously, I feel that a lot of the issues would actually be solved
if we were to outsource the service and have someone else maintain the
cluster for us. If that's the case I would be happy to open a council
ticket to start the conversation about this possibility .

>
>
> > I think that Elasticsearch offers quite a few advantages :
> >   - Powerful Query language
> >   - Python bindings
> >   - Javascript bindings
> >   - Can be deployed in our infrastructure or used as a service
> >   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> >
> > So what is the general feeling about using Elasticsearch in our
> > infrastructure ? Should we look at deploying a cluster in our infra /
> > Should we approach the Council to see if we can get founding to have
> > this service hosted by Elastic ?
> >
> > Thanks
> > Clément
> >
> > [0] - https://apps.fedoraproject.org/packages/
> > [1] - http://www.turbogears.org/
> > [2] - https://mokshaproject.github.io/mokshaproject.net/
> > [3] - https://github.com/fedora-infra/fedora-search
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
>
>
> --
> Stephen J Smoogen.
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.

Re: External access to the AMQP broker

[Clement Verna]

On Wed, 27 Feb 2019 at 18:27, Aurelien Bompard
 wrote:
>
> I'm assuming you're considering the solution where we have a single
> broker and we make it publicly accessible (option 1).
>
> > how easy would it be to turn off the possibility for external
> > publisher to flood the broker ?
>
> External clients won't publish anything, they'll be read-only (with a
> few exceptions like the CentOS CI folks). However they can create a
> huge amount of queues, subscribe to everything and never consume
> anything.
> We can mitigate that by setting up another vhost (in the cluster we
> already have) for external clients, limit the number of queues on that
> vhost, and enforce a time to live on messages in the queues. It'll
> require some fine tuning, though, and external clients will still be
> able to DoS other external clients if we don't do authentication
> (option A).

> I value option 2 (separate broker) higher than option 1 (same broker)
> because I'm not entirely sure those limits can prevent any kind of DoS
> on the broker. Attackers are creative. It's easier to make sure the
> resources used by a 2nd cluster don't impact the resources of the 1st
> cluster.
>
> > Can we configure the queues that are critical to have higher
> > priority to the external ones ?
>
> Yes, by  using a different vhost for internal (and CentOS) stuff and
> external stuff, and replicating messages from the internal to the
> external vhost.
>
> >  If we have on public broker with authentication can we easily kill the 
> > accounts that are flooding us ?
>
> Yes, that's the main advantage of option B.
>
> > What are the consequences of the service been down ? What is an
> > acceptable down time 1 min, 1h , 1 day , 1 week, 1 month ?
>
> I would say that the internal messaging service needs a high
> availability, while the SLA for the external service can be lower.
> That's also a reason for me prefering option 2.
>
> I hope that clarifies a bit.

Yes it does thanks for the answers :-), my overall feeling is that the
risk of DoS should be one of the factor we take into account to make
the decision but we should also consider how easy is it to use, how
easy is it to maintain, how much effort is it to setup. I feel that
are focusing on the risk of DoS as the main factor to favour one
option against the other and I am not sure this is right but that 's
my personal feeling and I am happy to be wrong on that.

On the SLA I really think that in the case of DoS attack we would not
have much trouble communicating with the community that we are facing
an attack and the service will be down or deprecated for X days.
Overall I think we should start being OK with taking the risk to have
our services down for multiple hours, days, ... if that allow us to
save on the daily maintenance burden.

Again just my 2 cents on the subject, so feel free to ignore it :-)

>
> Aurélien
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: External access to the AMQP broker

[Aurelien Bompard]

I'm assuming you're considering the solution where we have a single
broker and we make it publicly accessible (option 1).

> how easy would it be to turn off the possibility for external
> publisher to flood the broker ?

External clients won't publish anything, they'll be read-only (with a
few exceptions like the CentOS CI folks). However they can create a
huge amount of queues, subscribe to everything and never consume
anything.
We can mitigate that by setting up another vhost (in the cluster we
already have) for external clients, limit the number of queues on that
vhost, and enforce a time to live on messages in the queues. It'll
require some fine tuning, though, and external clients will still be
able to DoS other external clients if we don't do authentication
(option A).
I value option 2 (separate broker) higher than option 1 (same broker)
because I'm not entirely sure those limits can prevent any kind of DoS
on the broker. Attackers are creative. It's easier to make sure the
resources used by a 2nd cluster don't impact the resources of the 1st
cluster.

> Can we configure the queues that are critical to have higher
> priority to the external ones ?

Yes, by  using a different vhost for internal (and CentOS) stuff and
external stuff, and replicating messages from the internal to the
external vhost.

>  If we have on public broker with authentication can we easily kill the 
> accounts that are flooding us ?

Yes, that's the main advantage of option B.

> What are the consequences of the service been down ? What is an
> acceptable down time 1 min, 1h , 1 day , 1 week, 1 month ?

I would say that the internal messaging service needs a high
availability, while the SLA for the external service can be lower.
That's also a reason for me prefering option 2.

I hope that clarifies a bit.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

[Fedocal] Reminder meeting : Fedora Infrastructure

[smooge]

Dear all,

You are kindly invited to the meeting:
   Fedora Infrastructure on 2019-02-28 from 15:00:00 to 16:00:00 UTC
   At fedora-meetin...@irc.freenode.net

The meeting will be about:
Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day 
before or view it at 
https://infinote.fedoraproject.org/cgit/infinote/tree/fedora-infrastructure-meeting-next


Source: https://apps.fedoraproject.org/calendar/meeting/22/

___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: External access to the AMQP broker

[Stephen John Smoogen]

On Wed, 27 Feb 2019 at 07:57, Clement Verna  wrote:
>

> > I can say that there are quite a few people out there who look for
> > someone uttering "hypothetical DoS" to prove to them one will exist.
> > So now that you have done so.. we should assume we will have one and
> > plan on how to deal with.
>
> I am all for assuming this will happen but I am also all for
> considering what would be the impact and how we could mitigate it. For
> example how easy would it be to turn off the possibility for external
> publisher to flood the broker ? Could this be automated from a nagios
> alert ? Can we configure the queues that are critical to have higher

Usually monitoring is only going to see it way after it has started.
It also may also be on the bus with the new monitoring consuming and
deploying things there [because everyone loves buses.]

There are

> priority to the external ones ?  If we have on public broker with
> authentication can we easily kill the accounts that are flooding us ?
> What are the consequences of the service been down ? What is an
> acceptable down time 1 min, 1h , 1 day , 1 week, 1 month ?
>

These days.. people seem to think 1 minute downtime is way too long
for anything we do. With the faster artifact creation, we are probably
wanting to make sure we are 24x7x52 with large amounts of bus traffic.

I think in analogies so here is how I am looking at it. We have cities
of data and we have buses running through the city. One design has us
having every bus go to every city in the world just in case there is a
passenger which might want to get on. And if someone decides to send a
million busses into our city there isn't anything we can do. The same
with the other cities who we are asking to join our city transit
system. [This may sound incredibly daft, but that has sort of happened
in the past with early transit systems.. when anyone could put a train
on the early train systems... people did.. Another time someone saw
that the rules said a bus system was supposed to arrive at every stop
so they had to send busses around places to even other towns which
delayed things. Modern city congestion is mostly everyone having a car
and dropping it on the road because hey eveyrone else is and no one is
going to notice my car.]

Smart transit designs assume that you want to control traffic for
various reasons at certain points. It can be everything from security
to capacity to latency. I would prefer to have a train system between
us and CentOS and Brno and India versus finding out that we have
timeouts somewhere because a service assumed local bus and the
response from PHX2 to Pune was too long. Does that make sense?


> I have the feeling that answering these questions would help in taking
> an informed decision.
>
> >
> > > Just my 0.02 $ since I have very little knowledge about RabbitMQ ;-)
> > >
> > > >
> > > > Thanks
> > > > Aurélien
> > > > ___
> > > > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > > > To unsubscribe send an email to 
> > > > infrastructure-le...@lists.fedoraproject.org
> > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: 
> > > > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> > > ___
> > > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > > To unsubscribe send an email to 
> > > infrastructure-le...@lists.fedoraproject.org
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: 
> > > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> >
> >
> >
> > --
> > Stephen J Smoogen.
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send

Re: External access to the AMQP broker

[Clement Verna]

On Wed, 27 Feb 2019 at 13:32, Stephen John Smoogen  wrote:
>
> On Wed, 27 Feb 2019 at 05:17, Clement Verna  wrote:
> >
> > On Wed, 27 Feb 2019 at 10:57, Aurelien Bompard
> >  wrote:
> > >
> > > Hey y'all,
> > >
> > > Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a
> > > message broker. The current clusters we have deployed in staging and
> > > prod are only accessible from inside our infrastructure.
> > > There are two needs for an externally accessible broker:
> > > - the CentOS folks, who are outside of our infrastructure, would like
> > > to send messages
> > > - people from the community would like to subscribe to messages and do
> > > things based on them
> > >
> > > We have several options to make that happen.
> > >
> > > 1. Use our existing cluster and expose it to the world
> > > The advantage is we don't maintain another cluster, but the downside
> > > is in the case of a DoS attack we're directly affected. With RabbitMQ
> > > 3.7 there are some limits[0] you can set on vhosts (max connections
> > > and max queues), but we're not yet on 3.7.
> > > [0] https://www.rabbitmq.com/vhosts.html#limits
> >
> > What would it take to get 3.7 ? Could we react (shuting down the
> > queues ? ) to DoS early with monitoring ? What would it take for
> > someone to DoS our cluster, basically how easy would it be ?
> >
>
> I expect that even with 3.7 the limits just make it so we don't shut
> down but find anything relying on messaging aren't working. So if
> gating, bodhi, pkgs relies on messages from fedora-messaging ot do
> tasks.. we are basically not working until the DoS is over.
>
> > >
> > > 2. Use a separate cluster and copy messages over
> > > We could deploy a separate cluster that would get a copy of all
> > > messages, and would be more limited in resources. It truly isolates
> > > infrastructure, so it's better protected against DoS, but it's more
> > > work for sysadmins.
> > >
> > > In both cases, there are several paths we can take as regards to 
> > > authentication.
> > >
> > > A: make a single readonly account for everybody in the community to
> > > use, and a few read-write accounts (with X509 certs) for people who
> > > need to publish, ie CentOS CI. If we choose a separate broker we can
> > > copy those messages back to the main cluster.
> > > The issue here is that everybody in the community will be using the
> > > same account, so it's harder to shut down bad actors. It would also be
> > > theoretically possible for someone to consume from somebody else's
> > > queue (unless people make sure they use UUIDs in queues, I think we
> > > can enforce that but it way have side effects).
> > > However, it enables the same kind of usage that fedmsg provided before.
> > >
> > > B: require authentication with username & password but make it easy to
> > > get accounts. People could require accounts via tickets for example.
> > > It will make it much harder to abuse the service, and we could easily
> > > shut down bad actors. However it's an obviously heavier load on the
> > > people who will handle the tickets and create the accounts.
> > >
> > > My personal preference would be option 2A, so an external broker with
> > > an anonymous read-only account, but all combinations of options
> > > inflict different loads on the sysadmin (on deployment and in the
> > > longer term), so I think it's really up to them.
> > >
> > > What do you guys think?
> >
> > I am generally not a fan of creating more work for hypothetical DoS, I
> > my opinion we should try to mitigate as much as possible to effect of
> > a possible DoS .
> >
>
> I can say that there are quite a few people out there who look for
> someone uttering "hypothetical DoS" to prove to them one will exist.
> So now that you have done so.. we should assume we will have one and
> plan on how to deal with.

I am all for assuming this will happen but I am also all for
considering what would be the impact and how we could mitigate it. For
example how easy would it be to turn off the possibility for external
publisher to flood the broker ? Could this be automated from a nagios
alert ? Can we configure the queues that are critical to have higher
priority to the external ones ?  If we have on public broker with
authentication can we easily kill the accounts that are flooding us ?
What are the consequences of the service been down ? What is an
acceptable down time 1 min, 1h , 1 day , 1 week, 1 month ?

I have the feeling that answering these questions would help in taking
an informed decision.

>
> > Just my 0.02 $ since I have very little knowledge about RabbitMQ ;-)
> >
> > >
> > > Thanks
> > > Aurélien
> > > ___
> > > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > > To unsubscribe send an email to 
> > > infrastructure-le...@lists.fedoraproject.org
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject

Re: External access to the AMQP broker

[Stephen John Smoogen]

On Wed, 27 Feb 2019 at 05:17, Clement Verna  wrote:
>
> On Wed, 27 Feb 2019 at 10:57, Aurelien Bompard
>  wrote:
> >
> > Hey y'all,
> >
> > Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a
> > message broker. The current clusters we have deployed in staging and
> > prod are only accessible from inside our infrastructure.
> > There are two needs for an externally accessible broker:
> > - the CentOS folks, who are outside of our infrastructure, would like
> > to send messages
> > - people from the community would like to subscribe to messages and do
> > things based on them
> >
> > We have several options to make that happen.
> >
> > 1. Use our existing cluster and expose it to the world
> > The advantage is we don't maintain another cluster, but the downside
> > is in the case of a DoS attack we're directly affected. With RabbitMQ
> > 3.7 there are some limits[0] you can set on vhosts (max connections
> > and max queues), but we're not yet on 3.7.
> > [0] https://www.rabbitmq.com/vhosts.html#limits
>
> What would it take to get 3.7 ? Could we react (shuting down the
> queues ? ) to DoS early with monitoring ? What would it take for
> someone to DoS our cluster, basically how easy would it be ?
>

I expect that even with 3.7 the limits just make it so we don't shut
down but find anything relying on messaging aren't working. So if
gating, bodhi, pkgs relies on messages from fedora-messaging ot do
tasks.. we are basically not working until the DoS is over.

> >
> > 2. Use a separate cluster and copy messages over
> > We could deploy a separate cluster that would get a copy of all
> > messages, and would be more limited in resources. It truly isolates
> > infrastructure, so it's better protected against DoS, but it's more
> > work for sysadmins.
> >
> > In both cases, there are several paths we can take as regards to 
> > authentication.
> >
> > A: make a single readonly account for everybody in the community to
> > use, and a few read-write accounts (with X509 certs) for people who
> > need to publish, ie CentOS CI. If we choose a separate broker we can
> > copy those messages back to the main cluster.
> > The issue here is that everybody in the community will be using the
> > same account, so it's harder to shut down bad actors. It would also be
> > theoretically possible for someone to consume from somebody else's
> > queue (unless people make sure they use UUIDs in queues, I think we
> > can enforce that but it way have side effects).
> > However, it enables the same kind of usage that fedmsg provided before.
> >
> > B: require authentication with username & password but make it easy to
> > get accounts. People could require accounts via tickets for example.
> > It will make it much harder to abuse the service, and we could easily
> > shut down bad actors. However it's an obviously heavier load on the
> > people who will handle the tickets and create the accounts.
> >
> > My personal preference would be option 2A, so an external broker with
> > an anonymous read-only account, but all combinations of options
> > inflict different loads on the sysadmin (on deployment and in the
> > longer term), so I think it's really up to them.
> >
> > What do you guys think?
>
> I am generally not a fan of creating more work for hypothetical DoS, I
> my opinion we should try to mitigate as much as possible to effect of
> a possible DoS .
>

I can say that there are quite a few people out there who look for
someone uttering "hypothetical DoS" to prove to them one will exist.
So now that you have done so.. we should assume we will have one and
plan on how to deal with.

> Just my 0.02 $ since I have very little knowledge about RabbitMQ ;-)
>
> >
> > Thanks
> > Aurélien
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_li

Re: Future of fedora-packages

[Stephen John Smoogen]

On Tue, 26 Feb 2019 at 14:39, Clement Verna  wrote:
>
> Hi all,
>
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
>
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
>
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
>

The main issues to getting elasticsearch working in the past was the following:

1 The number of systems needed to make it work. There is a large
difference from their 'proof-of-concept see how great this is' to 'ok
you want to do anything with load' setups in everything from storage
to number of search nodes to network speeds. [The number of hardware
for the data we have was to start with 5-8 dedicated Dell systems,
some amount of shared fast storage, and N virtual machines with a
10-40GB backbone.. or throwing all of Fedora Infrastructure at once
into the cloud.. because the feed it from PHX2 to the cloud is
expensive.]

2. Packaging of elasticsearch was a mess. At the time we had rules
that all packages needed to be packaged in Fedora and follow Fedora
packaging rules. [This one has been relaxed.]

3. Running of elasticsearch was a large service in itself. It doesn't
take care of itself and we would need one or more people who know it
well to keep it running. [This goes down the ladder.. the logstash
backends are also full services.. ] Most of that was written in Java
which no one on the team at the time had good experiences with.

4. A kibana/elasticsearch query expert. Just like any database, most
of the queries you can make are the worse kind. They will take a lot
more CPU/memory/time than they should making just grepping for data
faster.

However that is 3-5 years ago.. so a lot has changed since then.


> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?
>
> Thanks
> Clément
>
> [0] - https://apps.fedoraproject.org/packages/
> [1] - http://www.turbogears.org/
> [2] - https://mokshaproject.github.io/mokshaproject.net/
> [3] - https://github.com/fedora-infra/fedora-search
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



-- 
Stephen J Smoogen.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: External access to the AMQP broker

[Clement Verna]

On Wed, 27 Feb 2019 at 10:57, Aurelien Bompard
 wrote:
>
> Hey y'all,
>
> Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a
> message broker. The current clusters we have deployed in staging and
> prod are only accessible from inside our infrastructure.
> There are two needs for an externally accessible broker:
> - the CentOS folks, who are outside of our infrastructure, would like
> to send messages
> - people from the community would like to subscribe to messages and do
> things based on them
>
> We have several options to make that happen.
>
> 1. Use our existing cluster and expose it to the world
> The advantage is we don't maintain another cluster, but the downside
> is in the case of a DoS attack we're directly affected. With RabbitMQ
> 3.7 there are some limits[0] you can set on vhosts (max connections
> and max queues), but we're not yet on 3.7.
> [0] https://www.rabbitmq.com/vhosts.html#limits

What would it take to get 3.7 ? Could we react (shuting down the
queues ? ) to DoS early with monitoring ? What would it take for
someone to DoS our cluster, basically how easy would it be ?

>
> 2. Use a separate cluster and copy messages over
> We could deploy a separate cluster that would get a copy of all
> messages, and would be more limited in resources. It truly isolates
> infrastructure, so it's better protected against DoS, but it's more
> work for sysadmins.
>
> In both cases, there are several paths we can take as regards to 
> authentication.
>
> A: make a single readonly account for everybody in the community to
> use, and a few read-write accounts (with X509 certs) for people who
> need to publish, ie CentOS CI. If we choose a separate broker we can
> copy those messages back to the main cluster.
> The issue here is that everybody in the community will be using the
> same account, so it's harder to shut down bad actors. It would also be
> theoretically possible for someone to consume from somebody else's
> queue (unless people make sure they use UUIDs in queues, I think we
> can enforce that but it way have side effects).
> However, it enables the same kind of usage that fedmsg provided before.
>
> B: require authentication with username & password but make it easy to
> get accounts. People could require accounts via tickets for example.
> It will make it much harder to abuse the service, and we could easily
> shut down bad actors. However it's an obviously heavier load on the
> people who will handle the tickets and create the accounts.
>
> My personal preference would be option 2A, so an external broker with
> an anonymous read-only account, but all combinations of options
> inflict different loads on the sysadmin (on deployment and in the
> longer term), so I think it's really up to them.
>
> What do you guys think?

I am generally not a fan of creating more work for hypothetical DoS, I
my opinion we should try to mitigate as much as possible to effect of
a possible DoS .

Just my 0.02 $ since I have very little knowledge about RabbitMQ ;-)

>
> Thanks
> Aurélien
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Clement Verna]

On Wed, 27 Feb 2019 at 10:27, Vít Ondruch  wrote:
>
>
> Dne 26. 02. 19 v 23:42 Ryan Lerch napsal(a):
>
>
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>
>
> Backend of fedora-packages is mdapi, isn't it? What would be the difference?

mdapi is one of the service used to populate the xapian search index.
The fedora-packages backend is providing the API to the xapian
database to perform the full text search. If we were to use
elasticsearch we would still need to have an indexing script but we
could rely on elasticsearch API and query language for the search
capabilities.

>
>
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>
>
> From an information point of view, the package-centric view of 
> Fedora-packages is quite similar to the pagure dist-git. Would it worth 
> investgating merging the front-end functionality of packages (lists of 
> builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora 
> packages front end?
>
>
> +1
>
>
> V.
>
>
>
> —ry
>>
>>
>>
>> [0] - https://apps.fedoraproject.org/packages/
>> [1] - http://www.turbogears.org/
>> [2] - https://mokshaproject.github.io/mokshaproject.net/
>> [3] - https://github.com/fedora-infra/fedora-search
>> ___
>> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
>> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

External access to the AMQP broker

[Aurelien Bompard]

Hey y'all,

Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a
message broker. The current clusters we have deployed in staging and
prod are only accessible from inside our infrastructure.
There are two needs for an externally accessible broker:
- the CentOS folks, who are outside of our infrastructure, would like
to send messages
- people from the community would like to subscribe to messages and do
things based on them

We have several options to make that happen.

1. Use our existing cluster and expose it to the world
The advantage is we don't maintain another cluster, but the downside
is in the case of a DoS attack we're directly affected. With RabbitMQ
3.7 there are some limits[0] you can set on vhosts (max connections
and max queues), but we're not yet on 3.7.
[0] https://www.rabbitmq.com/vhosts.html#limits

2. Use a separate cluster and copy messages over
We could deploy a separate cluster that would get a copy of all
messages, and would be more limited in resources. It truly isolates
infrastructure, so it's better protected against DoS, but it's more
work for sysadmins.

In both cases, there are several paths we can take as regards to authentication.

A: make a single readonly account for everybody in the community to
use, and a few read-write accounts (with X509 certs) for people who
need to publish, ie CentOS CI. If we choose a separate broker we can
copy those messages back to the main cluster.
The issue here is that everybody in the community will be using the
same account, so it's harder to shut down bad actors. It would also be
theoretically possible for someone to consume from somebody else's
queue (unless people make sure they use UUIDs in queues, I think we
can enforce that but it way have side effects).
However, it enables the same kind of usage that fedmsg provided before.

B: require authentication with username & password but make it easy to
get accounts. People could require accounts via tickets for example.
It will make it much harder to abuse the service, and we could easily
shut down bad actors. However it's an obviously heavier load on the
people who will handle the tickets and create the accounts.

My personal preference would be option 2A, so an external broker with
an anonymous read-only account, but all combinations of options
inflict different loads on the sysadmin (on deployment and in the
longer term), so I think it's really up to them.

What do you guys think?

Thanks
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Pierre-Yves Chibon]

On Tue, Feb 26, 2019 at 08:38:30PM +0100, Clement Verna wrote:
> Hi all,
> 
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
> 
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
> 
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
> 
> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
> 
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?

We look at Elasticsearch at one point (I believe Aurélien was even looking at
setting up an instance in our cloud) but it came down to two aspects:
- setting it up is another application to maintain
- an application we have no expertise on

On the other side, fedora-search is:
- setting up another application to maintain
- an application we have expertise on

Asking Council to get founding for a hosted instance would resolve both of
points for Elasticsearch without the time implication of developing it
ourself.
In both case, there will be some time implication of moving to the new system,
so that's not a tie-breaker.


Pierre
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Neal Gompa]

On Tue, Feb 26, 2019 at 5:43 PM Ryan Lerch  wrote:
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  wrote:
>>
>> Hi all,
>>
>> fedora-packages [0] code base is showing its age. The code base and
>> the technology stack  (Turbogears2 [1] web framework and the Moksha
>> [2] middleware) is currently not ready for Python3 and I am not
>> planning to do the work required to make it Python3 compatible, so the
>> application will stop working when Fedora 29 is EOL.
>>
>> In order to keep the service running, I have started a Proof Of
>> Concept (fedora-search [3]) to replace the backend of the application.
>> Fedora-search would be a REST API service offering full test search
>> API. Such a service would then be available for other application to
>> use, fedora-packages would then become a frontend only application
>> using the service provided by fedora-search.
>>
>> While the POC shows that this is a viable solution, I don't think that
>> we should be proceeding that way, for the simple reason that this add
>> yet another code base to maintain, I think we should use this
>> opportunity to consider using Elasticsearch instead of maintaining our
>> own "search engine".
>>
>> I think that Elasticsearch offers quite a few advantages :
>>   - Powerful Query language
>>   - Python bindings
>>   - Javascript bindings
>>   - Can be deployed in our infrastructure or used as a service
>>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>>
>> So what is the general feeling about using Elasticsearch in our
>> infrastructure ? Should we look at deploying a cluster in our infra /
>> Should we approach the Council to see if we can get founding to have
>> this service hosted by Elastic ?
>>
>> Thanks
>> Clément
>
>
> From an information point of view, the package-centric view of 
> Fedora-packages is quite similar to the pagure dist-git. Would it worth 
> investgating merging the front-end functionality of packages (lists of 
> builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora 
> packages front end?
>

How would you propose we'd allow people to search by binary packages
and seeing contents, changelog, and seeing which versions are shipped
in releases?

I don't think it would be a good idea to merge the two views, because
it really doesn't make this any easier. The package search is a
user-focused view into the package collection, and is a helpful
reference (when it works!). I think it still serves a purpose to have
a dedicated frontend.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Future of fedora-packages

[Vít Ondruch]

Dne 26. 02. 19 v 23:42 Ryan Lerch napsal(a):
>
>
> On Wed, 27 Feb 2019 at 05:39, Clement Verna  > wrote:
>
> Hi all,
>
> fedora-packages [0] code base is showing its age. The code base and
> the technology stack  (Turbogears2 [1] web framework and the Moksha
> [2] middleware) is currently not ready for Python3 and I am not
> planning to do the work required to make it Python3 compatible, so the
> application will stop working when Fedora 29 is EOL.
>
> In order to keep the service running, I have started a Proof Of
> Concept (fedora-search [3]) to replace the backend of the application.
>

Backend of fedora-packages is mdapi, isn't it? What would be the difference?


> Fedora-search would be a REST API service offering full test search
> API. Such a service would then be available for other application to
> use, fedora-packages would then become a frontend only application
> using the service provided by fedora-search.
>
> While the POC shows that this is a viable solution, I don't think that
> we should be proceeding that way, for the simple reason that this add
> yet another code base to maintain, I think we should use this
> opportunity to consider using Elasticsearch instead of maintaining our
> own "search engine".
>
> I think that Elasticsearch offers quite a few advantages :
>   - Powerful Query language
>   - Python bindings
>   - Javascript bindings
>   - Can be deployed in our infrastructure or used as a service
>   - Can be useful for other applications ( docs.fp.o, pagure, ??)
>
> So what is the general feeling about using Elasticsearch in our
> infrastructure ? Should we look at deploying a cluster in our infra /
> Should we approach the Council to see if we can get founding to have
> this service hosted by Elastic ?
>
> Thanks
> Clément
>
>
> From an information point of view, the package-centric view of
> Fedora-packages is quite similar to the pagure dist-git. Would it
> worth investgating merging the front-end functionality of packages
> (lists of builds, bugs, updates, etc) into pagure dist-git, and
> retiring the Fedora packages front end?


+1


V.


>
> —ry
>
>
>
> [0] - https://apps.fedoraproject.org/packages/
> [1] - http://www.turbogears.org/
> [2] - https://mokshaproject.github.io/mokshaproject.net/
> [3] - https://github.com/fedora-infra/fedora-search
> ___
> infrastructure mailing list --
> infrastructure@lists.fedoraproject.org
> 
> To unsubscribe send an email to
> infrastructure-le...@lists.fedoraproject.org
> 
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
>
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org