Re: Dependencies on non-default variants

[Craig Treleaven]

> On Apr 2, 2019, at 5:40 PM, Ryan Schmidt  wrote:
> 
> On Mar 25, 2019, at 08:48, Craig Treleaven wrote:
> 
>> On Mar 24, 2019, at 4:05 PM, Mojca Miklavec wrote:
>> 
>>> I now checked the first two random MythTV ports, which basically boils down 
>>> to:
>>> 
>>> require_active_variants qt5-mysql-plugin mariadb55
>>> require_active_variants p${perl5.major}-dbd-mysql mariadb
>>> require_active_variants ${pymodver}-mysql mariadb55
>>> 
>>> out of which only the second one provides the wrong default variant,
>>> and that port doesn't have the maintainer. Neither do any of the mysql
>>> packages have any maintainer at the moment.
>>> 
>>> My first question is: why exactly does p5-dbd-mysql need a different
>>> default variant?
>> 
>> Myth requires a database backend and I’ve chosen mariadb 5.5.  I was pushed 
>> to add variants for mysql and different db versions.  That would have been a 
>> nightmare to support.  The perl dbd-mysql modules have to know where to find 
>> the db socket (AIUI) and therefore need a different variant depending on 
>> which database/version they are connecting to.  I’ve documented the required 
>> variants in the cookbook instructions on the MythTV wiki.
>> 
>> If I change the default variant for p5-dbd-mysql to suit me, I just push the 
>> problem to someone else.  I have no idea how many others are using this port 
>> and depend on it connecting to MySQL 5.7 by default.
> 
> It seems arbitrary and unnecessary for the MythTV ports to be imposing a 
> requirement on MariaDB 5.5. If MythTV works with other versions or forks of 
> MySQL, it should allow the user to choose that. It would not be surprising if 
> the user would need to configure MythTV to tell it what socket path to use, 
> and the MythTV ports might want to give the user some guidance on that, for 
> example in the port notes ("port notes php73-mysql" shows an example of that) 
> or in the wiki page.
> 
This is water long under the bridge.  Over the years, I’ve answered hundreds of 
emails and postings about MythTV on Mac.  I have been the de facto Mac support 
person for the project for a long time.  A significant fraction of the issues 
have dealt with database problems.  While allowing multiple databases would be 
a good theory it would have increased the support load…on me.  I selfishly 
chose to standardize on one database at a particular version.  Truth be told, I 
made the similar choice for a web server (apache).  

> Ideally, all of the ports that support MySQL would have the same set of 
> variants, to allow the user to choose which version of which fork of MySQL to 
> use, and we would decide on the same default MySQL variant to use in all 
> those ports. When Oracle bought MySQL, some Linux distributions switched 
> their default MySQL fork to MariaDB, and I think MacPorts wanted to do that 
> as well. But a problem occurred when MariaDB 10.0 was released and the 
> mariadb-10.0 port was created, which is that we could not decide what the 
> variant name for using that version of MariaDB should be. For all prior 
> versions of MySQL and its forks, the variant name was the same as the port 
> name (e.g. "mysql57") but a dash is not legal in variant names, so 
> "mariadb-10.0" could not be used as a variant name. (MacPorts would interpret 
> that as "enable the variant 'mariadb' and disable the variant '10.0'".) 
> "mariadb10.0" couldn't be used either because periods were not legal in 
> variant names either. And "mariadb100" looks strange; the user likely does 
> not know that they are supposed to infer a period between the two zeroes. As 
> of MacPorts 2.5.0 periods are allowed in variant names now, so the variant 
> name "mariadb10.0" is now legal and someone who cares about MySQL could 
> pursue updating the variants of all the MySQL-using ports.
> 

Strangely enough, I filed enhancement tickets 5 years ago suggesting that 
database variants be harmonized for a couple of ports that you maintain:

https://trac.macports.org/ticket/44484
https://trac.macports.org/ticket/44481

> 
>> More generally, a relatively common issue on MacPorts is X11 versus Quartz. 
>> Say the support library (such as gtk2) defaults to one (like +x11) but your 
>> package really needs the other.  Changing the default messes up others.  
>> BTW, this would be a compelling use of installation statistics.  If we 
>> determined that, say, 80% of gtk2 installs are +quartz, then it would be a 
>> no-brainer to change the default.
> 
> We still recommend that the user should decide whether to use +x11 or +quartz 
> before installing any ports; if the user has already installed ports, the 
> user should uninstall them if switching from +x11 to +quartz or vice-versa. 
> There's currently no functionality in MacPorts base that would automate 
> uninstalling and reinstalling all ports. So I don't see how we could switch 
> the default variants of ports from +x11 to +quartz without breaking existing 
> users' installations.
> 
In my made-up 

Re: Mail delivery of macports-changes disabled for Gmail users

[Ryan Schmidt]

This happened again yesterday.


> On Feb 25, 2019, at 13:53, Rainer Müller wrote:
> 
> Hello,
> 
> if you are using Gmail or a Google hosted mail service for your domain,
> your subscription to the macports-changes mailing list was likely
> disabled on 2019-02-22 by mailman due to "excessive or fatal bounces".
> 
> If you still want to receive the mailing list, but are currently not
> getting the emails, you have to take manual action. Please follow the
> link below to the mailman options. After logging in, check that the
> option "Mail delivery" is Enabled for your account.
> 
> https://lists.macports.org/mailman/options/macports-changes
> 
> 
> The technical cause of this was an email being rejected by Google due to
> a strict DMARC policy for the From address. The way Google handled this
> was correct as this mail was in fact not originating from a source
> whitelisted by the domain owner of the From address.
> 
> Although we configured all our lists on mailman exactly for this reason
> with dmarc_moderation_action="Munge From" [1], we now discovered that
> this setting is an action that only applies to mails going through the
> moderation queue. However, as posting to macports-changes is meant to be
> restricted, legitimate mails are pre-approved. As we have learned now,
> skipping the moderation unfortunately also skips the DMARC munging that
> was supposed to prevent such bounces.
> 
> In order to prevent this from happening again, I would like to no longer
> sent mails with the author's address as From, but use a generic From
> address and merely the author to CC. Unless someone else comes up with a
> better idea, I will look into applying this in the next days.
> 
> Rainer
> 
> [1] https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html

Re: Dependencies on non-default variants

[Ryan Schmidt]

On Mar 25, 2019, at 08:48, Craig Treleaven wrote:

> On Mar 24, 2019, at 4:05 PM, Mojca Miklavec wrote:
> 
>> I now checked the first two random MythTV ports, which basically boils down 
>> to:
>> 
>> require_active_variants qt5-mysql-plugin mariadb55
>> require_active_variants p${perl5.major}-dbd-mysql mariadb
>> require_active_variants ${pymodver}-mysql mariadb55
>> 
>> out of which only the second one provides the wrong default variant,
>> and that port doesn't have the maintainer. Neither do any of the mysql
>> packages have any maintainer at the moment.
>> 
>> My first question is: why exactly does p5-dbd-mysql need a different
>> default variant?
> 
> Myth requires a database backend and I’ve chosen mariadb 5.5.  I was pushed 
> to add variants for mysql and different db versions.  That would have been a 
> nightmare to support.  The perl dbd-mysql modules have to know where to find 
> the db socket (AIUI) and therefore need a different variant depending on 
> which database/version they are connecting to.  I’ve documented the required 
> variants in the cookbook instructions on the MythTV wiki.
> 
> If I change the default variant for p5-dbd-mysql to suit me, I just push the 
> problem to someone else.  I have no idea how many others are using this port 
> and depend on it connecting to MySQL 5.7 by default.

It seems arbitrary and unnecessary for the MythTV ports to be imposing a 
requirement on MariaDB 5.5. If MythTV works with other versions or forks of 
MySQL, it should allow the user to choose that. It would not be surprising if 
the user would need to configure MythTV to tell it what socket path to use, and 
the MythTV ports might want to give the user some guidance on that, for example 
in the port notes ("port notes php73-mysql" shows an example of that) or in the 
wiki page.

Ideally, all of the ports that support MySQL would have the same set of 
variants, to allow the user to choose which version of which fork of MySQL to 
use, and we would decide on the same default MySQL variant to use in all those 
ports. When Oracle bought MySQL, some Linux distributions switched their 
default MySQL fork to MariaDB, and I think MacPorts wanted to do that as well. 
But a problem occurred when MariaDB 10.0 was released and the mariadb-10.0 port 
was created, which is that we could not decide what the variant name for using 
that version of MariaDB should be. For all prior versions of MySQL and its 
forks, the variant name was the same as the port name (e.g. "mysql57") but a 
dash is not legal in variant names, so "mariadb-10.0" could not be used as a 
variant name. (MacPorts would interpret that as "enable the variant 'mariadb' 
and disable the variant '10.0'".) "mariadb10.0" couldn't be used either because 
periods were not legal in variant names either. And "mariadb100" looks strange; 
the user likely does not know that they are supposed to infer a period between 
the two zeroes. As of MacPorts 2.5.0 periods are allowed in variant names now, 
so the variant name "mariadb10.0" is now legal and someone who cares about 
MySQL could pursue updating the variants of all the MySQL-using ports.


> More generally, a relatively common issue on MacPorts is X11 versus Quartz. 
> Say the support library (such as gtk2) defaults to one (like +x11) but your 
> package really needs the other.  Changing the default messes up others.  BTW, 
> this would be a compelling use of installation statistics.  If we determined 
> that, say, 80% of gtk2 installs are +quartz, then it would be a no-brainer to 
> change the default.

We still recommend that the user should decide whether to use +x11 or +quartz 
before installing any ports; if the user has already installed ports, the user 
should uninstall them if switching from +x11 to +quartz or vice-versa. There's 
currently no functionality in MacPorts base that would automate uninstalling 
and reinstalling all ports. So I don't see how we could switch the default 
variants of ports from +x11 to +quartz without breaking existing users' 
installations.


> I believe there is a very, very old ticket on this issue but I can’t find it 
> right now.

Do you mean https://trac.macports.org/ticket/126 ? I'm on record there and 
elsewhere opposing adding support for dependencies on variants. I understand 
how in extremely limited specific cases it could be useful, but I don't see how 
it can work if we consider the problem just a little further.

Currently, mythtv-core.28 requires p5.26-dbd-mysql with the mariadb variant. If 
the user does not already have p5.26-dbd-mysql installed, we could possibly 
improve MacPorts so that it automatically installs p5.26-dbd-mysql with the 
mariadb variant when mythtv-core.28 is installed. But what if the user has 
already installed p5.26-dbd-mysql with a different variant such as mariadb10_2? 
Presumably they did so for a reason: Presumably they already have a MariaDB 
10.2 server set up and they want to use it from Perl. If they now try to 

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

On Wed, Apr 3, 2019 at 2:35 AM Mojca Miklavec  wrote:
>
> Dear Karan,
>
> you may put such statistics and suggestions for action plans in the proposal 
> (or appendix to proposal).

Ok, I'll do that soon

> What about pypackage -> py-pypackage? (I miss such entries on the list, 
> unless those are the ones you intentionally didn't count.)

They are counted under   package ->  py-package (769) as I had just
applied a split on "py-".
Will generate the detailed stats if necessary.

> Did you ignore capitalisation of names?

Yep

> Mojca
>
> On Tue, 2 Apr 2019 at 22:51, KARAN SHETH  wrote:
>>
>> Hey,
>>
>> Just to see how many ports will we have to rename if we take on this
>> path I ran a quick script and following were the results:
>>
>> PyPI NameMacports NameNumber
>>
>> package package   10
>>
>> package py-package   769
>>
>> package py-pypackage1
>>
>> pypackage py-package14
>>
>> --  unclassified30
>>
>> Total PyPI Ports = 824
>>
>> So according to this results it would be best to name all the PyPI as
>> py-package.
>>
>> Note: The script might be a bit off.
>>
>> Thanks,
>> Karan Sheth

-- 

           
      

Re: GSoC Proposal

[Mojca Miklavec]

Dear Karan,

you may put such statistics and suggestions for action plans in the
proposal (or appendix to proposal).

What about pypackage -> py-pypackage? (I miss such entries on the list,
unless those are the ones you intentionally didn't count.)
Did you ignore capitalisation of names?

Mojca

On Tue, 2 Apr 2019 at 22:51, KARAN SHETH  wrote:

> Hey,
>
> Just to see how many ports will we have to rename if we take on this
> path I ran a quick script and following were the results:
>
> PyPI NameMacports NameNumber
>
> package package   10
>
> package py-package   769
>
> package py-pypackage1
>
> pypackage py-package14
>
> --  unclassified30
>
> Total PyPI Ports = 824
>
> So according to this results it would be best to name all the PyPI as
> py-package.
>
> Note: The script might be a bit off.
>
> Thanks,
> Karan Sheth
>

Re: GSoC Proposal

[KARAN SHETH via macports-dev]

Hey,

Just to see how many ports will we have to rename if we take on this
path I ran a quick script and following were the results:

PyPI NameMacports NameNumber

package package   10

package py-package   769

package py-pypackage1

pypackage py-package14

--  unclassified30

Total PyPI Ports = 824

So according to this results it would be best to name all the PyPI as
py-package.

Note: The script might be a bit off.

Thanks,
Karan Sheth

On Tue, Apr 2, 2019 at 1:10 PM Mojca Miklavec  wrote:
>
>
> On Tue, 2 Apr 2019 at 01:33, KARAN SHETH wrote:
>>
>>
>> Is it possible to rename the existing Python ports as per PYPI so that 
>> issues of duplicates ports can be avoided?
>
>
> Yes, I would be in favour of coming up with an "algorithm" for naming all 
> python ports and then stick to it. Ports which don't follow the convention 
> could and should be renamed.
>
> Pypi packages might be called "foo", "pyfoo", "py-foo", ... and we just need 
> to agree on how to do the name transformation in some well-defined way. Some 
> maintainers changed "pyfoo" into "py-foo", others into "py-pyfoo". This needs 
> to be unified.
>
> Mojca
>

-- 

           
      

Re: Dependencies on non-default variants

[Ryan Schmidt]

On Mar 25, 2019, at 01:56, Mojca Miklavec wrote:

> would prefer if he or she would perhaps spend time adding novel
> functionality, which could even be allowing user comments on each port
> (which you could use to explain why the port appears to be broken)

The ability to add comments on web pages is usually terrible. At best, you'll 
get a user adding a comment providing a useful recommendation or helpful tip. 
But I expect the majority of comments would actually be spam (which we would 
have to delete) or bug reports. Instead of implementing comments, we should 
refer users to existing support channels like the mailing lists and issue 
tracker. A list of open tickets and PRs for this port could even be shown on 
the page.

Re: Enabled commenting on GSoC Proposal

[Jackson Isaac]

On Tue 2 Apr, 2019, 21:07 Mihir Luthra, <1999mihir.lut...@gmail.com> wrote:

>
>> Mihir,
>>
>> The proposal is view-only. You might need to give comment-access for
>> us to leave feedback on the draft.
>>
>> Done
>

Try to avoid creating multiple mail threads. Thanks

>

GSoC 2019

[db]

On 2 Apr 2019, at 17:11, macports-dev-requ...@lists.macports.org wrote:
> GSoC 2019

Could it be possible to relocate all this topics somewhere else?

Enabled commenting on GSoC Proposal

[Mihir Luthra]

>
>
> Mihir,
>
> The proposal is view-only. You might need to give comment-access for
> us to leave feedback on the draft.
>
> Done

Re: Request to help potential GSOC students

[Jackson Isaac]

Hi Mojca,

On Fri, Mar 29, 2019 at 3:55 AM Mojca Miklavec  wrote:
>
> Dear MacPorts users & developers,
>
> You probably noticed a slightly increased traffic on this mailing list 
> recently.
>

Really appreciate your emails and response to the GSoC students :).

> GSOC is an amazing opportunity for the students to learn something
> new, and for the MacPorts community to greatly benefit from their
> work. Last year we had almost no applications, while this year we seem
> to be "bombarded" compared to the number of workforce available.
>
> I would be really grateful to get some helping hands reviewing student
> proposals and answering their questions. While not everyone of us is
> heavily skilled in every aspect of the code, some of the proposals are
> pretty generic, easy to understand and could greatly benefit from your
> feedback.

I have added my feedback to the proposals which I had access to.

>
> The deadline for the final submission is the 9th of April.
>
> Proposals that we received so far:
>
> === MacPorts Base
>
> * Speed up trace mode by Mihir Luthra
> https://docs.google.com/document/d/1qH5VMtrQ3tvd5gFPf51lmJtd6dYfUuEmO1AvXmU_4qM
>

Mihir,

The proposal is view-only. You might need to give comment-access for
us to leave feedback on the draft.

> === Web / Buildbot projects
>
> * Custom Views Plugin for Buildbot by Rajdeep Bharati
> https://docs.google.com/document/d/12wRjA8sOWNOuApHZ_fm0n1aIPLVPt9Xm2yGiMwiK3AI
>
> * Web app for port info, build and installation stats by Arjun Salyan
> https://docs.google.com/document/d/198Ivygxb2NJQz_sqzDrbDPVEYZ5Ye5Yw0LV6Bt2QmG4
>

-- 
Jackson Isaac

Re: GSoC 2019 [Buildbot ideas]

[Mojca Miklavec]

On Tue, 2 Apr 2019 at 16:01, Pierre Tardy wrote:
>
> Not sure if this is applicable, but there is no import method from
> buildbot 0.8.x to buildbot 0.9+

I don't believe we really need that.

It would be nice to "export" all the relevant (summary) data once
before decommissioning the old master (or rather import it from an
external script using the existing json api and store it into an
independent database, for example like shown on this demo page:
https://frozen-falls-98471.herokuapp.com/ports/SoapyAirspy/), but I
don't have absolutely any expectations about keeping the old data
around in the new interface. You already loose all the data even by
trivial actions such as renaming the builder, so ...

(An expected downside is that we sometimes link to buildbot logs from
our bug tracker, but we keep loosing those logs all the time already
since the buildbot master only keeps the last 10k builds.)

Maybe I didn't understand what Rajdeep was asking. If the "old
database" referred to our existing 0.8 setup: no, we don't have any
expectations to keep that one around after the upgrade.

Mojca

Re: GSoC 2019 [Buildbot ideas]

[Pierre Tardy]

Not sure if this is applicable, but there is no import method from
buildbot 0.8.x to buildbot 0.9+

Buildbot do not import previous commit  as changes when starting, it
only looks for new things.

Pierre

Le lun. 1 avr. 2019 à 23:03, Mojca Miklavec  a écrit :
>
> On Mon, 1 Apr 2019 at 18:52, Rajdeep Bharati wrote:
> >
> > When I would be configuring the real (production) macports buildmaster, 
> > then would the old database be used?
>
> Not if you deploy it on a fresh server, unless you explicitly put the
> database there (import it from elsewhere).
>
> (I don't know how many commits buildbot pulls in when you first start it.)
>
> Mojca

Re: GSoC 2019 [Collect build statistics]

[Arjun Salyan via macports-dev]

Hi Mojca,

On Tue, Apr 2, 2019 at 3:14 AM Mojca Miklavec  wrote:

> The drawbacks may include:
> - some ports will be skipped on the builder, for various reasons (port
> is known not to build on a particular builder, it may not be
> distributable, ...)
> - the buildbot master may be down or experience problems, so data
> might go missing
>

Thanks. I will consider these factors when improving upon this.


> A strange observation from your source code: you synced portindex and
> ran the conversion, but then loaded the data from another json file?
> Am I missing something?
>

No, the conversion "tclsh portindex2json.tcl portindex" is writing to the
file "syncedportindex.json". And I am reading from the same file. I am
really sorry that I did not submit a PR and it was difficult for you to
review the code.


> There are various ways to achieve the goal. Note that if you run
> portindex yourself, it will detect which files have been updated and
> only ever touch data of those ports. The portindex command could be
> modified to only output the file with changes (when you pass some
> options to it). This will still miss deletes, but it would be an
> efficient way with almost no dependencies.
>

Does this imply that we will keep a clone of macports-contrib locally and
run a modified 'portindex' command to generate a file with only the updated
ports?


> One way would be to generate portindex yourself and always remember
> what git shasum has been used, and store that shasum to the database.
> Next time when you update, check and store the latest shasum, then ask
> git which paths have changed between the two commits, and only update
> ports whose paths match the paths reported by git as changed.
>
> It could also help if you stored a "complete" git history to the
> database (shasum, which ports changed at that point, timestamp,
> parents). Not sure if that's really so helpful, just as an option.
>
> What might be an interesting approach would be to try to squeeze the
> git shasum to the PortIndex. This could also help when submitting
> statistics as it would be easier to determine how old the database is
> / when the user last synced. (It would not work for people with their
> own modifications of the tree.) If you had the shasum in portindex,
> you could still run git independently to check for the difference.
>

These methods are not very clear to me, I haven't dealt with shasums yet. I
will discuss about them, after my research.


> Just some random ideas.


Thank you so much.

Regarding updates of builds: just ask the database about which build
> you synced last, and then sync any builds newer than that, up to the
> last one. You may need to check whether a build was complete when you
> last enquired.
>

Thanks, I am already using the same method.

Arjun

Re: GSoC Proposal

[Mojca Miklavec]

On Tue, 2 Apr 2019 at 01:33, KARAN SHETH wrote:

>
> Is it possible to rename the existing Python ports as per PYPI so that
> issues of duplicates ports can be avoided?
>

Yes, I would be in favour of coming up with an "algorithm" for naming all
python ports and then stick to it. Ports which don't follow the convention
could and should be renamed.

Pypi packages might be called "foo", "pyfoo", "py-foo", ... and we just
need to agree on how to do the name transformation in some well-defined
way. Some maintainers changed "pyfoo" into "py-foo", others into
"py-pyfoo". This needs to be unified.

Mojca