Bug#1007717: Native source package format with non-native version

[Sean Whitton]

Hello,

On Tue 29 Mar 2022 at 09:06PM +02, Lucas Nussbaum wrote:

> On 28/03/22 at 16:21 -0700, Sean Whitton wrote:
>>
>> I don't think it's the preferred method.  I believe most of the project
>> sees git histories are the preferred tool for achieving the goal you
>> state.
>>
>> If we had only source packages for this purpose, then yes, 3.0 (quilt)
>> plus DEP-3 would be preferred.  But we have git, too, and indeed dgit.
>> Repositories for packages contain both upstream history and Debian
>> packaging history, and we have powerful git subcommands for extracting
>> information.  In many cases there is a good argument to be made that the
>> git history is part of the preferred form of modification.
>
> I think there are three different use cases to consider:

I agree that we should consider them separately.

>   A. preferred form for regular contributions to the package (typically
>   by the package maintainer)
>
>   B. preferred form for occasional contributions to the package (typically
>   by an NMUer, or by the security team)
>
>   C. preferred form for reviewing the packaging and Debian-specific
>   changes.
>
>
> I fully agree that the git repository is the preferred form for A.
> However, for B and C, I think that our lingua franca is the source
> package, and thus a source package that doesn't make it hard to
> understand things such as Debian-specific patches. Of course that
> could change if we were able to standardize on a git workflow (or
> a small set of git workflows), but I don't see this happenning anytime
> soon.

What you get from 'dgit clone' is designed to serve (B) and (C) well,
and somewhat sidesteps precisely the problems created by our having
multiple git workflows.

Please consider trying out dgit-user(7) and/or dgit-nmu-simple(7) next
time you need to engage in (B) or (C).  It's really very nice :)

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1007717: Native source package format with non-native version

[Bastian Blank]

On Tue, Mar 15, 2022 at 04:29:17PM +, Ian Jackson wrote:
>  1. Declare explicitly that there is nothing wrong with a package with
> a native format, but a non-native version number.
>  2. Request that the dpkg maintainer relax the restriction which
> prevents the use of 3.0 native with Debian revision.

One additional point:
We currently have packages in the archive, which must be "3.0 (native)",
but are generated from other packages: all the secure boot signed
packages.

Example source package:
| Package: linux-signed-amd64
| Version: 5.17~rc8+1~exp1

is generated from binary package:
| Package: linux-image-amd64-signed-template
| Version: 5.17~rc8-1~exp1

Generating the source package needs to munge the version, just to
appease that version restriction.  This also means, adding dependencies
on versions is difficult, as both versions sort differently.  And bugs
are assigned to wrong versions, so version tracking does not work.

Regards,
Bastian

-- 
Kirk to Enterprise -- beam down yeoman Rand and a six-pack.

Bug#1007717: Native source package format with non-native version

[Lucas Nussbaum]

Hi Sean,

On 28/03/22 at 16:21 -0700, Sean Whitton wrote:
> Hello Lucas,
> 
> Many thanks for providing this summary of your position.  Let me just
> note a point of disagreement:
> 
> On Tue 15 Mar 2022 at 10:06PM +01, Lucas Nussbaum wrote:
> 
> > A point that I find important is the following: as a project, I think
> > that we care about facilitating the review of changes we make to
> > upstream source.
> 
> Certainly we do.
> 
> > I think that the preferred method (and widely accepted method) for
> > that is currently to use the 3.0 (quilt) format with DEP-3-documented
> > patches, on top of a tarball that contains the pristine upstream
> > source.
> >
> > The git packaging workflows that generate source packages using either
> > 1.0 native packages, or 1.0 non-native packages without patches, make it
> > harder to identify and review those changes, as they require browsing
> > the git repository, which sometimes is not properly documented using
> > Vcs-*.
> 
> I don't think it's the preferred method.  I believe most of the project
> sees git histories are the preferred tool for achieving the goal you
> state.
> 
> If we had only source packages for this purpose, then yes, 3.0 (quilt)
> plus DEP-3 would be preferred.  But we have git, too, and indeed dgit.
> Repositories for packages contain both upstream history and Debian
> packaging history, and we have powerful git subcommands for extracting
> information.  In many cases there is a good argument to be made that the
> git history is part of the preferred form of modification.

I think there are three different use cases to consider:

  A. preferred form for regular contributions to the package (typically
  by the package maintainer)

  B. preferred form for occasional contributions to the package (typically
  by an NMUer, or by the security team)

  C. preferred form for reviewing the packaging and Debian-specific
  changes.


I fully agree that the git repository is the preferred form for A.
However, for B and C, I think that our lingua franca is the source
package, and thus a source package that doesn't make it hard to
understand things such as Debian-specific patches. Of course that
could change if we were able to standardize on a git workflow (or
a small set of git workflows), but I don't see this happenning anytime
soon.

Lucas

Bug#1007717: Native source package format with non-native version

[Sean Whitton]

Hello Lucas,

Many thanks for providing this summary of your position.  Let me just
note a point of disagreement:

On Tue 15 Mar 2022 at 10:06PM +01, Lucas Nussbaum wrote:

> A point that I find important is the following: as a project, I think
> that we care about facilitating the review of changes we make to
> upstream source.

Certainly we do.

> I think that the preferred method (and widely accepted method) for
> that is currently to use the 3.0 (quilt) format with DEP-3-documented
> patches, on top of a tarball that contains the pristine upstream
> source.
>
> The git packaging workflows that generate source packages using either
> 1.0 native packages, or 1.0 non-native packages without patches, make it
> harder to identify and review those changes, as they require browsing
> the git repository, which sometimes is not properly documented using
> Vcs-*.

I don't think it's the preferred method.  I believe most of the project
sees git histories are the preferred tool for achieving the goal you
state.

If we had only source packages for this purpose, then yes, 3.0 (quilt)
plus DEP-3 would be preferred.  But we have git, too, and indeed dgit.
Repositories for packages contain both upstream history and Debian
packaging history, and we have powerful git subcommands for extracting
information.  In many cases there is a good argument to be made that the
git history is part of the preferred form of modification.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1007717: Native source package format with non-native version

[Matthew Vernon]

On 17/03/2022 17:52, Russ Allbery wrote:

Helmut Grohne  writes:


Do you think it would be impossible to move forward on this matter in a
consensus-based way?


I don't know.  I have some reasons to be dubious, but it's possible that
I'm being excessively pessimistic.


I'm inclined to agree with Russ here; my impression is there are one or 
two long-standing areas of disagreement here, and that consensus hasn't 
been arrived at.



In the spirit of consensus: Do you agree that retrying this in a
consensus-based way is still possible?


If the relevant people required to implement a decision are willing to
tackle it that way, I am certainly willing to participate from the Policy
side.


For the avoidance of doubt, if that is the case, I am not going to 
suggest the TC gets in the way!


Regards,

Matthew

Bug#1007717: Native source package format with non-native version

[Lucas Nussbaum]

Hi Ian,

On 15/03/22 at 16:29 +, Ian Jackson wrote:
> Part I - belss continued use of 1.0 native format, for now at least:
> 
>  1. Declare explicitly that there is nothing wrong with a package with
> a native format, but a non-native version number.
> 
>  2. Request that the dpkg maintainer relax the restriction which
> prevents the use of 3.0 native with Debian revision.
> 
>  3. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against packages where simply changing the
> source format does not currently work.  That would include at
> least 1.0 native packages with Debian revisions.
> 
> Part II - bless continued use of 1.0-with-diff, for now at least:
> 
>  4. Declare that sometimes the use of 1.0-with-diff can be the best
> tradeoff between different considerations.  In particular,
> because 1.0 is the only format which botH:
>  (a) Optimises bandwidth and storage by reusing the upstream 
>  data when it hasn't changed.
>  (b) Avoids polluting the working tree (package source code)
>  with [patches], which cause trouble especially with
>git-based workflows.
> 
>  5. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against 1.0 with diff packages, at least
> without some further filter.

I did the MBF mentioned in (5) (about suggesting the move from the 1.0
format to one of the 3.0 formats), and agreed to pause those efforts in
.

However, it might be worth clarifying if the MBF in (3) is mine, or
Guillem's one (with usertag dpkg-mismatch-source-vs-version-format and
user debian-d...@lists.debian.org) about "Mismatched source format vs
source version" (an example bug is #1007088). I think it's mine, but I'm
not sure. It might also be both.

Lucas

Bug#1007717: Native source package format with non-native version

[Felix Lechner]

Hi,

On Thu, Mar 17, 2022 at 11:57 AM Russ Allbery  wrote:
>
> source package format

While everyone is receptive to new labels, I prefer "upload format" or
"archive format". Either one helps us to distinguish the intermediate
product from any workflow objects a maintainer may have.

> single tarball as a source package format

It is also not necessary to "define" unpatched sources in the archive
by how many tarballs they have, or any tarballs at all.

In fact, I wrote a manifest utility that could one day replace tarball
signatures with per-file hashes. The latter remain useful even when
sources are repackaged, because manifests can be cryptographically
daisy-chained in a "chain of custody." Of course, they would be more
often used for patched upload formats.

Kind regards
Felix Lechner

Bug#1007717: Native source package format with non-native version

[Russ Allbery]

Sam Hartman  writes:
>> "Russ" == Russ Allbery  writes:

> Russ> Switching terminology to completely leave behind the terms
> Russ> with ambiguous meanings isn't a bad idea, but if so we really
> Russ> need a term that captures "is a packaging of an upstream
> Russ> software package with a separate existence" versus "exists
> Russ> solely as a Debian package."  "with-revision" or
> Russ> "without-revision" doesn't feel to me like it does this.
> Russ> Native and non-native do, which is why I was sticking with
> Russ> them, but maybe we can come up with some other equally-good
> Russ> terminology.

> Why do we need that distinction?

I think we need some way to talk about the various things that change
about a package if it is a packaging of an upstream package.  Examples
include a debian/watch file, an upstream metadata file, an upstream
signing key (still useful with single-tarball formats if upstream uses
signed tags, etc.), and provenance information in debian/copyright.

Those things are not tied to the source package format.  We want
provenance information for the upstream source even if you're using a
single tarball as a source package format, but we don't need it if there
is no upstream.

> Looking at current policy, 5.6.12 talks about having a debian revision
> or not having a debian revision.

It seems less useful to me to frame this solely in terms of the version
number format, rather than having the version number format indicate
whether this is a packaging of an upstream source with an independent
existence.  That would imply that if you just omit the debian revision,
you don't need to record the provenance of the upstream source, which
doesn't seem correct.

-- 
Russ Allbery (r...@debian.org)  

Bug#1007717: Native source package format with non-native version

[Sam Hartman]

> "Russ" == Russ Allbery  writes:

Russ> Switching terminology to completely leave behind the terms
Russ> with ambiguous meanings isn't a bad idea, but if so we really
Russ> need a term that captures "is a packaging of an upstream
Russ> software package with a separate existence" versus "exists
Russ> solely as a Debian package."  "with-revision" or
Russ> "without-revision" doesn't feel to me like it does this.
Russ> Native and non-native do, which is why I was sticking with
Russ> them, but maybe we can come up with some other equally-good
Russ> terminology.

Why do we need that distinction?

Looking at current policy, 5.6.12 talks about having a debian revision
or not having a debian revision.

Other parts of policy talk about  what parts of the source package there
are.

Why do we need more than these two distinctions.

I think that current policy has mostly left behind the work native
(although their are a few uses still).
My suspicion is that avoiding native allowed us to get a broader
consensus in the policy process.

Why isn't what we have good enough?

Bug#1007717: Native source package format with non-native version

[Russ Allbery]

Helmut Grohne  writes:

> Do you think it would be impossible to move forward on this matter in a
> consensus-based way?

I don't know.  I have some reasons to be dubious, but it's possible that
I'm being excessively pessimistic.

> Yes, please. Though as is evidenced in the replies to your mail, I would
> try to avoid "native" and "non-native" as much as possible given the
> existing confusion. I suggest using something like with-revision vs
> without-revision and single-tarball (from your mail) vs
> patches-separated to transport the concepts.

Switching terminology to completely leave behind the terms with ambiguous
meanings isn't a bad idea, but if so we really need a term that captures
"is a packaging of an upstream software package with a separate existence"
versus "exists solely as a Debian package."  "with-revision" or
"without-revision" doesn't feel to me like it does this.  Native and
non-native do, which is why I was sticking with them, but maybe we can
come up with some other equally-good terminology.

> More and more, it seems to me that we are looking into design work as
> opposed to picking an existing option.

*I* was doing design work, for sure.  But I'm not a member of the TC.  :)
The point was to offer you a design to consider as part of submitting the
request to the TC.

> In the spirit of consensus: Do you agree that retrying this in a
> consensus-based way is still possible?

If the relevant people required to implement a decision are willing to
tackle it that way, I am certainly willing to participate from the Policy
side.

-- 
Russ Allbery (r...@debian.org)  

Bug#1007717: Native source package format with non-native version

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> Hi Russ,
Helmut> On Tue, Mar 15, 2022 at 09:22:09PM -0700, Russ Allbery wrote:
>> > Specifically, I'd like to ask the TC to come up with policy on
>> native > packages and debian revisions using its power under
>> 6.1.1.
>> 
>> As a Policy Editor, I support this request.

Helmut> As a TC member I admit disliking this. While there is
Helmut> disagreement on how things are supposed to work, the views
Helmut> don't seem to be that far apart. Urgency is effectively
Helmut> removed by Lucas agreeing to pause further deprecation
Helmut> work. Do you think it would be impossible to move forward on
Helmut> this matter in a consensus-based way?

I think this is like usrmerge.
I actually think there is a rough consensus, but there are one or two
parties who find that consensus unacceptable and who are in a position
to block the consensus.

As an example, the text in policy 5.6.12 has been significantly revised
since Russ and I started discussing this issue in 2014.
It's possible that policy could be improved, but honestly I think that
policy is already consistent with the hypothesis I presented to the TC.

I also already tried to seek consensus where this is possible as DPL,
and included a pointer to my consensus call on the issues where there
was consensus in the material I already presented to the TC.

This is a case where people have slowly been building consensus since
2014.  One of the key stakeholders has not effectively participated
constructively, and so some things haven't been possible.

We're asking the TC to do the last mile work--review the existing
consensus-forming discussions, figure out where we are, and wrap it up
just like you did with usrmerge.

I've even tried to give you the key discussions that happened on
debian-devel.
I suspect one of the policy editors could go pull that together for
discussions on debian-policy, but if not, I'd be happy to do that work
if requested.

So, no, I do think this is ready for the TC to finish things up now.

--Sam


signature.asc
Description: PGP signature

Bug#1007717: Native source package format with non-native version

[Lucas Nussbaum]

On 16/03/22 at 23:54 +, Wookey wrote:
> On 2022-03-16 15:29 +, Ian Jackson wrote:
> > In practice, the vast majority of packages are maintained in git on
> > salsa.  The maintainers use those git repositories as the PFM.
> 
> > but almost everyone is already treating git as primary.
> 
> Is this definitely true? For example: I know I'm not doing this. I did
> try, and I do have some git repos on salsa, but I've mostly given up
> with it all and stuck with uscan and tarballs and quilt (and my trusty
> 'packages' directory). It's much easier for me. (The salsa repos that
> exist for my packages are not canonical and often stale).
> 
> I'm sure Ian is right that there is a trend towards git from tarballs
> and dscs, but I just question whether we know it is 'the vast
> majority'? Are there really now very few maintainers using the
> 'classic tooling'? How do we know?

Hi,

If you look at https://trends.debian.net/#version-control-system, I
think it's fair to say that the vast majority of packages are now
maintained in git.

And if you look at the status of those repositories according to
vswatch, most of them are actively used:

udd=> select status, count(distinct source) from vcswatch group by status order 
by 2;
 status  | count 
-+---
 UNREL   |   119
 OLD |  1218
 ERROR   |  1328
 COMMITS |  3620
 NEW |  7127
 OK  | 17873

(the last three lines being 'normal' states.)

Lucas

Bug#1007717: Native source package format with non-native version

[Helmut Grohne]

Hi Russ,

On Tue, Mar 15, 2022 at 09:22:09PM -0700, Russ Allbery wrote:
> > Specifically, I'd like to ask the TC to come up with policy on native
> > packages and debian revisions using its power under 6.1.1.
> 
> As a Policy Editor, I support this request.

As a TC member I admit disliking this. While there is disagreement on
how things are supposed to work, the views don't seem to be that far
apart. Urgency is effectively removed by Lucas agreeing to pause further
deprecation work. Do you think it would be impossible to move forward on
this matter in a consensus-based way?

> I've been involved in a lot of these discussions over the years, and the
> tentative conclusion that I've come to is that we have unfortunately mixed
> several different concepts in how we talk about source packages.  This has
> not *technically* reduced the flexibility of the packaging format because
> there are various workarounds, but it's *conceptually* reduced the
> flexibility in a way that makes those workarounds look conceptually
> incorrect.  As a result, we have constant repetitions of similar arguments
> stemming from well-meaning attempts to clean up the apparent
> inconsistencies (including some I have started).

While I did point out the conflation of matters in my initial reply to
Ian, I didn't put it as clearly as you did. Thank you. In the later
replies to your mail, we see that even trying to disentangle this is
difficult terminology-wise.

> There are really two separate concepts in play here:

I am wondering whether maybe one of you could work on a consensus
seeking process about these matters. I believe that resolving this using
consensus is far better than having 8 semi-random people decide upon a
policy. The consensus-based approach does not seem impossible to me at
this stage.

> What I would propose is to separate these concepts cleanly so that we can
> talk about them in a clearer way.  We should define native and non-native
> packages in terms of version numbers, and allow both native and non-native
> packages to use single-tarball source package formats.  We may want to
> steer people away from using the single-tarball source package format for
> the *normal* case of packaging upstream software, but I think it's well
> within the sorts of trade-off decisions that packagers already make and
> there are cases where a single tarball source format makes sense.

Yes, please. Though as is evidenced in the replies to your mail, I would
try to avoid "native" and "non-native" as much as possible given the
existing confusion. I suggest using something like with-revision vs
without-revision and single-tarball (from your mail) vs
patches-separated to transport the concepts. Of course "3.0 (native)" as
the content of debian/source/format cannot be avoided, but we can
describe its current and possibly desired properties in a less confusing
way.

> The name of the 3.0 (native) source package format is unfortunate in this
> context since it strongly implies that only native packages will use that
> source format.  The most formally correct thing to do would probably be to
> introduce a new 3.0 (single-tarball) source format and use 1.0 until then,
> but of course introducing a new source package format is not a fast
> process.  A more expedient thing to do may be to allow use of 3.0 (native)
> with non-native packages, since it's identical to 3.0 (single-tarball)
> except for the name, but it does leave a long-term confusing naming scheme
> in place.  Guillem may well have other and better suggestions from the
> dpkg perspective; I haven't thought all that much about a transition plan.

More and more, it seems to me that we are looking into design work as
opposed to picking an existing option. I think it is preferable to do
that with "normal" Debian processes without involving the TC, because
the latter always comes with constitutional powers attached (used or
unused). If history has taught us anything, involving the TC always
comes with escalation. We may still reach a point where it is clear that
consensus is not possible, but I don't think we're there yet.

In the spirit of consensus: Do you agree that retrying this in a
consensus-based way is still possible?

Helmut

Bug#1007717: Native source package format with non-native version

[Wookey]

On 2022-03-16 15:29 +, Ian Jackson wrote:
> In practice, the vast majority of packages are maintained in git on
> salsa.  The maintainers use those git repositories as the PFM.

> but almost everyone is already treating git as primary.

Is this definitely true? For example: I know I'm not doing this. I did
try, and I do have some git repos on salsa, but I've mostly given up
with it all and stuck with uscan and tarballs and quilt (and my trusty
'packages' directory). It's much easier for me. (The salsa repos that
exist for my packages are not canonical and often stale).

I'm sure Ian is right that there is a trend towards git from tarballs
and dscs, but I just question whether we know it is 'the vast
majority'? Are there really now very few maintainers using the
'classic tooling'? How do we know?

Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Bug#1007717: Native source package format with non-native version

[Russ Allbery]

Felix Lechner  writes:
> On Tue, Mar 15, 2022 at 9:33 PM Russ Allbery  wrote:

>> We should define native and non-native packages in terms of version
>> numbers, and allow both native and non-native packages to use
>> single-tarball source package formats.

> I co-maintaintain several Debian-internal tools, and that description is
> backwards. "Native" sources are characterized by their lack of Debian
> patches.

This is exactly the semantic confusion that I believe is wrong and we
should undo.  "Native" vs. "non-native" should be a property of the
relationship between the package and a separate upstream release, which is
represented in the version.  That should be decoupled from the source
package format.

The point that Ian and Sam are raising is that a single tarball is a good
way of representing the Debian package source in several non-native cases
where we are still packaging a piece of software with an independent
upstream existence, and where having a version with a Debian revision is
still semantically correct.

> On that note, the term "native" is also not great. The words "patched"
> and "unpatched" describe the relationship between sources in the archive
> and their respective upstreams more accurately.

I think that would add even more confusion.  It's very common for
non-native packages to switch between patched and unpatched from upload to
upload, depending on whether Debian has to carry additional patches,
without any change in their version number format or in the source package
format.  This is correct and should continue to be supported.

-- 
Russ Allbery (r...@debian.org)  

Bug#1007717: Native source package format with non-native version

[Ian Jackson]

Helmut Grohne writes ("Re: Bug#1007717: Native source package format with 
non-native version"):
> Beyond the content of your request, I have a meta-question. Do you see a
> particular urgency with the processing of your request? What is - in
> your opinion - a reasonable time for resolving this? Of course, if Lucas
> et al are to proceed with their deprecation work, urgency may arise in
> your view, but let us assume that we can ask them to pause their
> efforts for a while.

As you surmise, the only urgency I feel is due to these deprecation
efforts (some of which take other forms).

> Lucas, please pause further work on deprecating the 1.0 format in order
> to give time to settle the dispute at hand. This implies not sending
> further bugs about it. On the other hand, I think closing all of the
> existing ones would not do any good at this time either.

It is a shame that these bugs were filed and a lot of maintainers
(including at least one of my sponsees) now find themselves in a
confusing position.  Perhaps it would be worth sending an agreed
holding message to these bugs, and/or blocking them by this one.

> Can I ask you to step back a bit and help us figure out what you really
> want to achieve?
> 
> Given your other messages on the matter, I suppose that you want to be
> able to use a source package format that can faithfully represent any
> valid source tree. In particular, a source tree with a Debian revision
> should be representable. You care less about being able to represent
> differences to a tarball in some cases as long as the changed tree as a
> whole is representable. This includes changes to permissions, symbolic
> links and binary files. Which source format is used is not important to
> you as long as it is able to meet these requirements.

Yes, that is by and large my SNPS (solution neutral problem statement)
for the ideal situation.

But, note that "faithfully" implies not including a copy of the Debian
delta in the form of patches within the unpacked source tree (as "3.0
(quilt)" does), since those patches must then be kept up to date.

And, I want a format that offers the bandwidth and disk space
optimisation of not re-uploading large files when only small changes
are made.  This is necessary for large packages.  The various
with-diffs/with-patches formats achieve this, but compromise on other
aspects.

> > Part II - bless continued use of 1.0-with-diff, for now at least:
> > 
> >  4. Declare that sometimes the use of 1.0-with-diff can be the best
> > tradeoff between different considerations.  In particular,
> > because 1.0 is the only format which botH:
> >  (a) Optimises bandwidth and storage by reusing the upstream 
> >  data when it hasn't changed.
> >  (b) Avoids polluting the working tree (package source code)
> >  with [patches], which cause trouble especially with
> >  git-based workflows.
> 
> I note that the 1.0-with-diff format does not meet my perception of your
> earlier requirement: Being able to represent any source tree.

You are correct.  But there is not currently any format that meets all
of my requirements.

In particular, 1.0-with-diff achieves the bandwdith/storage
optimisation, at the cost of representational capability.
Looked at this way "3.0 (quilt)" is mixed - it has some additional
representational capabilities, but reduced fidelity.

>  I also note that it is possible to choose a different conversion
> mechanism than dpkg-source -b between working tree and .dsc that
> does not incur your reported pollution. In particular, you do have
> experience in implementing such conversions as is evidenced in dgit
> and such conversions do exist.

Um.  The conversions done by dgit exist to *keep the pollution up to
date*.  And they are really quite inconvenient.  dgit must make
commits on your branch, or maintain multiple branches.  There is a
substantial amount of code there, dealing with a large variety of
different situations.

> Your request also touches the question of the preferred form for
> modification. That used to be the .dsc, but for the way you use the
> archive, it merely is an export for a preferred form for modification
> that resides elsewhere. I do not think we have consensus for this view
> either.

In practice, the vast majority of packages are maintained in git on
salsa.  The maintainers use those git repositories as the PFM.
The git repositories contain much information which is not in the
source packages (generated by gbp or dgit on maintainers' machines),
and, depending on the package, that information is sometimes critical.
IOW that ship has sailed.

My programme of properly supporting git in Debian is trying to sort
out this situation, by providing a discoverable, coherent, and
reliable git view of 

Bug#1007717: Native source package format with non-native version

[Felix Lechner]

Hi,

On Tue, Mar 15, 2022 at 9:33 PM Russ Allbery  wrote:
>
> We should define native and non-native
> packages in terms of version numbers, and allow both native and non-native
> packages to use single-tarball source package formats.

I co-maintaintain several Debian-internal tools, and that description
is backwards. "Native" sources are characterized by their lack of
Debian patches.

On that note, the term "native" is also not great. The words "patched"
and "unpatched" describe the relationship between sources in the
archive and their respective upstreams more accurately.

As for version strings, we need no additional restrictions. The use of
patches is declared in source format 3.0. Some folks even use Debian
revisions for unpatched sources. [1]

Most significantly, Lintian's parser is unable to deduce "nativeness"
from the version number. The native status is a required input! [2]

Please do not "define" a source's patch status via the version string.
It's what got us here in the first place. Debian version numbers are
complicated enough already. Thank you!

Kind regards,
Felix Lechner

[1] for example, https://tracker.debian.org/pkg/python3-defaults
[2] 
https://salsa.debian.org/lintian/lintian/-/blob/master/lib/Lintian/Changelog/Version.pm#L79-80

Bug#1007717: Native source package format with non-native version

[Russ Allbery]

Sam Hartman  writes:

> Specifically, I'd like to ask the TC to come up with policy on native
> packages and debian revisions using its power under 6.1.1.

As a Policy Editor, I support this request.

I've been involved in a lot of these discussions over the years, and the
tentative conclusion that I've come to is that we have unfortunately mixed
several different concepts in how we talk about source packages.  This has
not *technically* reduced the flexibility of the packaging format because
there are various workarounds, but it's *conceptually* reduced the
flexibility in a way that makes those workarounds look conceptually
incorrect.  As a result, we have constant repetitions of similar arguments
stemming from well-meaning attempts to clean up the apparent
inconsistencies (including some I have started).

There are really two separate concepts in play here:

1. Different version numbering conventions for native and non-native
   packages.  For non-native packages, we want to preserve the upstream
   version number as much as possible so that our users can correlate our
   packages to upstream releases.  We therefore have the Debian revision
   component of the version number.  But we don't want to use that
   component for native packages, since there is no corresponding upstream
   version.

   And I do believe that we need to keep the concept of native packages.
   There are configuration-only packages, metapackages, and other cases
   where even with an extremely aggressive idea of what should entail a
   separate upstream release, a separate upstream makes no sense.  And I
   can also confirm that outside of Debian native packages are heavily
   used by Debian users to package local things.

2. Source package formats.  Here, people want different things based on
   different workflows.  We may have opinions about which workflows are
   better and which are worse, but by and large Debian tries to provide
   room for innovation on workflows, and there are some important
   workflows outside of the archive (such as automated builds from CI as
   Sam has mentioned) where simplicity is far more important than careful
   tracking of distinctions between upstream and Debian changes.

   There is a clearly-expressed desire to support a package format with as
   simple of a representation as possible for some of those workflows,
   either because the complexity is being handled at another level or
   because the complexity is make-work for the desired goal that isn't
   worth doing (true for a lot of out-of-archive use cases).

   The current native format of a single tarball that you unpack and can
   then build as a Debian package with no further work is exactly what
   many of those workflows want.

We have conflated 1 and 2, but I don't think they are as closely related
as the project has conceptually presented them.  There is a tie in that
*if* there is a separate upstream release, then someone may want a package
format that allows keeping the Debian pieces and the upstream pieces
clearly separate.  But in practice there are reasons to want to use the
single-tarball source format for a non-native package.

(I admit I am unable to think of a plausible reason to use the multiple
tarball or tarball with patch format for a native package, but that may be
a failure of my imagination.  I can think of some space-conserving reasons
to want to use that packaging format if the archive would reuse the
tarballs, but the archive software won't when using native versioning, and
changing that would be extremely difficult and probably not worth the
effort.)

What I would propose is to separate these concepts cleanly so that we can
talk about them in a clearer way.  We should define native and non-native
packages in terms of version numbers, and allow both native and non-native
packages to use single-tarball source package formats.  We may want to
steer people away from using the single-tarball source package format for
the *normal* case of packaging upstream software, but I think it's well
within the sorts of trade-off decisions that packagers already make and
there are cases where a single tarball source format makes sense.

The name of the 3.0 (native) source package format is unfortunate in this
context since it strongly implies that only native packages will use that
source format.  The most formally correct thing to do would probably be to
introduce a new 3.0 (single-tarball) source format and use 1.0 until then,
but of course introducing a new source package format is not a fast
process.  A more expedient thing to do may be to allow use of 3.0 (native)
with non-native packages, since it's identical to 3.0 (single-tarball)
except for the name, but it does leave a long-term confusing naming scheme
in place.  Guillem may well have other and better suggestions from the
dpkg perspective; I haven't thought all that much about a transition plan.

Note that this only addresses the first part of Ian's request.  The second

Bug#1007717: Native source package format with non-native version

[Sam Hartman]

Dear TC,
I cannot speak for what Ian wants,
but I would also like to formally ask the TC to rule on this issue.
My hope is that what Ian and I are asking for is similar enough that the
TC can consider the issues together.

Specifically, I'd like to ask the TC to come up with policy on native
packages and debian revisions using its power under  6.1.1.
In particular I ask the TC to consider the hypothesis that:

* Debian revisions are useful when there is a separate upstream flow
  from the packaging flow.  For example when Debian versions are
  released at different times than upstream times, Debian versions have
  different content (for example a debian directory), or Debian versions
  are maintained by different people.  IN these cases a debian revision
  may be useful.

* Native source package formats (1.0 tar only and 3.0 (native)) are
  useful when the work flow
  is easier to maintain without separated Debian changes.  There are
  several cases involving git work flows where package maintenance is
  improved by not requiring this separation.

* Git work flows are a best practice.  There may be other best practices
  too.  We want to encourage git work flows to become more streamlined
  and easier; when people have found git work flows that make their job
  as maintainers easier, we want to encourage that.

I don't think the TC needs to do out-of-charter design work to come up
with policy in this regard.
I think there's been a lot of work within debian-policy, within
discussions of git packaging flows, and within the  consensus-seeking
discussions run over the years.

I'll point outt that the behavior of non-experimental package
building tools explicitly falls under 6.1.1 and does not require the TC
to override a maintainer.

I believe it is also appropriate for the TC to rule under 6.1.2.
It's clear that policy, the packaging tools, archive build tools, and
tools to support git work flows need to be in harmony for things to work
well.
It's clear there is overlap, and it is clear to me we do not have
harmony today.

I'd also like to provide some initial briefing material to the TC to
help you all come up to speend on this issue.

* Ian's mail that you were forwarded

* #bug 737634

   * My messages starting at
 
https://lists.debian.org/0144017b42b6-b65ba883-c94b-472c-89b7-7341c14ce8ab-000...@email.amazonses.com
 and following the thread.  I explain one case where you want to use
 a debian revision with native packages.  I think Ian has described
 cases that are more clearly directly related to the Debian archive,
 but I clearly outline in that bug report problems I was having and
 why proposed alternatives did not work.

* Around that time  a claim was made that debian-policy did not permit
  debian revisions with native packages.  I challenged that claim and in
  https://lists.debian.org/8761otp2r1@windlord.stanford.edu Russ
  acknowledges that the section of policy he was pointing at did not
  actually forbid debian revisions with native packages.  So at the time
  dpkg made the change it was breaking things and there was no policy
  requirement for the change.

* Ubuntu had to make a change to dpkg-source at least as used by
  launchpad for recipe builds to continue to work.  I don't know if they
  still maintain that patch, but it seems like a bad sign when a major
  downstream needs to revert a change  we've done to get work done.  It
  would perhaps be valuable to look into whether they still have that
  delta and if not how they work around things.

* https://lists.debian.org/tsla738bpwz@suchdamage.org the most
  interesting bit there is a consensus of the project that if your
  upstream uses git, best practice is for your packaging to use git.

* https://lists.debian.org/tslr252ioa3.fsf...@suchdamage.org Consensus:
  native packages are sometimes an appropriate tool to use in contrast
  to earlier consensus that we wanted to move away from them.

* Plus the recent thread on debian-devel.

* The git packaging BOF at DebConf19.

Thanks for your consideration,

--Sam


signature.asc
Description: PGP signature

Bug#1007717: Native source package format with non-native version

[Lucas Nussbaum]

Hi,

First, some context with numbers:
Source packages in testing: 32247
Source packages in testing using 3.0 (native): 690 (2.1%)
Source packages in testing using 3.0 (quilt): 30937 (95.9%)
Source packages in testing using 1.0: 620 (1.9%)

Those 620 packages probably fit in two categories:
A) those whose maintainers are inactive, or unaware that the
  package uses 1.0, and would switch to 3.0 if they knew about it
B) those whose maintainers choose to stay with 1.0 for a reason

Estimating the number of packages in both categories in difficult.
Packages obviously in (B) are those maintained by the Debian X team,
by iwj, by tfheen, or by az. Those that provide debian/source/format
are also good candidates (but there might be some false positives).
The number of each case and for the combination of both are:
... with explicit debian/source/format: 178
... maintained or co-maintained by Debian X, iwj, tfheen, az: 129
... with explicit debian/source/format OR maintained or
co-maintained by Debian X, iwj, tfheen, az: 262 (0.8%)

So, we are talking about approximately 0.8% of packages here.
There's a page at https://udd.debian.org/cgi-bin/format10.cgi that
allows exploring the list of packages (the classification criteria are
the ones used for the MBF, not the ones above).
There are also graphs at
https://trends.debian.net/#source-formats-and-patch-systems to see the
evolution over time.

On 15/03/22 at 20:08 +0100, Helmut Grohne wrote:
> On Tue, Mar 15, 2022 at 04:29:17PM +, Ian Jackson wrote:
> > (Sorry for the resend, this should have gone to the BTS the first
> > time; have fixed a slip in the wording.)
> 
> Thank you for resubmitting your issue as a bug report.
> 
> Beyond the content of your request, I have a meta-question. Do you see a
> particular urgency with the processing of your request? What is - in
> your opinion - a reasonable time for resolving this? Of course, if Lucas
> et al are to proceed with their deprecation work, urgency may arise in
> your view, but let us assume that we can ask them to pause their
> efforts for a while.
> 
> Lucas, please pause further work on deprecating the 1.0 format in order
> to give time to settle the dispute at hand. This implies not sending
> further bugs about it. On the other hand, I think closing all of the
> existing ones would not do any good at this time either.

I do not plan to upgrade the severity of the bugs, nor to file other
bugs. I am fine with maintainers/co-maintainers closing bugs (especially
with a rationale), and do not plan to reopen bugs that will be closed. I
do not plan to try to do further cleanup using a more complex heuristic
as I think that it's valuable to use those bugs to further identify
what are the blocking factors for migration to 3.0.

I can agree that the heuristic to determine which packages to file bugs
against could have been designed better. However, given that it's
impossible to read maintainers' mind, it's probably impossible to
achieve perfection here. (For example, I identified that the Debian X
team was using 1.0 with a specific workflow and excluded its packages, I
should probably have excluded Ian's packages as well, but only
discovered the objection of other maintainers because I did the MBF.)


A point that I find important is the following: as a project, I think
that we care about facilitating the review of changes we make to
upstream source. I think that the preferred method (and widely accepted
method) for that is currently to use the 3.0 (quilt) format with
DEP-3-documented patches, on top of a tarball that contains the pristine
upstream source.

The git packaging workflows that generate source packages using either
1.0 native packages, or 1.0 non-native packages without patches, make it
harder to identify and review those changes, as they require browsing
the git repository, which sometimes is not properly documented using
Vcs-*.

I do not think that the practice of making "fake" native source packages
should be encouraged by allowing 3.0 (native) packages to have Debian
revisions.

So my "ideal" TC ruling on those matters would be something along the
lines of:
- Use of 3.0 (quilt) or 3.0 (native) is recommended
- Use of 1.0 is discouraged
- After a proper work to identify affected packages and notify
  maintainers in advance, providing a debian/source/format file
  can be made mandatory to accelerate the transition to 3.0 (which the
  ability for packages to stay with 1.0 if explicit)
- Packages still using 1.0 must provide an adequate way to review Debian
  changes, such as a functional VCS repository.
- Maintainers that rely on workflows that require the use of 1.0 are
  encouraged to explore how to move to 3.0 without compromising the
  capability to review changes using the source package.

Lucas

Bug#1007717: Native source package format with non-native version

[Felix Lechner]

Hi Ian,

On Tue, Mar 15, 2022 at 11:54 AM Ian Jackson
 wrote:
>
> I do remember this coming up
> before, but I don't seem to be able to find a record of it now.

Perhaps you were thinking about this discussion in a bug against Lintian?

Ideally I would like dpkg-source to permit source format
`3.0 (native)' packages to have a non-native version number. [1]

Kind regards,
Felix Lechner

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953554#30

Bug#1007717: Native source package format with non-native version

[Helmut Grohne]

Hi Ian,

On Tue, Mar 15, 2022 at 04:29:17PM +, Ian Jackson wrote:
> (Sorry for the resend, this should have gone to the BTS the first
> time; have fixed a slip in the wording.)

Thank you for resubmitting your issue as a bug report.

Beyond the content of your request, I have a meta-question. Do you see a
particular urgency with the processing of your request? What is - in
your opinion - a reasonable time for resolving this? Of course, if Lucas
et al are to proceed with their deprecation work, urgency may arise in
your view, but let us assume that we can ask them to pause their
efforts for a while.

Lucas, please pause further work on deprecating the 1.0 format in order
to give time to settle the dispute at hand. This implies not sending
further bugs about it. On the other hand, I think closing all of the
existing ones would not do any good at this time either.

> Please:

I note that you request a number of partially independent matters that
form disagreements with various people. The request is fairly entangled
(as is the matter at hand). I'm also wondering whether your request may
be over-specific.

Can I ask you to step back a bit and help us figure out what you really
want to achieve?

Given your other messages on the matter, I suppose that you want to be
able to use a source package format that can faithfully represent any
valid source tree. In particular, a source tree with a Debian revision
should be representable. You care less about being able to represent
differences to a tarball in some cases as long as the changed tree as a
whole is representable. This includes changes to permissions, symbolic
links and binary files. Which source format is used is not important to
you as long as it is able to meet these requirements.

Do you consider the above representation of your view accurate? If not,
please point out differences.

>  3. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against packages where simply changing the
> source format does not currently work.  That would include at
> least 1.0 native packages with Debian revisions.

On a technical level, there is not much point in ruling that something
should not have happened. It did happen. The strongest thing that would
be possible here would be to rule that these bugs should be closed.

> Part II - bless continued use of 1.0-with-diff, for now at least:
> 
>  4. Declare that sometimes the use of 1.0-with-diff can be the best
> tradeoff between different considerations.  In particular,
> because 1.0 is the only format which botH:
>  (a) Optimises bandwidth and storage by reusing the upstream 
>  data when it hasn't changed.
>  (b) Avoids polluting the working tree (package source code)
>  with [patches], which cause trouble especially with
>git-based workflows.

I note that the 1.0-with-diff format does not meet my perception of your
earlier requirement: Being able to represent any source tree. I also
note that it is possible to choose a different conversion mechanism than
dpkg-source -b between working tree and .dsc that does not incur your
reported pollution. In particular, you do have experience in
implementing such conversions as is evidenced in dgit and such
conversions do exist.

Your request also touches the question of the preferred form for
modification. That used to be the .dsc, but for the way you use the
archive, it merely is an export for a preferred form for modification
that resides elsewhere. I do not think we have consensus for this view
either.

I also caution that whenever we perform a fundamental change, some
things that used to work well, stop working well. Many improvements
regress on other aspects that are deemed niche features. I think it is
fair to say that use of 1.0 packaging formats is a niche at this point.
Consistency has value, but it comes with a price of making some niche
features impossible. We are discussing a trade-off between consistency
and niche features here.

Helmut

Bug#1007717: Native source package format with non-native version

[Ian Jackson]

A friend suggested some information that I could usefully provide:

Firstly, I posted to -devel a summary of why format 3.0 is not
pareto-better than 1.0.  I have c it below.

Secondly, I was asked if there was a bug against src:dpkg asking for
"3.0 (native)" to allow packages with Debian revisions.  I don't think
there is - at least, I didn't find one.  I do remember this coming up
before, but I don't seem to be able to find a record of it now.

Sean Whitton writes ("Re: Bug#1007717: Native source package format with 
non-native version"):
> I think that you are asking us to use 6.1.1/6.1.2 for most of this, but
> for I.3 are you asking for us to use 6.1.4 or 6.1.5?  "Request" seems
> more like 6.1.5 but I would like to check.

I think you mean "for I.2 are you asking".  I am not asking for the TC
to overrule the dpkg maintainer under 6.1.4.

Let me requote myself with annotations:

> On Tue 15 Mar 2022 at 04:29pm GMT, Ian Jackson wrote:
> > Please:
> >
> > Part I - belss continued use of 1.0 native format, for now at least:
> >
> >  1. Declare explicitly that there is nothing wrong with a package with
> > a native format, but a non-native version number.

 6.1(1) Decide on any matter of technical policy.

> >  2. Request that the dpkg maintainer relax the restriction which
> > prevents the use of 3.0 native with Debian revision.

 6.1(5) Offer advice.

> >  3. Consequently, declare that the recent MBF on this topic ought not
> > to have been filed against packages where simply changing the
> > source format does not currently work.  That would include at
> > least 1.0 native packages with Debian revisions.

 6.1(5) Offer advice.

> > Part II - bless continued use of 1.0-with-diff, for now at least:
> >
> >  4. Declare that sometimes the use of 1.0-with-diff can be the best
> > tradeoff between different considerations.  In particular,
> > because 1.0 is the only format which botH:
> >  (a) Optimises bandwidth and storage by reusing the upstream
> >  data when it hasn't changed.
> >  (b) Avoids polluting the working tree (package source code)
> >  with [patches], which cause trouble especially with
> >  git-based workflows.

 6.1(5) Offer advice.

> >  5. Consequently, declare that the recent MBF on this topic ought not
> > to have been filed against 1.0 with diff packages, at least
> > without some further filter.

 6.1(5) Offer advice.

Thanks,
Ian.


From: Ian Jackson 
To: Sean Whitton 
Cc: Russ Allbery ,
debian-de...@lists.debian.org
Subject: Re: proposed MBF: packages still using source format 1.0
Date: Wed, 9 Mar 2022 16:38:43 +

Sean Whitton writes ("Re: proposed MBF: packages still using source format 
1.0"):
> On Wed 09 Mar 2022 at 01:08pm +01, Lucas Nussbaum wrote:
> > On 08/03/22 at 17:33 -0700, Sean Whitton wrote:
> >> Lucas, as I've had a lot to do with these git workflows and have
> >> probably done the most work documenting them, I can help with any
> >> specific follow-up questions you might have.
> >
> > Thanks!
> >
> > So the main question I think I have is:
> >
> > can we find a middleground where the git workflows don't require staying
> > with 1.0? Even if that means switching to 3.0 (quilt) using the
> > single-debian-patch approach?
> 
> dgit-maint-merge(7) works with single-debian-patch and that's what I
> use.  But it is not clear to me that there are any advantages at all to
> that over 3.0 (quilt) with single-debian-patch?

The situation here is complicated.


The tl;dr is that

 * there are several situations where 1.0-native is the best answer,
 * there are several situations where 1.0-with-diff is the best answer,

The root cause of both of these situations is that 3.0, sadly,
is not always better in every respect than 1.0.


1. Why is 1.0-without-diff not always worse than 3.0 (native) ?

1.0 native is sometimes better than 3.0 (native) because dpkg-source
refuses to build a 3.0 native package with a Debian revision in its
version number.

This prohibition exists solely because of a doctrinal objection to
native-format packages with Debian revisions.  There is no technical
reason why this restriction could not be lifted.  I sometimes upload
this way and I have never had anyone report problems[1] with it.

IMO there is nothing wrong with native format packages with Debian
revisions.  They work just fine.  For a small paockage, this is often
a good choice, because it avoids dealing with patches at all.

For anything but a small package, use of diffs is needed as a storage
and distribution optimisation.


2. Why is 1.0-with-diff not always worse than some 3.0 format ?

1.0-with-diff has the following adv

Bug#1007717: Native source package format with non-native version

[Sam Hartman]

> "Ian" == Ian Jackson  writes:
Ian>  3. Consequently, declare that the recent MBF on this topic
Ian> ought not to have been filed against packages where simply
Ian> changing the source format does not currently work.  That would
Ian> include at least 1.0 native packages with Debian revisions.

I'm a little confused.
My reading of the debian-devel discussion was that Lucas had already
modified the MBF in a manner consistent with what you're asking for.
It looks like there was a bug in the criteria he used, and it sounds
like he already agrees that a few bugs were filed that should not have
been.
It sounds like he's actively working to fix that.


I think asking the TC to get involved here is a great idea on the
other issues you ask about.
I'm just not sure there is a contraversyfor the TC on the specific
case of the MBF itself.

Bug#1007717: Native source package format with non-native version

[Sean Whitton]

Hello,

On Tue 15 Mar 2022 at 04:29pm GMT, Ian Jackson wrote:

> Please:
>
> Part I - belss continued use of 1.0 native format, for now at least:
>
>  1. Declare explicitly that there is nothing wrong with a package with
> a native format, but a non-native version number.
>
>  2. Request that the dpkg maintainer relax the restriction which
> prevents the use of 3.0 native with Debian revision.
>
>  3. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against packages where simply changing the
> source format does not currently work.  That would include at
> least 1.0 native packages with Debian revisions.
>
> Part II - bless continued use of 1.0-with-diff, for now at least:
>
>  4. Declare that sometimes the use of 1.0-with-diff can be the best
> tradeoff between different considerations.  In particular,
> because 1.0 is the only format which botH:
>  (a) Optimises bandwidth and storage by reusing the upstream
>  data when it hasn't changed.
>  (b) Avoids polluting the working tree (package source code)
>  with [patches], which cause trouble especially with
>git-based workflows.
>
>  5. Consequently, declare that the recent MBF on this topic ought not
> to have been filed against 1.0 with diff packages, at least
> without some further filter.

I think that you are asking us to use 6.1.1/6.1.2 for most of this, but
for I.3 are you asking for us to use 6.1.4 or 6.1.5?  "Request" seems
more like 6.1.5 but I would like to check.

-- 
Sean Whitton

Bug#1007717: Native source package format with non-native version

[Ian Jackson]

Package: tech-ctte

(Sorry for the resend, this should have gone to the BTS the first
time; have fixed a slip in the wording.)

Please:

Part I - belss continued use of 1.0 native format, for now at least:

 1. Declare explicitly that there is nothing wrong with a package with
a native format, but a non-native version number.

 2. Request that the dpkg maintainer relax the restriction which
prevents the use of 3.0 native with Debian revision.

 3. Consequently, declare that the recent MBF on this topic ought not
to have been filed against packages where simply changing the
source format does not currently work.  That would include at
least 1.0 native packages with Debian revisions.

Part II - bless continued use of 1.0-with-diff, for now at least:

 4. Declare that sometimes the use of 1.0-with-diff can be the best
tradeoff between different considerations.  In particular,
because 1.0 is the only format which botH:
 (a) Optimises bandwidth and storage by reusing the upstream 
 data when it hasn't changed.
 (b) Avoids polluting the working tree (package source code)
 with [patches], which cause trouble especially with
 git-based workflows.

 5. Consequently, declare that the recent MBF on this topic ought not
to have been filed against 1.0 with diff packages, at least
without some further filter.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.