Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Tuesday, 3 January 2017 at 20:07:59 UTC, Andrei Alexandrescu 
wrote:
Arguing that local imports have been successful so we should 
simply do
more of it is not a good argument, as there comes a point of 
diminishing
returns.  You need to show that there are still worthile gains 
to be
made from changing the language again, which is why I want to 
benchmark

this feature with Walter before deciding.


You can be reasonably certain the benchmarks will improve as 
projected - starting in proportion to the the transitive fanout 
of top-level imports (10.5x for the standard library) and going 
down thereafter with the size, complexity, and actual 
dependencies of the module being compiled. All in all 
measurable but not dramatic. The same improvement will be 
brought about by lazy imports, and it won't be the deal 
maker/breaker. If you're waiting for the numbers to get 
convinced of anything, you already consider DIP1005 useless.


I don't need dramatic, a solid win is fine. A small 10% or less 
improvement in the speed of the import phase of compilation 
probably isn't worth it.



Rather, the
argument is that local imports mostly solved this problem, so 
why bother
adding new syntax for the dozen remaining symbols from 2-3 
modules that

are commonly used in template constraints?


I understand. It is my humble opinion that we are "mostly 
pregnant" for as long as we require top-level imports. The real 
change of phase occurs when there are ZERO imports at top 
level. That's the prize DIP1005 is after.


If you believe the benchmarks won't be dramatically different, I 
wonder why you think there will be a phase change.


In the end, this is a minor DIP that is easily bikeshedded, as 
everybody

can grasp it and have an opinion on it.  I have refrained from
commenting recently because I will let benchmarking settle it 
for me.

Obviously, that won't suffice for others.


I do agree that if framed as a modest improvement in build 
economics, it is quite unimportant. But that's a problem with 
the DIP; its main strength is better encapsulation, which is no 
small thing.


Encapsulation is not an end in itself.  You have listed four main 
advantages of this change in the DIP.  I have noted that I don't 
find the first three relating to reasoning and refactoring worth 
adding syntax for, while admitting that may be subjective.  The 
fourth talks about scalability, which I interpreted as either 
compilation speed or some other measurable gain, hence my asking 
for benchmarks.


If scalability refers to something else- it can't be reasoning or 
refactoring as those are listed as separate advantages- I'd like 
to hear what it is.  If it's measurable, I'd like to measure it 
first.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timon Gehr via Digitalmars-d]

On 05.01.2017 01:51, Andrei Alexandrescu wrote:

On 12/31/16 9:43 AM, Timon Gehr wrote:

On an unrelated note: I'm still not a fan of the with(import) syntax as
it morally promotes a lack of turtles (even if not technically so).


Could you please provide more detail? Thanks! -- Andrei


The declaration

with(import foo){ ... }

looks like an orthogonal combination of some import expression and the usual

with(foo){ ... }

statement. (This is true for all other statements and expressions with 
similar syntax.)


However, this is not in fact true here, the two constructs have 
different scoping rules.


Hence I think that the 'with(import foo){ ... }'-syntax would be better 
split into two orthogonal features:


1. allow 'import foo' as an expression that evaluates to the 
corresponding module symbol.


2. add 'static with' that is basically like 'with' but is a declaration 
and has different scoping rules.



I.e., my objection is that 'with' should not become 'static' just 
because it is applied to an import expression.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/31/16 9:43 AM, Timon Gehr wrote:

On an unrelated note: I'm still not a fan of the with(import) syntax as
it morally promotes a lack of turtles (even if not technically so).


Could you please provide more detail? Thanks! -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[deadalnix via Digitalmars-d]

On Wednesday, 4 January 2017 at 15:56:13 UTC, Timon Gehr wrote:
I don't fully agree. Nested imports, the way they have been 
implemented, pose a new symbol hijacking hazard.




I'd argue this was an existing bug in import handling. This is 
why I like to have very orthogonal definitions.


It adds basically no implementation complexity [1]. I consider 
the benefit real, but minor enough to oppose the DIP based on 
its wacky syntax.


[1] Both static if and static foreach (once it lands) need the 
same kind of scoping rules.


I know about [1], this is why I did not mentioned it. I don't 
really mind about implementation complexity, I care about 
complexity of the definition. For the following reasons:
 - If the implementation may be complex, it can be isolated 
and/or abstracted away.
 - Interaction with other parts of the language are more 
predictable, including future parts that do not exists yet.
 - It obviate the explosion of trivia experienced devs needs to 
know to use the language.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timon Gehr via Digitalmars-d]

On 04.01.2017 16:00, deadalnix wrote:


Nested import are a language simplification. Declaration can appear
anywhere, import is a declaration, the fact that import couldn't appear
anywhere was an arbitrary limitation, and removing it makes the language
simpler. As such, the burden of proof is on maintaining the limitation
rather than removing it.
...


I don't fully agree. Nested imports, the way they have been implemented, 
pose a new symbol hijacking hazard.


(A symbol hijacking anecdote (not directly related): 
https://github.com/tgehr/d-compiler/pull/1#discussion-diff-89697186L85 )



This DIP is a language addition. Therefore, contrary to nested or lazy
import, the burden of proof is on it. This DIP should be considered as
follow: how much complexity does it add and how much benefit does it
bring, compared to alternatives.


It adds basically no implementation complexity [1]. I consider the 
benefit real, but minor enough to oppose the DIP based on its wacky syntax.



[1] Both static if and static foreach (once it lands) need the same kind 
of scoping rules.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[deadalnix via Digitalmars-d]

There are quite a few fallacies in there.

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu 
wrote:
Regarding the ongoing doubts about the advantages of inline 
imports: they are first and foremost a completion of the nested 
import feature. As such, most, if not all, arguments against 
inline imports apply equally to nested imports. Come to think 
of it, lazy imports vs nested imports:


* same improvement in compilation speed? check
* no language changes? check
* no nasty bugs in the aftermath (such as the infamous 
https://issues.dlang.org/show_bug.cgi?id=10378)? check

* scalable builds? check

Yet local imports are overwhelmingly superior to lazy imports 
because of one thing: they localize dependencies. They 
introduce modularity and its ancillary perks (fast and scalable 
builds, easier review and refactoring) not by engineering, but 
by organically placing dependencies spatially with their 
dependents. (The scope statement does the same thing with 
temporal dependencies.) That the DIP does not make it clear 
that it is a necessary and sufficient extension of local 
imports is a problem with it.




There is a major difference with this DIP.

Lazy import is not a language change, but a compiler 
implementation detail. As such, it doesn't require a DIP or 
anything specific.


Nested import are a language simplification. Declaration can 
appear anywhere, import is a declaration, the fact that import 
couldn't appear anywhere was an arbitrary limitation, and 
removing it makes the language simpler. As such, the burden of 
proof is on maintaining the limitation rather than removing it.


This DIP is a language addition. Therefore, contrary to nested or 
lazy import, the burden of proof is on it. This DIP should be 
considered as follow: how much complexity does it add and how 
much benefit does it bring, compared to alternatives.


The obvious benefit is localizing dependencies. I think I'm not 
too far off track by considering most of the speedup and scalable 
build can be achieved with lazy import and, while I'm sure there 
are example where this is superior, we are talking marginal gains 
as lazy and nested imports squeezed most of the juice already.


The cost is the language addition. The first obvious improvement 
that can be made to this DIP to reduce its cost is to not 
introduce a new syntax. As such, the addition is limited to 
allowing the existing syntax in a new place rather than adding a 
whole new syntax for imports.


I like the extra expressivity. I'm not 100% convinced it is worth 
the extra cost, but the more the cost is reduced, the more 
rational it seems to me that this option should be pursued.


I now am really glad we slipped local imports before the 
formalization of DIPs. The feature could have been easily 
demeaned out of existence.




Good you also notice how broken the DIP process is.

One suggestion: let's keep the DIP describing the change to be 
made. Some examples are fine to illustrate, but it is not the 
DIp's purpose to be easy to understand or expand too much in 
argumentation, or it'll be useless as a spec document, and trying 
to have the DIP be a spec, a tutorial, a essay on why the 
feature, and so on just lead to endless rewriting lead to 
perpetual motion but no progress.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Dominikus Dittes Scherkl via Digitalmars-d]

On Tuesday, 3 January 2017 at 12:57:22 UTC, Andrei Alexandrescu 
wrote:

On 01/03/2017 02:19 AM, Dominikus Dittes Scherkl wrote:
On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu 
wrote:
DIP1005 gives consideration to the speed of compilation 
aspect in
larger proportion than speed's importance; the first and 
foremost
benefit of DIP1005 is it closes the gap on dependency 
encapsulation,

which had been very successfully narrowed by local imports.
I love that idea. But I still can't see why this requires any 
new
syntax. Simply extending the scope of local inports to include 
the

function header is enough.


I'll mention this possibility in the DIP.

Good.


Only for .di-generation it may be useful to move all local 
imports to
the declaration (maybe with this new syntax "with" before it) 
- but that
should be done with ALL local imports, because today the 
.di-files are

incomplete and will stay so if the new syntax is introduced but
"old-style" local imports still valid and not exported to the 
.di.

Or the old local imports become deprecated together with the
introduction of the new "with" syntax and vanish soon after 
that.


Local imports don't pertain to the interface, they are an 
implementation detail. Why would those be made a part of the 
.di?
.di are needed to ship with a library. If a function locally 
imports some type, the library is dependant on that import, so if 
I want to use the library, I have to install all stuff it depends 
on too. And to find out, what exactly the library depends on the 
.di is the place I look, so there need to be mentioned each and 
every import - no matter how deeply local that may be. If that is 
not the case it renders .di-files completely useless to me (as 
they are at the moment), because I need to find out the 
dependencies somewhere else (e.g. in some documentation of the 
library).
While it is likely that the dependencies of a library may be 
documented somewhere, this is not guaranteed. But .di-files are 
guaranteed to ship with a lib because else it cannot link - at 
least if it contains any templates, which is about 100% sure for 
a language like D. So I would like to see local imports in the 
.di-file, even if they are not strictly needed to compile because 
the imported types are not exposed in the function signature.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 01/03/2017 03:07 PM, Andrei Alexandrescu wrote:

If you're waiting for the numbers to get convinced of anything, you
already consider DIP1005 useless.


Eh, sorry, this came out wrong - as if I'm trying to put words in 
someone's mouth. I meant to say: "If you're waiting for the numbers to 
get convinced of anything, they won't be overwhelming so you may already 
write off DIP1005 as useless." -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 01/03/2017 04:28 AM, Paulo Pinto wrote:

Looking from the outside, and watching what was reached from 2016
roadmap, it is clear the DIPs evaluated thus dar aren't about fixing the
library or runtime issues that prevent D's adoption at large as a
systems programming language.

Meanwhile Swift, Go and Rust have a clear roadmap how their future is
supposed to look like, and drive just in that direction, with C++ taking
all remaining good D ideas.

This DIP discussion and the latest ones about splitting the runtime
again, don't do anything to earn D any credibility it might still have
left.


Thanks for taking the time to write this. All languages have an 
improvement process, which runs as a background task that is not always 
correlated with the current principal objectives of the language's 
leadership. We have recently improved our DIP process, but positive 
examples of compelling DIPs are missing. We were facing the problem that 
upcoming DIPs, if unguided, could create busywork for the leadership on 
trivial matters. Whether approved, rejected, or put on hold, DIP1005 
will serve as a yardstick for future DIPs regarding the necessary work 
involved in defining the problem, investigating it as rigorously as 
possible, comparing with alternatives, and defining the feature.


Walter and I do agree that the DIP is ahead of its time because (a) 
there are more pressing and more important matters to keep me busy, and 
(b) encapsulation in D is a larger topic that we haven't yet put front 
and center because of (a).


For DIP1005 I invite interested collaborators to contact me about future 
iterations; it is important for the matter of encapsulation and also for 
the meta-challenge of setting an example for other DIPs to follow. Thanks.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 01/03/2017 03:10 AM, Joakim wrote:

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu wrote:

Regarding the ongoing doubts about the advantages of inline imports:
they are first and foremost a completion of the nested import feature.
As such, most, if not all, arguments against inline imports apply
equally to nested imports. Come to think of it, lazy imports vs nested
imports:

* same improvement in compilation speed? check
* no language changes? check
* no nasty bugs in the aftermath (such as the infamous
https://issues.dlang.org/show_bug.cgi?id=10378)? check
* scalable builds? check

Yet local imports are overwhelmingly superior to lazy imports because
of one thing: they localize dependencies. They introduce modularity
and its ancillary perks (fast and scalable builds, easier review and
refactoring) not by engineering, but by organically placing
dependencies spatially with their dependents. (The scope statement
does the same thing with temporal dependencies.) That the DIP does not
make it clear that it is a necessary and sufficient extension of local
imports is a problem with it.

I now am really glad we slipped local imports before the formalization
of DIPs. The feature could have been easily demeaned out of existence.


Except that almost nobody has argued against local imports.


I don't have time to research this, but my recollection is that at least 
some framed the bugs regarding lookups as a fundamental problem of local 
imports, not a simple matter of getting it right.


Overall, yes, local imports have been a success (really saving 
scalability of large project builds which looked pretty bleak at a 
time), which should increase trust in the authors of the feature... 
hmmm... :o)



Rather, the
argument is that local imports mostly solved this problem, so why bother
adding new syntax for the dozen remaining symbols from 2-3 modules that
are commonly used in template constraints?


I understand. It is my humble opinion that we are "mostly pregnant" for 
as long as we require top-level imports. The real change of phase occurs 
when there are ZERO imports at top level. That's the prize DIP1005 is after.



Arguing that local imports have been successful so we should simply do
more of it is not a good argument, as there comes a point of diminishing
returns.  You need to show that there are still worthile gains to be
made from changing the language again, which is why I want to benchmark
this feature with Walter before deciding.


You can be reasonably certain the benchmarks will improve as projected - 
starting in proportion to the the transitive fanout of top-level imports 
(10.5x for the standard library) and going down thereafter with the 
size, complexity, and actual dependencies of the module being compiled. 
All in all measurable but not dramatic. The same improvement will be 
brought about by lazy imports, and it won't be the deal maker/breaker. 
If you're waiting for the numbers to get convinced of anything, you 
already consider DIP1005 useless.



Allow me to make an appeal regarding the review of any DIP. There
seems to be a tendency of some reviewers to get attached and
emotionally invested to their opinion, to the extent they'd be hurt by
being "wrong" and would go to great lengths to argue they're "right".
This has obvious negative effects on the entire process. Please don't.
There's no loss of face to worry about. The only commitment we all
should have is to the good of the D language. If DIP1005 reaches the
conclusion of its own uselessness, I'd be the first one to write it up
and close the PR.


We could level this analysis back at you: you consider this DIP so
"obvious" that you are not engaging with our concerns, making flip,
incorrect remarks about how we would have bikeshedded local imports also.


I apologize it my remarks seem flippant, though I honestly have 
difficulty finding evidence of that. All I did was note that many of the 
arguments pitching lazy imports as a better solution than DIP1005 
(except of course for the Amdahl one) apply directly to local imports.



In the end, this is a minor DIP that is easily bikeshedded, as everybody
can grasp it and have an opinion on it.  I have refrained from
commenting recently because I will let benchmarking settle it for me.
Obviously, that won't suffice for others.


I do agree that if framed as a modest improvement in build economics, it 
is quite unimportant. But that's a problem with the DIP; its main 
strength is better encapsulation, which is no small thing.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Tue, 03 Jan 2017 08:10:04 +, Joakim wrote:
> Except that almost nobody has argued against local imports. Rather, the
> argument is that local imports mostly solved this problem, so why bother
> adding new syntax for the dozen remaining symbols from 2-3 modules that
> are commonly used in template constraints?

That was my misunderstanding, and it only applied to the speed argument 
anyway.

This is usable with top-level functions, top-level templates, and classes 
with inheritance.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 01/03/2017 02:19 AM, Dominikus Dittes Scherkl wrote:

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu wrote:

DIP1005 gives consideration to the speed of compilation aspect in
larger proportion than speed's importance; the first and foremost
benefit of DIP1005 is it closes the gap on dependency encapsulation,
which had been very successfully narrowed by local imports.

I love that idea. But I still can't see why this requires any new
syntax. Simply extending the scope of local inports to include the
function header is enough.


I'll mention this possibility in the DIP.


Only for .di-generation it may be useful to move all local imports to
the declaration (maybe with this new syntax "with" before it) - but that
should be done with ALL local imports, because today the .di-files are
incomplete and will stay so if the new syntax is introduced but
"old-style" local imports still valid and not exported to the .di.
Or the old local imports become deprecated together with the
introduction of the new "with" syntax and vanish soon after that.


Local imports don't pertain to the interface, they are an implementation 
detail. Why would those be made a part of the .di?



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Daniel N via Digitalmars-d]

On Tuesday, 3 January 2017 at 10:18:57 UTC, Daniel N wrote:

static if(is(isArray(...)))
{
  import std.range : join;
  join(...);
}


PS Actually double {} is needed for the 'alternate' style to be 
meaningful, preventing the import from bleeding into the outer 
scope.


static if(isArray(...))
{{
  import std.range : join;
  join(...);
}}

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Daniel N via Digitalmars-d]

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu 
wrote:


No worries, I'll make a pass (a rewrite, really) taking all 
feedback into account.


Are all the proposals mutually exclusive? When it comes to 
declarations, the DIP actually won me over! However I was hoping 
for it to solve a different issue inside the body with local 
imports. I tend to write short helper functions, with the result 
that in the common case, an imported symbol is often used only 
once per function.


Currently I consider it good style to list all imports in the 
beginning of a scope, but based on past experience with C89, this 
is not entirely optimal, however Timon Gehr's proposal could 
solve it.


import and directly invoke:
import std.traits : isArray(...)

===
[Proposed Style - analogous to C99]
body
{
  static if(...)
import std.range : zip(...);
  static if(import std.traits : isArray(...)))
import std.range : join(...);
}
===
[Old Style - analogous to C89 - Suffers from "unused" imports]
body
{
  import std.range : zip, join;
  import std.traits : isArray;

  static if(...)
zip(...)
  static if(isArray(...)))
join(...)
}
===
[Alternate Style - Suffers from DRY and can't limit scope of 
isArray]

body
{
  import std.traits : isArray;

  static if(...)
  {
import std.range : zip;
zip(...);
  }
  static if(is(isArray(...)))
  {
import std.range : join;
join(...);
  }
}

C89 vs C99 style I was referring to if it wasn't clear.
===
[C89]
void foo(void)
{
#ifdef ... /* needed to silence unused variable warnings */
  int x;
#endif
... lots of code ...
#ifdef ...
  x = bar();
#endif
}

[C99]
void foo(void)
{
#ifdef ...
  int x = bar();
#endif
}
===

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Paulo Pinto via Digitalmars-d]

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu 
wrote:

On 12/31/2016 12:20 PM, Martin Nowak wrote:
On Saturday, 24 December 2016 at 10:54:08 UTC, Stefan Koch 
wrote:
If that were made more lazy, we could import half of the 
world with

noticing impact.

(Which espcially in std.traits, would not make that much of a
difference since every template in there depends on nearly 
every other

template in there)


Also the established technique of serializing precompiled AST 
(after

semantic3) of modules to a cache should be applicable as well.

Cross-posting from
https://github.com/dlang/DIPs/pull/51#issuecomment-269107966, 
b/c it

wasn't answered yet.


No worries, I'll make a pass (a rewrite, really) taking all 
feedback into account.


Were any other means considered? This is proposing to add 
plenty of
additional annotations only to speed up compilation, but none 
of the

classical tools for pre-compilation were assessed.
Since D's modules don't have the header problem, even 
pre-compilation

and reuse of semantic3 should be possible, or not?


DIP1005 gives consideration to the speed of compilation aspect 
in larger proportion than speed's importance; the first and 
foremost benefit of DIP1005 is it closes the gap on dependency 
encapsulation, which had been very successfully narrowed by 
local imports. However, the DIP will keep the experiments and 
results on speed measurements because they are relevant to it 
and any related methods of lazily loading of modules.


Regarding the ongoing doubts about the advantages of inline 
imports: they are first and foremost a completion of the nested 
import feature. As such, most, if not all, arguments against 
inline imports apply equally to nested imports. Come to think 
of it, lazy imports vs nested imports:


* same improvement in compilation speed? check
* no language changes? check
* no nasty bugs in the aftermath (such as the infamous 
https://issues.dlang.org/show_bug.cgi?id=10378)? check

* scalable builds? check

Yet local imports are overwhelmingly superior to lazy imports 
because of one thing: they localize dependencies. They 
introduce modularity and its ancillary perks (fast and scalable 
builds, easier review and refactoring) not by engineering, but 
by organically placing dependencies spatially with their 
dependents. (The scope statement does the same thing with 
temporal dependencies.) That the DIP does not make it clear 
that it is a necessary and sufficient extension of local 
imports is a problem with it.


I now am really glad we slipped local imports before the 
formalization of DIPs. The feature could have been easily 
demeaned out of existence.


Allow me to make an appeal regarding the review of any DIP. 
There seems to be a tendency of some reviewers to get attached 
and emotionally invested to their opinion, to the extent they'd 
be hurt by being "wrong" and would go to great lengths to argue 
they're "right". This has obvious negative effects on the 
entire process. Please don't. There's no loss of face to worry 
about. The only commitment we all should have is to the good of 
the D language. If DIP1005 reaches the conclusion of its own 
uselessness, I'd be the first one to write it up and close the 
PR.



Thanks,

Andrei


Allow me just to share a worthless outsider opinion.

I never contributed anything worthwhile and decided it was better 
to just focus on JVM, .NET languages., alongside C++, as those 
are the skills I get paid for, thus stop polluting D forums.


Looking from the outside, and watching what was reached from 2016 
roadmap, it is clear the DIPs evaluated thus dar aren't about 
fixing the library or runtime issues that prevent D's adoption at 
large as a systems programming language.


Meanwhile Swift, Go and Rust have a clear roadmap how their 
future is supposed to look like, and drive just in that 
direction, with C++ taking all remaining good D ideas.


This DIP discussion and the latest ones about splitting the 
runtime again, don't do anything to earn D any credibility it 
might still have left.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Tue, 03 Jan 2017 07:19:41 +, Dominikus Dittes Scherkl wrote:

> On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu wrote:
>> DIP1005 gives consideration to the speed of compilation aspect in
>> larger proportion than speed's importance; the first and foremost
>> benefit of DIP1005 is it closes the gap on dependency encapsulation,
>> which had been very successfully narrowed by local imports.
> I love that idea. But I still can't see why this requires any new
> syntax. Simply extending the scope of local inports to include the
> function header is enough.

You mean:

  Socket connect()
  {
import std.socket;
return new Socket();
  }

Which should already work if you used auto for the return type.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu 
wrote:
Regarding the ongoing doubts about the advantages of inline 
imports: they are first and foremost a completion of the nested 
import feature. As such, most, if not all, arguments against 
inline imports apply equally to nested imports. Come to think 
of it, lazy imports vs nested imports:


* same improvement in compilation speed? check
* no language changes? check
* no nasty bugs in the aftermath (such as the infamous 
https://issues.dlang.org/show_bug.cgi?id=10378)? check

* scalable builds? check

Yet local imports are overwhelmingly superior to lazy imports 
because of one thing: they localize dependencies. They 
introduce modularity and its ancillary perks (fast and scalable 
builds, easier review and refactoring) not by engineering, but 
by organically placing dependencies spatially with their 
dependents. (The scope statement does the same thing with 
temporal dependencies.) That the DIP does not make it clear 
that it is a necessary and sufficient extension of local 
imports is a problem with it.


I now am really glad we slipped local imports before the 
formalization of DIPs. The feature could have been easily 
demeaned out of existence.


Except that almost nobody has argued against local imports.  
Rather, the argument is that local imports mostly solved this 
problem, so why bother adding new syntax for the dozen remaining 
symbols from 2-3 modules that are commonly used in template 
constraints?


Arguing that local imports have been successful so we should 
simply do more of it is not a good argument, as there comes a 
point of diminishing returns.  You need to show that there are 
still worthile gains to be made from changing the language again, 
which is why I want to benchmark this feature with Walter before 
deciding.


Allow me to make an appeal regarding the review of any DIP. 
There seems to be a tendency of some reviewers to get attached 
and emotionally invested to their opinion, to the extent they'd 
be hurt by being "wrong" and would go to great lengths to argue 
they're "right". This has obvious negative effects on the 
entire process. Please don't. There's no loss of face to worry 
about. The only commitment we all should have is to the good of 
the D language. If DIP1005 reaches the conclusion of its own 
uselessness, I'd be the first one to write it up and close the 
PR.


We could level this analysis back at you: you consider this DIP 
so "obvious" that you are not engaging with our concerns, making 
flip, incorrect remarks about how we would have bikeshedded local 
imports also.


In the end, this is a minor DIP that is easily bikeshedded, as 
everybody can grasp it and have an opinion on it.  I have 
refrained from commenting recently because I will let 
benchmarking settle it for me.  Obviously, that won't suffice for 
others.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Dominikus Dittes Scherkl via Digitalmars-d]

On Monday, 2 January 2017 at 21:23:19 UTC, Andrei Alexandrescu 
wrote:
DIP1005 gives consideration to the speed of compilation aspect 
in larger proportion than speed's importance; the first and 
foremost benefit of DIP1005 is it closes the gap on dependency 
encapsulation, which had been very successfully narrowed by 
local imports.
I love that idea. But I still can't see why this requires any new 
syntax. Simply extending the scope of local inports to include 
the function header is enough.
Only for .di-generation it may be useful to move all local 
imports to the declaration (maybe with this new syntax "with" 
before it) - but that should be done with ALL local imports, 
because today the .di-files are incomplete and will stay so if 
the new syntax is introduced but "old-style" local imports still 
valid and not exported to the .di.
Or the old local imports become deprecated together with the 
introduction of the new "with" syntax and vanish soon after that.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/31/2016 12:20 PM, Martin Nowak wrote:

On Saturday, 24 December 2016 at 10:54:08 UTC, Stefan Koch wrote:

If that were made more lazy, we could import half of the world with
noticing impact.

(Which espcially in std.traits, would not make that much of a
difference since every template in there depends on nearly every other
template in there)


Also the established technique of serializing precompiled AST (after
semantic3) of modules to a cache should be applicable as well.

Cross-posting from
https://github.com/dlang/DIPs/pull/51#issuecomment-269107966, b/c it
wasn't answered yet.


No worries, I'll make a pass (a rewrite, really) taking all feedback 
into account.



Were any other means considered? This is proposing to add plenty of
additional annotations only to speed up compilation, but none of the
classical tools for pre-compilation were assessed.
Since D's modules don't have the header problem, even pre-compilation
and reuse of semantic3 should be possible, or not?


DIP1005 gives consideration to the speed of compilation aspect in larger 
proportion than speed's importance; the first and foremost benefit of 
DIP1005 is it closes the gap on dependency encapsulation, which had been 
very successfully narrowed by local imports. However, the DIP will keep 
the experiments and results on speed measurements because they are 
relevant to it and any related methods of lazily loading of modules.


Regarding the ongoing doubts about the advantages of inline imports: 
they are first and foremost a completion of the nested import feature. 
As such, most, if not all, arguments against inline imports apply 
equally to nested imports. Come to think of it, lazy imports vs nested 
imports:


* same improvement in compilation speed? check
* no language changes? check
* no nasty bugs in the aftermath (such as the infamous 
https://issues.dlang.org/show_bug.cgi?id=10378)? check

* scalable builds? check

Yet local imports are overwhelmingly superior to lazy imports because of 
one thing: they localize dependencies. They introduce modularity and its 
ancillary perks (fast and scalable builds, easier review and 
refactoring) not by engineering, but by organically placing dependencies 
spatially with their dependents. (The scope statement does the same 
thing with temporal dependencies.) That the DIP does not make it clear 
that it is a necessary and sufficient extension of local imports is a 
problem with it.


I now am really glad we slipped local imports before the formalization 
of DIPs. The feature could have been easily demeaned out of existence.


Allow me to make an appeal regarding the review of any DIP. There seems 
to be a tendency of some reviewers to get attached and emotionally 
invested to their opinion, to the extent they'd be hurt by being "wrong" 
and would go to great lengths to argue they're "right". This has obvious 
negative effects on the entire process. Please don't. There's no loss of 
face to worry about. The only commitment we all should have is to the 
good of the D language. If DIP1005 reaches the conclusion of its own 
uselessness, I'd be the first one to write it up and close the PR.



Thanks,

Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Mon, 02 Jan 2017 00:47:41 +, pineapple wrote:

> On Saturday, 31 December 2016 at 17:02:55 UTC, Chris Wright wrote:
>>> This extension removes an unforced limitation of the current with
>>> syntax (allows it to occur at top level)
>>
>> In other words, another aspect of this DIP is that I can write:
>>
>>   module foo;
>>   static import std.traits;
>>   static import bar;
>>   with (std.traits)
>>   {
>> template Foo(T) if (isAbstractClass!T) {}
>>   }
>>   with (bar.SomeEnum)
>>   {
>> enum something = SomeEnumValue;
>>   }
> 
> This is the only expression of the feature I've seen so far that makes
> intuitive sense to me.
> 
> I'm still not sold on it being a worthy addition but, if it were, then
> this is the most promising syntax I've seen so far.

This is a minor part of the proposal that's mentioned as an aside, 
though, and I'm not even sure it was even intentional.

This alone doesn't get the two benefits Andrei is most after -- 
specifically, that you can add the import in place without going to the 
top of the file and that you can move the declaration to another module 
without worrying about imports.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timon Gehr via Digitalmars-d]

On 02.01.2017 01:47, pineapple wrote:

On Saturday, 31 December 2016 at 17:02:55 UTC, Chris Wright wrote:

This extension removes an unforced limitation of the current with
syntax (allows it to occur at top level)


In other words, another aspect of this DIP is that I can write:

  module foo;
  static import std.traits;
  static import bar;
  with (std.traits)
  {
template Foo(T) if (isAbstractClass!T) {}
  }
  with (bar.SomeEnum)
  {
enum something = SomeEnumValue;
  }


This is the only expression of the feature I've seen so far that makes
intuitive sense to me.

I'm still not sold on it being a worthy addition but, if it were, then
this is the most promising syntax I've seen so far.


Should be 'static with'. ('with' already has a meaning.)

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[pineapple via Digitalmars-d]

On Saturday, 31 December 2016 at 17:02:55 UTC, Chris Wright wrote:
This extension removes an unforced limitation of the current 
with syntax (allows it to occur at top level)


In other words, another aspect of this DIP is that I can write:

  module foo;
  static import std.traits;
  static import bar;
  with (std.traits)
  {
template Foo(T) if (isAbstractClass!T) {}
  }
  with (bar.SomeEnum)
  {
enum something = SomeEnumValue;
  }


This is the only expression of the feature I've seen so far that 
makes intuitive sense to me.


I'm still not sold on it being a worthy addition but, if it were, 
then this is the most promising syntax I've seen so far.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Martin Nowak via Digitalmars-d]

On Saturday, 24 December 2016 at 10:54:08 UTC, Stefan Koch wrote:
If that were made more lazy, we could import half of the world 
with noticing impact.


(Which espcially in std.traits, would not make that much of a 
difference since every template in there depends on nearly 
every other template in there)


Also the established technique of serializing precompiled AST 
(after semantic3) of modules to a cache should be applicable as 
well.


Cross-posting from 
https://github.com/dlang/DIPs/pull/51#issuecomment-269107966, b/c 
it wasn't answered yet.


Were any other means considered? This is proposing to add plenty 
of additional annotations only to speed up compilation, but none 
of the classical tools for pre-compilation were assessed.
Since D's modules don't have the header problem, even 
pre-compilation and reuse of semantic3 should be possible, or not?

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Tue, 13 Dec 2016 17:33:24 -0500, Andrei Alexandrescu wrote:

> Destroy.
> 
> https://github.com/dlang/DIPs/pull/51/files
> 
> 
> Andrei

> Inside any function, with (Import ImportList) is a statement that
> introduces a scope.

So I can't write:

  with (import std.stdio, std.conv)
  int count = readln().to!int;
  assert(count > 0);

It is in fact entirely equivalent to write:

  {
import std.stdio, std.conv;
int count = readln().to!int;
  }

I think I'd just put a line in the DIP: for consistency, this syntax 
works anywhere a with statement is currently allowed, but it's not 
recommended to use it inside function bodies in general.

> This extension removes an unforced limitation of the current with
> syntax (allows it to occur at top level)

In other words, another aspect of this DIP is that I can write:

  module foo;
  static import std.traits;
  static import bar;
  with (std.traits)
  {
template Foo(T) if (isAbstractClass!T) {}
  }
  with (bar.SomeEnum)
  {
enum something = SomeEnumValue;
  }

Which *almost*, but not quite, obviates the "you can put the import list 
in the with clause" part of the DIP. It's got all the same benefits for 
reading, but you might occasionally have to jump to the top of the file 
to add a new static import.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 31 Dec 2016 16:41:16 +, Chris Wright wrote:
> Far less ambiguous, but your examples all have unmodified with-imports,
> and you seem less than keen about selective imports.

Correction: *tend to* have unmodified with-imports.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 30 Dec 2016 20:56:54 -0500, Andrei Alexandrescu wrote:

> On 12/30/16 7:32 PM, Chris Wright wrote:
>> On Fri, 30 Dec 2016 22:42:39 +, Stefan Koch wrote:
>>
>>> On Friday, 30 December 2016 at 22:29:18 UTC, Chris Wright wrote:
 * Performance improvements, primarily when a module imports another,
 bulky module for optional functionality.
>>> That is solved by selective imports.
>>
>> Well, not today, but it could be. Same with static imports. Both
>> accomplish this goal better than DIP1005.
> 
> Is this fact or opinion? If the former, could you please point where
> DIP1005 is getting it wrong. Thanks.

Consider:

  with(import std.range.primitives, std.traits)
  {
template Foo(Range) if (isForwardRange!Range && !isArray!Range) {}
template Bar(Range) if (isArray!Range) {}
  }

If I invoke Bar, the compiler must read and parse both 
std.range.primitives and std.traits.

With static imports:

  static import std.range.primitives, std.traits;
  template Foo(Range) if (std.range.primitives.isForwardRange!Range
  && !std.traits.isArray!Range) {}
  template Bar(Range) if (std.traits.isArray!Range) {}

If I invoke Bar, the compiler must read and parse std.traits but not 
std.range.primitives because it is unambiguous that isArray is in 
std.traits, not std.range.primivites.

With selective imports likewise.

You're going to respond that I'm only complaining that people can abuse 
the syntax. Consider that it also happens if I change the code and forget 
to update the imports. Also consider that people are likely to 
incorporate several related definitions that tend to use the same imports 
in order to save typing. "Be more virtuous" isn't a mantra that actually 
leads to better code.

 * Making it easier to locate where things are defined when reading
 code.
>>>
>>> That is solved by selective imports.
>>
>> Static imports do that better than selective imports, though it's more
>> typing.
> 
> So whether that's overall better is not settled, is it? We can't really
> define "better" as whatever each participant believes.

Whether the feature is overall better is not settled. Whether locating 
where symbols are defined is easier with DIP1005 than with static imports 
*is*.

Static imports tell you exactly where the symbol is defined at a glance, 
whereas with DIP1005, you have to look through at least two modules.

>> With selective imports, you can either find the declaration in the
>> current module, or find the symbol in an import list. If it's imported,
>> and the module author is using standard code formatting, you will find
>> the import at the top of the module, which will be fast. This is less
>> true with arbitrarily scoped imports.
>>
>> With DIP1005, you rely on there also being selective or static imports.
>> If the module author didn't use them, then you have to pull out grep.
> 
> DIP1005 allows the declaration to encapsulate its own dependencies. Of
> course if top-level imports are also present, the benefit erodes.

Consider:

  // tons of code
  with (import std.traits)
  template Bar(Range) if (isArray!Range && !isAbstractClass!Range) {}
  // tons more code

I can reasonably guess that the import is required for at least one of 
isArray and isAbstractClass, though for all I know, if I'm unfamiliar 
with std.traits, the import is unnecessary and both isArray and 
isAbstractClass are defined in the current module.

You can fix that with a static or selective import:

  with (import std.traits : isArray, isAbstractClass)
  template Bar(Range) if (isArray!Range && !isAbstractClass!Range) {}

Far less ambiguous, but your examples all have unmodified with-imports, 
and you seem less than keen about selective imports.

>> -- Another thing that just occurred to me: if you're modifying a
>> function signature and that brings in another imported symbol, you
>> don't have to move to the top of the file or struct to add the
>> necessary import. It's a small thing for vim users, who are used to
>> using marks and fast movement commands, but if you're using, say,
>> VSCode, that might be painful.
> 
> That's one of the many benefits of encapsulation.

It is a benefit of this DIP. There are many ways to achieve 
encapsulation, and there are many effects of encapsulation, and most of 
that is irrelevant to this discussion.

FWIW, you could also use current imports immediately before the relevant 
declaration. That would be a bad idea in general because it makes it much 
harder to locate imports when trying to locate a definition. 

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Meta via Digitalmars-d]

On Saturday, 31 December 2016 at 15:05:00 UTC, Adam D. Ruppe 
wrote:

Consider the following:

with (import std.stdio) void process(File input) ;

Let's rewrite that to be:

template process() {
   import std.stdio;
   void process(File input) {

   }
}


I absolutely hate making functions templated if they don't have 
to be. It breaks a whole bunch of things and is not at all 
transparent to the user. You can't take its address, pass it as a 
delegate/function pointer, use certain traits to introspect it 
(e.g. isSomeFunction, ReturnType, etc.), not to mention how 
templates really don't work well with inheritance. This all goes 
out the window when you turn a regular function into a template.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 31 Dec 2016 15:05:00 +, Adam D. Ruppe wrote:
> static import r = std.range.primitives;

For that to pass code review, you'd need a readable name -- perhaps 
'ranges' instead of 'r' -- which makes it more obvious, more searchable, 
and more typing.

> with (import std.stdio) void process(File input) ;
> 
> Let's rewrite that to be:
> 
> template process() {
> import std.stdio;
> void process(File input) {
> 
> }
> }

But that's a lot of typing, so nobody's going to do it. Just like how we 
use normal imports rather than static or selective imports everywhere. 
It's even more verbose than DIP1005.

> BTW it is my opinion that the "one file, one module" rule is a mistake.
> Even with all these things, the declarations are still not *guaranteed*
> to carry their dependencies, since top-level imports still leak in.
> Separate modules don't have that problem, and if we could just define
> several modules in one file, we'd basically destroy this DIP in one
> swift stroke.

That's rather elegant, though I don't see how you'd import a module in a 
file that defines several modules. If they were anonymous and importing 
the file gave you access to all of them, that makes sense. Otherwise it's 
tricky to figure out which sub-file modules exist.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 31 Dec 2016 07:23:14 -0500, Andrei Alexandrescu wrote:

> On 12/30/16 11:10 PM, Chris Wright wrote:
>> On Fri, 30 Dec 2016 21:13:19 -0500, Andrei Alexandrescu wrote:
>>> DIP1005 can't be in the business of arguing that encapsulation is good
>>> by means of examples.
>>
>> Right. I said we should talk about the concrete benefits of the
>> proposal instead of talking about encapsulation.
> 
> Why would the DIP hamstring itself by not discussing its most important
> advantage? -- Andrei

DIP1005 provides concrete advantages. We can productively discuss its 
concrete advantages.

"Encapsulation" is an abstraction over a number of features. Many 
features can increase potential encapsulation, and this is only one of 
them. Encapsulation can provide a wide range of benefits, and this 
feature only provides one or two of them.

It saves us no effort to say "encapsulation". We still have to peel back 
that label to see what's underneath and evaluate that. But 
"encapsulation" is an applause light, so we automatically think better of 
the proposal than if we were merely looking at its concrete benefits.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Adam D. Ruppe via Digitalmars-d]

On Tuesday, 13 December 2016 at 22:33:24 UTC, Andrei Alexandrescu 
wrote:

Destroy.

https://github.com/dlang/DIPs/pull/51/files


FYI, from the DIP, this is false:

"The static import setup does not share the issue above, at the 
cost of being cumbersome to use---all imported symbols must use 
full lookup everywhere. A reasonable engineering approach would 
be to define shorter names:"


It then proceeds to list a bunch of local aliases. That's not how 
I'd do it because you can also simply use a renamed import:


static import r = std.range.primitives;

void foo(R)(R range) if(r.isInputRange!(range)) // works today


If the import were lazy - which there's no reason not to be with 
a static import, renamed or not, since its use is always explicit 
- then this gives most the same advantages of the DIP.


Granted, you'd still have to look up what `r` means, so it isn't 
100% dependency carrying.




Another current language option not discussed in the document is 
an eponymous template.



But there is another language option today: an eponymous template.

Consider the following:

with (import std.stdio) void process(File input) ;

Let's rewrite that to be:

template process() {
   import std.stdio;
   void process(File input) {

   }
}


The whole symbol is now lazy and encapsulated... and thanks to 
the existing eponymous rules, it'd be semi-transparent to the 
user (just error messages would be the main source of leakage, oh 
god the error messages could be so ugly).


But it works in today's D.



BTW it is my opinion that the "one file, one module" rule is a 
mistake. Even with all these things, the declarations are still 
not *guaranteed* to carry their dependencies, since top-level 
imports still leak in. Separate modules don't have that problem, 
and if we could just define several modules in one file, we'd 
basically destroy this DIP in one swift stroke.


I doubt y'all would change that, but still, I'd list that as an 
alternative in the section that lists many small files as being a 
major disadvantage, since while we are discussing changing the 
language, that's a possible change too unless explicitly off the 
table.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timon Gehr via Digitalmars-d]

On 31.12.2016 13:23, Andrei Alexandrescu wrote:

On 12/30/16 11:10 PM, Chris Wright wrote:

On Fri, 30 Dec 2016 21:13:19 -0500, Andrei Alexandrescu wrote:

DIP1005 can't be in
the business of arguing that encapsulation is good by means of examples.


Right. I said we should talk about the concrete benefits of the proposal
instead of talking about encapsulation.


Why would the DIP hamstring itself by not discussing its most important
advantage? -- Andrei


Because it is a slippery slope if you subscribe to the notion that "D's 
unit of encapsulation is the module".


Do we need to rethink encapsulation in D? Should there be an additional 
'internal' visibility modifier that hides members even to other 
declarations in the same module?



On an unrelated note: I'm still not a fan of the with(import) syntax as 
it morally promotes a lack of turtles (even if not technically so).

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/30/16 11:10 PM, Chris Wright wrote:

On Fri, 30 Dec 2016 21:13:19 -0500, Andrei Alexandrescu wrote:

DIP1005 can't be in
the business of arguing that encapsulation is good by means of examples.


Right. I said we should talk about the concrete benefits of the proposal
instead of talking about encapsulation.


Why would the DIP hamstring itself by not discussing its most important 
advantage? -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 30 Dec 2016 21:13:19 -0500, Andrei Alexandrescu wrote:
> DIP1005 can't be in
> the business of arguing that encapsulation is good by means of examples.

Right. I said we should talk about the concrete benefits of the proposal 
instead of talking about encapsulation.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/30/16 7:34 PM, Chris Wright wrote:

On Fri, 30 Dec 2016 23:49:23 +, Andrei Alexandrescu wrote:

The main win, which indeed is not emphasized enough, is better
encapsulation. Walter pointed that out, and I will redo the DIP to put
that front and center.


Encapsulation is an abstraction over several concrete benefits. We're not
horribly constrained in time or cognitive effort, so we can talk about
the concrete benefits instead of the abstraction.


Framing things this way would be really ignorant. DIP1005 can't be in 
the business of arguing that encapsulation is good by means of examples. 
First off, there is no undeniable proof that encapsulation is 
advantageous and not even simple obvious examples; encapsulation has had 
many opponents, most notably Fred Brooks (the author of "The Mythical 
Man-Month"), who used a process specifically antithetic to encapsulation 
in the development of IBM System/360 and OS/360. Far as I recall Fred 
ultimately ceded the point that encapsulation is superior, but IIRC that 
was after 2000. I have no doubt there are competent folks out there who 
think encapsulation is a crock.


So it would be goofy if DIP1005 took the onus to show the advantages of 
improved encapsulation by means of concrete examples (such don't exist 
outside of large-scale projects for any kind of encapsulation). What 
DIP1005 can do is show to someone who already believes that 
encapsulation is good, that the proposed feature improves encapsulation.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/30/16 7:32 PM, Chris Wright wrote:

On Fri, 30 Dec 2016 22:42:39 +, Stefan Koch wrote:


On Friday, 30 December 2016 at 22:29:18 UTC, Chris Wright wrote:

* Performance improvements, primarily when a module imports another,
bulky module for optional functionality.

That is solved by selective imports.


Well, not today, but it could be. Same with static imports. Both
accomplish this goal better than DIP1005.


Is this fact or opinion? If the former, could you please point where 
DIP1005 is getting it wrong. Thanks.



* Making it easier to locate where things are defined when reading
code.


That is solved by selective imports.


Static imports do that better than selective imports, though it's more
typing.


So whether that's overall better is not settled, is it? We can't really 
define "better" as whatever each participant believes.



With selective imports, you can either find the declaration in the
current module, or find the symbol in an import list. If it's imported,
and the module author is using standard code formatting, you will find
the import at the top of the module, which will be fast. This is less
true with arbitrarily scoped imports.

With DIP1005, you rely on there also being selective or static imports.
If the module author didn't use them, then you have to pull out grep.


DIP1005 allows the declaration to encapsulate its own dependencies. Of 
course if top-level imports are also present, the benefit erodes.



* Making it easier to move declarations between files.


That is solved by selective imports.


That helps a bit. You still need to exercise discipline in only using
selective imports for what you need for visible declarations.

Static imports work about as well, except you can probably write a
complicated regex replacement to extract out the imports you need.

DIP1005 is a solid improvement here.

Unfortunately, this is a rather marginal usecase.

-- Another thing that just occurred to me: if you're modifying a function
signature and that brings in another imported symbol, you don't have to
move to the top of the file or struct to add the necessary import. It's a
small thing for vim users, who are used to using marks and fast movement
commands, but if you're using, say, VSCode, that might be painful.


That's one of the many benefits of encapsulation.


Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 30 Dec 2016 23:49:23 +, Andrei Alexandrescu wrote:
> The main win, which indeed is not emphasized enough, is better
> encapsulation. Walter pointed that out, and I will redo the DIP to put
> that front and center.

Encapsulation is an abstraction over several concrete benefits. We're not 
horribly constrained in time or cognitive effort, so we can talk about 
the concrete benefits instead of the abstraction.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 30 Dec 2016 22:42:39 +, Stefan Koch wrote:

> On Friday, 30 December 2016 at 22:29:18 UTC, Chris Wright wrote:
>> * Performance improvements, primarily when a module imports another,
>> bulky module for optional functionality.
> That is solved by selective imports.

Well, not today, but it could be. Same with static imports. Both 
accomplish this goal better than DIP1005.

>> * Making it easier to locate where things are defined when reading
>> code.
> 
> That is solved by selective imports.

Static imports do that better than selective imports, though it's more 
typing.

With selective imports, you can either find the declaration in the 
current module, or find the symbol in an import list. If it's imported, 
and the module author is using standard code formatting, you will find 
the import at the top of the module, which will be fast. This is less 
true with arbitrarily scoped imports.

With DIP1005, you rely on there also being selective or static imports. 
If the module author didn't use them, then you have to pull out grep.

>> * Making it easier to move declarations between files.
> 
> That is solved by selective imports.

That helps a bit. You still need to exercise discipline in only using 
selective imports for what you need for visible declarations.

Static imports work about as well, except you can probably write a 
complicated regex replacement to extract out the imports you need.

DIP1005 is a solid improvement here.

Unfortunately, this is a rather marginal usecase.

-- Another thing that just occurred to me: if you're modifying a function 
signature and that brings in another imported symbol, you don't have to 
move to the top of the file or struct to add the necessary import. It's a 
small thing for vim users, who are used to using marks and fast movement 
commands, but if you're using, say, VSCode, that might be painful.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[ArturG via Digitalmars-d]

On Friday, 30 December 2016 at 23:49:23 UTC, Andrei Alexandrescu 
wrote:


The main win, which indeed is not emphasized enough, is better 
encapsulation. Walter pointed that out, and I will redo the DIP 
to put that front and center.


Maybe i can provide an example where i think DCD's would be 
usefull.
Im experimenting with a more Qt like signals and slots 
implementation,
it introspects some type for function declarations with the 
@("Signal") uda

and mixes them into the current type.
It currently looks like this:

// moda.d
interface TestSignals
{
   import modb: TestType;
   @("Signal"):
  void someTestSig(TestType);
}

// modb.d
struct TestType{}

// test.d
class A : SignalObject
{
   import moda: TestSignals;
   import modb: TestType; // has to be manually imported for the 
generated signal


   mixin signalsOf!(SignalList, TestSignals);

   interface SignalList
   {
  @("Signal"):
 void foo(int);
 void bar(string, int = 100);
 int bar(int a, int b);
   }

   void someFun(string s) { s.writeln; }
}

class B : SignalObject
{
   struct SignalList
   {
  @("Signal"):
 void someSig(string);
   }

   mixin signalsOf!SignalList;

   void fooHandler(int i){ i.writeln; }

   int onBar(int a, int b){ writeln(a + b); return a + b; }
   void onBar(string s, int i){ writeln(s, i); }

   import modb: TestType;
   void someTestSigHandler(TestType t){ t.writeln; }
}

class C : SignalObject
{
   import modb: TestType; // has to be manually imported for the 
generated signal


   mixin signalsOf!(A, B);

   // i wanted to support string based signal declaration
   // but that would require double manual import declarations

   // import modb: TestType;

   // enum signalList =
   // q{
   //import modb: TestType;
   //@("Signal"):
   //void someSig(TestType);
   //int someOtherSig();
   // }

   // mixin signalsOf!signalList;
}

void main(string[] args)
{
   auto a = new A;
   auto b = new B;
   auto c = new C;

   a.connect!"foo"();
   a.connect(b); // connect all bar's with all onBar's

   a.foo(66);
   a.bar("asd");
   a.bar(4, 6);

   c.connect!"someSig"();
   b.connect!"someSig"();

   c.someSig("emitted by c forwarded by b handled by a");

   a.connect!"someTestSig"();
   c.connect!"someTestSig"();

   import modb: TestType;
   TestType t;
   a.someTestSig(t);
   c.someTestSig(t);
}

i assume with DCD's i could remove those manual imports if they 
get some traits.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

Stefan Koch  wrote:
> On Friday, 30 December 2016 at 22:29:18 UTC, Chris Wright wrote:
>> * Performance improvements, primarily when a module imports 
>> another,
>> bulky module for optional functionality.
> That is solved by selective imports.
> 
>> * Making it easier to locate where things are defined when 
>> reading code.
> 
> That is solved by selective imports.
> 
>> * Making it easier to move declarations between files.
> 
> That is solved by selective imports.
> 
> Inline imports are really just a addition retrive the the 
> deprecated behavior of fullyQualifedNames which would implicitly 
> import.
> 

The main win, which indeed is not emphasized enough, is better
encapsulation. Walter pointed that out, and I will redo the DIP to put that
front and center.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Friday, 30 December 2016 at 22:29:18 UTC, Chris Wright wrote:
* Performance improvements, primarily when a module imports 
another,

bulky module for optional functionality.

That is solved by selective imports.

* Making it easier to locate where things are defined when 
reading code.


That is solved by selective imports.


* Making it easier to move declarations between files.


That is solved by selective imports.

Inline imports are really just a addition retrive the the 
deprecated behavior of fullyQualifedNames which would implicitly 
import.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 30 Dec 2016 18:18:12 +, Stefan Koch wrote:
> I am still not sure what problem it is trying to solve :)

* Performance improvements, primarily when a module imports another, 
bulky module for optional functionality.
* Making it easier to locate where things are defined when reading code.
* Making it easier to move declarations between files.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Friday, 30 December 2016 at 18:11:54 UTC, deadalnix wrote:

I think the performance gain we are looking at here is 
marginal, and I don't expect people to change their code to get 
a marginal benefit, so I suggest the performance aspect of the 
change to be simply left aside.


Yes, imports are (in most cases) not performance relevant!

So the question now is would the added expressivity of per 
declaration import be worth the language change. I have to 
admit I'm not convinced either way.


Neither am I.

Finally, while I proposed a variation of the "with import" 
combo in the past, I'm now much more convinced that using the 
plain import syntax, without the ';' is better. I was afraid 
there was a syntax conflict, but it doesn't looks like there is 
one. This will not introduce a new syntax, but allow an 
existing one to be used in a new location. This is, IMO, much 
more valuable.


I am still not sure what problem it is trying to solve :)

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[deadalnix via Digitalmars-d]

On Wednesday, 28 December 2016 at 23:14:48 UTC, Andrei 
Alexandrescu wrote:

On 12/28/16 10:48 AM, deadalnix wrote:
On Saturday, 24 December 2016 at 15:44:18 UTC, Andrei 
Alexandrescu wrote:
A compiler enhancement can do this _without_ a language 
change.


The language addition has additional benefits as described by 
DIP1005.

-- Andrei


Yes, question is, are these specific benefits worth adding a 
language

change ?

Right now you got :

A/ No language change, get X.
B/ Language change, get X and Y.

It stand to reason that B should be evaluated on the benefit 
provided by
Y, and Y only, rather than X and Y, as X can be provided 
without the

language change.


This is exactly what the DIP describes in detail. It 
consecrates sections to alternatives. Is anything missing? -- 
Andrei


This was more a comment about the current discussion than the 
DIP. A lot of discussion effort was focused on performance, but 
most of it can be achieved without language change - as the DIP 
states.


I think the performance gain we are looking at here is marginal, 
and I don't expect people to change their code to get a marginal 
benefit, so I suggest the performance aspect of the change to be 
simply left aside.


So the question now is would the added expressivity of per 
declaration import be worth the language change. I have to admit 
I'm not convinced either way.


Finally, while I proposed a variation of the "with import" combo 
in the past, I'm now much more convinced that using the plain 
import syntax, without the ';' is better. I was afraid there was 
a syntax conflict, but it doesn't looks like there is one. This 
will not introduce a new syntax, but allow an existing one to be 
used in a new location. This is, IMO, much more valuable.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/28/16 9:47 PM, Chris Wright wrote:

On Wed, 28 Dec 2016 15:48:46 +, deadalnix wrote:


On Saturday, 24 December 2016 at 15:44:18 UTC, Andrei Alexandrescu
wrote:

A compiler enhancement can do this _without_ a language change.


The language addition has additional benefits as described by DIP1005.
-- Andrei


Yes, question is, are these specific benefits worth adding a language
change ?


And the associated detriments. Just imagine opening up phobos and seeing:

  // 1200 lines of code
  with (import foo)
  {
// 300 lines of declarations...
with (import bar)
{
  // 250 lines of additional declarations...
}
  }

All mixed in with structs and functions and conditional compilation, so
you have a sea of curly braces to wade through and indentation is only a
mild help.


The D Standard Library will generally specify dependencies with 
declarations. If the point was that the feature can be misused or abused 
if one really tries, I agree. -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Wed, 28 Dec 2016 15:48:46 +, deadalnix wrote:

> On Saturday, 24 December 2016 at 15:44:18 UTC, Andrei Alexandrescu
> wrote:
>>> A compiler enhancement can do this _without_ a language change.
>>
>> The language addition has additional benefits as described by DIP1005.
>> -- Andrei
> 
> Yes, question is, are these specific benefits worth adding a language
> change ?

And the associated detriments. Just imagine opening up phobos and seeing:

  // 1200 lines of code
  with (import foo)
  {
// 300 lines of declarations...
with (import bar)
{
  // 250 lines of additional declarations...
}
  }

All mixed in with structs and functions and conditional compilation, so 
you have a sea of curly braces to wade through and indentation is only a 
mild help.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/28/16 10:48 AM, deadalnix wrote:

On Saturday, 24 December 2016 at 15:44:18 UTC, Andrei Alexandrescu wrote:

A compiler enhancement can do this _without_ a language change.


The language addition has additional benefits as described by DIP1005.
-- Andrei


Yes, question is, are these specific benefits worth adding a language
change ?

Right now you got :

A/ No language change, get X.
B/ Language change, get X and Y.

It stand to reason that B should be evaluated on the benefit provided by
Y, and Y only, rather than X and Y, as X can be provided without the
language change.


This is exactly what the DIP describes in detail. It consecrates 
sections to alternatives. Is anything missing? -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[deadalnix via Digitalmars-d]

On Saturday, 24 December 2016 at 19:16:40 UTC, Joakim wrote:
Can we hold off on the performance discussion?  Walter says 
this DIP isn't hard to implement 
(https://github.com/dlang/DIPs/pull/51#issuecomment-269077790), 
so we will run some numbers and see what we get.  As you know, 
I too am skeptical of the benefit but Andrei has shown that 
import fanout has a non-negligible cost with his latest 
benchmark, and actual measurement will be the best way to 
decide.


The "problem" is that you'd be essentially measuring how 
inefficient the current DMD import management is rather than 
anything else.


When you push something on the user, you want to make sure it 
bring an actual benefit, not just allow to work around lousy 
existing implementation.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[deadalnix via Digitalmars-d]

On Saturday, 24 December 2016 at 15:44:18 UTC, Andrei 
Alexandrescu wrote:

A compiler enhancement can do this _without_ a language change.


The language addition has additional benefits as described by 
DIP1005. -- Andrei


Yes, question is, are these specific benefits worth adding a 
language change ?


Right now you got :

A/ No language change, get X.
B/ Language change, get X and Y.

It stand to reason that B should be evaluated on the benefit 
provided by Y, and Y only, rather than X and Y, as X can be 
provided without the language change.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 24 Dec 2016 17:45:35 +, Stefan Koch wrote:
> I just read over the dip, and it is a giant wall of text.
> I cannot really make heads or tails of it.
> Maybe you could write the advantages you hit at in short bullet-point
> form ?

We add the syntax:

with (import foo, bar, baz)
SingleDeclaration

* If you're reading it as source code, you have, usually, 1-3 modules to 
look in for related definitions, instead of lots (plain imports) or one 
(static / selective imports).

* These modules are listed one line up from the declaration (as with 
static imports, unlike selective imports).

* It doesn't look quite as cluttered as static imports.

* It saves a little typing compared to static imports when you use the 
same import twice in one declaration.

* If you need to move this declaration to another module, you don't have 
to worry about imports (provided you assiduously avoid global imports). 
You still have to worry about everything the declaration references from 
the module it was previously defined in.

* The compiler can delay importing the module until you use the 
declaration it's attached to.

* It might be slightly easier to implement in DMD than lazy static / 
selective imports.



We extend that syntax:

with (import foo)
with (import bar, baz)
{
  // multiple declarations
  // possibly other with-import blocks
}

* You have even less typing if you reuse the same import in a lot of 
declarations.

* The compiler can still delay importing the module until you use the 
declaration it's attached to.

* You get to have a fun search through the entire module to try to find 
which modules are imported when reading a declaration, instead of looking 
up one line (first pass on the syntax), looking at the symbol being used 
(static imports), or looking at the top of the file (status quo or 
selective imports).

* That also applies when you try to move a declaration to another module.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 24 Dec 2016 14:10:30 -0500, Andrei Alexandrescu wrote:

> On 12/24/2016 12:45 PM, Stefan Koch wrote:
>> I just read over the dip, and it is a giant wall of text.
>> I cannot really make heads or tails of it.
> 
> When it was shorter, people complained it had too little information.

It's not the length that's giving trouble; it's the organization.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 24 Dec 2016 14:03:26 -0500, Andrei Alexandrescu wrote:
> * Your post provided no description of the methodology used beyond
> "relatively simple regex" to come up with the magic number 26, so I
> tried to use interpretation.
> 
> * There is no answer to "where is cmp on that list for example?" to shed
> light on what 26 is about.

You mentioned that in another branch of the conversation. It took some 
time for me to get to it. I'm a human.

>> In part, you are assuming that imports on non-template declarations
>> will be handled lazily, even though that is not part of this DIP, even
>> though that is likewise possible with static and selective imports.
> 
> Imports in non-template declarations ARE handled lazily when those
> non-templates are imported. Consider:

...

Y'know, I make assumptions, but I tend to test them. But that kind of 
depends on me writing valid tests, and in this case, I didn't.

My face is egged, and I concede.

Thank you for providing example code. That cleared everything up very 
fast.

With this new understanding, most modules become nearly free to import 
under your proposal. It does require a lot of work to get there, but on 
the plus side, it should be possible to automate that with a tool. I'd be 
willing to add that to my build process for anything I publish if it'll 
help others.

>> In part, you are using lines of code as a proxy for compile time.
> 
> What do you suggest to use?

You did write a script to check compile times for importing individual 
phobos modules.

>>> The findings of DIP1005 are the following:
>>>
>>> * Importing a single std module also imports on average 10.5 other
>>> modules.
>>
>> Seems reasonable. Between 2 and 3.5 direct dependencies, by my count,
>> and you're counting transitive dependencies.
> 
> Shouldn't I? Shouldn't you?

I was agreeing with you. At the time, I didn't have a convenient way to 
check the number of transitive dependencies of a module.

>> We're concerned with the effects of DIP1005, though, which only affects
>> template constraints.
>>
>>> * Importing a single std module costs on average 64.6 ms.
>>
>> 55-ish for your hardware, you reported elsewhere. 47-ish for mine.
> 
> Let's stick with the numbers published in the DIP.

Sure. You insinuated that I should run your script to verify your 
numbers, so I did.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 23 Dec 2016 23:41:46 -0500, Andrei Alexandrescu wrote:

> On 12/23/16 9:23 PM, Chris Wright wrote:
>> The comparison to mach.d is a strawman.
> 
> The mach.d is given as an example of the approach of breaking code into
> fine-grained module. No comparison is made or implied.

"Assuming module size is a project invariant, the number of files scales 
roughly with project size. This means mach.d would need 2000 files to 
scale up to the same size as the D standard library"

That's a direct comparison that you made. You might want to remove that 
from the proposal if you don't want to compare them.

> Could you please give more detail about the method you used?

An insufficient one.

(I would think this sort of analysis would generally be the burden of the 
proposer, generally.)

https://gist.github.com/dhasenan/681b5178672556aa0e5ec8fb4c9eae7e uses 
dmd's json output instead of regular expressions and should be rather 
better.

This shows a total of 35 templates used in template constraints, 124 
usages in total.

The quick usage breakdown:

 1 std.experimental.ndslice.internal
 1 std.experimental.ndslice.slice
24 std.meta
31 std.range.primitives
66 std.traits
 1 std.typecons

Notable mentions: the ndslice templates are used only in other ndslice 
modules, and the std.typecons reference is from 
std.experimental.typecons. So it's still ~10ms of overhead, by both our 
measurements.

> And again the
> feature's benefits go well beyond making general projects faster to
> build.

You may as well remove the parts about compilation efficiency from the 
proposal entirely, then.

The DIP's points here aren't invalid. However, static and selective 
imports work nearly as well. A doc generator that turns referenced 
identifiers into links (like dpldocs.info) works nearly as well for 
reading. For moving files between modules, that's relatively rare, and 
fixing imports is a small part of it. (In the worst case, I can copy all 
imports over and use dustmite to reduce them. Or just leave them.)

It provides small occasional benefits for a nontrivial maintenance cost. 
At least it doesn't mandate that I use it.

> A simple way to look at things is - local imports are The Right Thing,
> for many reasons beyond compilation speed. There needs to be a way to do
> the right thing for the declaration part as well.

Function-scoped imports are a good way to hide implementation details 
from people who don't need them.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/24/2016 02:16 PM, Joakim wrote:

On Saturday, 24 December 2016 at 17:53:04 UTC, Chris Wright wrote:

On Sat, 24 Dec 2016 09:20:19 +, John Colvin wrote:

There are a lot of templates in Phobos that never use the template
keyword. The proposal doesn't only apply to constraints, it applies
to the whole declaration.


If you have a better way of estimating the impact, I'd love to see it.


Can we hold off on the performance discussion?  Walter says this DIP
isn't hard to implement
(https://github.com/dlang/DIPs/pull/51#issuecomment-269077790), so we
will run some numbers and see what we get.  As you know, I too am
skeptical of the benefit but Andrei has shown that import fanout has a
non-negligible cost with his latest benchmark, and actual measurement
will be the best way to decide.


Also (from the cycle "sounding like a broken record") the impact is in 
other dimension than import speed. From DIP1005:


==
[...] Dependency-Carrying Declarations have multiple benefits:

* Specifies dependencies at declaration level, not at module level. This 
allows reasoning about the dependency cost of declarations in separation 
instead of aggregated at module level.


* If all declarations use Dependency-Carrying style and there is no 
top-level import, human reviewers and maintainers can immediately tell 
where each symbol in a given declaration comes from. This is a highly 
nontrivial exercise without specialized editor support in projects that 
pull several other modules and packages wholesale. Even a project 
newcomer could gather an understanding of a declaration without needing 
to absorb an arbitrary amount of implied context from the declaration at 
the top of the module.


* Dependency-Carrying Declarations are easier to move around, making for 
simpler and faster refactorings.


* Dependency-Carrying Declarations allow scalable template libraries. [...]

Dependency-Carrying Declarations also have drawbacks:

* If most declarations in a module need the same imports, then factoring 
them outside the declarations at top level is simpler and better than 
repeating them.


* Related, renaming one module is likely to require more edits in a 
Dependency-Carrying Declarations setup.


* Traditional dependency-tracking tools such as make and other build 
systems assume file-level dependencies and need special tooling (such as 
rdmd) in order to work efficiently.


* Dependencies at the top of a module are easier to inspect quickly than 
dependencies spread through the module.

==


Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/24/2016 02:03 PM, Andrei Alexandrescu wrote:

With no top-level imports and all imports hoisted into inline imports,
the fixed cost of importing one module will be that module alone.


I'd really like to clarify this because that's where scalability comes 
from. The moment you need to open an unbounded amount of files for one 
import, it's gone. If I missed something and more than one module needs 
to be looked at I need to rethink the whole DIP. -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Saturday, 24 December 2016 at 17:53:04 UTC, Chris Wright wrote:

On Sat, 24 Dec 2016 09:20:19 +, John Colvin wrote:
There are a lot of templates in Phobos that never use the 
template keyword. The proposal doesn't only apply to 
constraints, it applies to the whole declaration.


If you have a better way of estimating the impact, I'd love to 
see it.


Can we hold off on the performance discussion?  Walter says this 
DIP isn't hard to implement 
(https://github.com/dlang/DIPs/pull/51#issuecomment-269077790), 
so we will run some numbers and see what we get.  As you know, I 
too am skeptical of the benefit but Andrei has shown that import 
fanout has a non-negligible cost with his latest benchmark, and 
actual measurement will be the best way to decide.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/24/2016 12:45 PM, Stefan Koch wrote:

I just read over the dip, and it is a giant wall of text.
I cannot really make heads or tails of it.


When it was shorter, people complained it had too little information.

If you're using the text form, the nicey formatted version may be 
helpful: 
https://github.com/dlang/DIPs/blob/0e15d550d6774dfd40ec78dc201dfad33f47a740/DIPs/DIP1005.md



Maybe you could write the advantages you hit at in short bullet-point
form ?


There is. Search for "The analysis above reveals that 
Dependency-Carrying Declarations have multiple benefits:".



Thanks,

Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/24/2016 12:52 PM, Chris Wright wrote:

On Sat, 24 Dec 2016 04:34:03 -0500, Andrei Alexandrescu wrote:

Upon more investigation, I see a large discrepancy between the findings
of DIP1005 and yours.


There's no discrepancy.

In part, you are misinterpreting most of what I said.


Yah, I suspected so.

* Your post provided no description of the methodology used beyond 
"relatively simple regex" to come up with the magic number 26, so I 
tried to use interpretation.


* There is no answer to "where is cmp on that list for example?" to shed 
light on what 26 is about.


* I have no access to your method of measurement, the scripts you used, 
or really what the claims are.


So I really don't have a good understanding of what hypothesis you have 
and how you arrived to it. DIP1005 carefully describes methodology and 
publishes measurements and results (which I'd be in your debt if you 
used - not those "published elsewhere"). Then it interprets the results.



In part, you are assuming that imports on non-template declarations will
be handled lazily, even though that is not part of this DIP, even though
that is likewise possible with static and selective imports.


Imports in non-template declarations ARE handled lazily when those 
non-templates are imported. Consider:


module test;
void fun()
{
import std.stdio;
writeln("hello");
}

module test2;
import test;
void main() {} // let's not call fun

dmd -v test2.d


binarydmd
version   v2.073.0-devel-bd0dec2
config/home/andrei/bin/dmd.conf
parse test2
importall test2
importobject(/home/andrei/d/druntime/import/object.d)
importtest  (test.d)
semantic  test2
entry main  test2.d
semantic2 test2
semantic3 test2
code  test2
function  D main
cc test2.o -o test2 -m64 
-L/home/andrei/d/phobos/generated/linux/release/64 -Xlinker -Bstatic 
-lphobos2 -Xlinker -Bdynamic -lpthread -lm -lrt -ldl



Destroyed? This might be another misinterpretation of what you said though.


In part, you are using lines of code as a proxy for compile time.


What do you suggest to use?


In part, you dispute that this only affects template constraints, but:
* An import used only in the body of a template can be made a local
import today.


Correct. To the best of my knowledge, that fruit has been picked in Phobos.


* An import used in the declaration of a templated type or function can
be addressed by using explicit template syntax, offering a place to
insert your imports.


Good thought. Would be a bit overkill though.


* An import used anywhere else must still be processed, even assuming
this DIP is implemented.


How do you mean that? On the face of it the sentence is true. Are you 
saying that you need to remove top-level imports to benefit from the 
DIP? That would indeed be the case.



* If, in a future DIP, we make it so that `with(import)` is handled
lazily, we can also make it so that static and selective imports are
handled lazily.


That would be nice, but DIP1005 provides advantages beyond latency.

BTW it would be great if the discussion focused on (a) things that are 
true but the DIP doesn't mention (or misrepresents), or (b) things that 
are false that the DIP claims. I clearly concede that the DIP cannot 
convince someone with enough cognitive bias without an implementation 
and a few years of experience (and even then - aren't there folks out 
there who believe things like constraints, static if, or ranges are 
failures?).



The findings of DIP1005 are the following:

* Importing a single std module also imports on average 10.5 other
modules.


Seems reasonable. Between 2 and 3.5 direct dependencies, by my count, and
you're counting transitive dependencies.


Shouldn't I? Shouldn't you?


We're concerned with the effects of DIP1005, though, which only affects
template constraints.


* Importing a single std module costs on average 64.6 ms.


55-ish for your hardware, you reported elsewhere. 47-ish for mine.


Let's stick with the numbers published in the DIP.


* (Not stated in the DIP) A majority of std templates would acquire
inline imports.


Again, that wouldn't impact compile times because these aren't template
constraints.


I don't understand this. YES it would impact compile times!


You can make a separate DIP to make imports lazy. That can impact static,
selective, and `with` imports equally well. But it's not part of what
we're discussing today.


According to the DIP, one may estimate that the proposed feature would
reduce additional imports to 0 and the average time to import a single
module by a factor of 10 to under 10 ms.


"The proposed feature" must be lazy semantic analysis, especially of
imports. That isn't part of DIP1005.


The proposed feature is DIP1005. I really don't understand how your 
discourse goes.



You won't get to zero additional imports. You might get to zero
*extraneous* imports -- that is, only the set of imports required to
create a custom *.di file containing only 

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Saturday, 24 December 2016 at 17:52:04 UTC, Chris Wright wrote:


The minimum isn't terribly useful because it gets to the point 
of testing the process scheduler and IO more than the compiler. 
If we want numbers that we can trust on the low end, we'll need 
to put timing information into the compiler, maybe control for 
IO by using a ramfs, that sort of thing.


As a good approximation you can use a profile build of dmd.
It will give you a breakdown of how long the individual functions 
took.
Due to the way the profiling works, it does homogenize the values 
a bit.

Files are not streamed into dmd, it reads them as one Block.
Therefore there is no need for a ramfs.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 24 Dec 2016 04:34:03 -0500, Andrei Alexandrescu wrote:
> Upon more investigation, I see a large discrepancy between the findings
> of DIP1005 and yours.

There's no discrepancy.

In part, you are misinterpreting most of what I said.

In part, you are assuming that imports on non-template declarations will 
be handled lazily, even though that is not part of this DIP, even though 
that is likewise possible with static and selective imports.

In part, you are using lines of code as a proxy for compile time.

In part, you dispute that this only affects template constraints, but:
* An import used only in the body of a template can be made a local 
import today.
* An import used in the declaration of a templated type or function can 
be addressed by using explicit template syntax, offering a place to 
insert your imports.
* An import used anywhere else must still be processed, even assuming 
this DIP is implemented.
* If, in a future DIP, we make it so that `with(import)` is handled 
lazily, we can also make it so that static and selective imports are 
handled lazily.

> The findings of DIP1005 are the following:
> 
> * Importing a single std module also imports on average 10.5 other
> modules.

Seems reasonable. Between 2 and 3.5 direct dependencies, by my count, and 
you're counting transitive dependencies.

We're concerned with the effects of DIP1005, though, which only affects 
template constraints.

> * Importing a single std module costs on average 64.6 ms.

55-ish for your hardware, you reported elsewhere. 47-ish for mine.

> * (Not stated in the DIP) A majority of std templates would acquire
> inline imports.

Again, that wouldn't impact compile times because these aren't template 
constraints.

You can make a separate DIP to make imports lazy. That can impact static, 
selective, and `with` imports equally well. But it's not part of what 
we're discussing today.

> According to the DIP, one may estimate that the proposed feature would
> reduce additional imports to 0 and the average time to import a single
> module by a factor of 10 to under 10 ms.

"The proposed feature" must be lazy semantic analysis, especially of 
imports. That isn't part of DIP1005.

You won't get to zero additional imports. You might get to zero 
*extraneous* imports -- that is, only the set of imports required to 
create a custom *.di file containing only the parts of the module that 
your application uses.

> By your estimates:
> 
> * 26 templates in std need inline imports.

I said that 26 templates *could possibly benefit from* your new style of 
imports. There's a difference between possibly benefitting from a change 
and needing that change.

> * Importing a single std module today would only imports 1-3 other
> modules most of the time (one or more of std.traits, std.meta, and
> std.range.primitives).

No, that's not what I said at all. I said that the only modules you would 
sometimes *stop* processing because of DIP1005 are std.traits, std.meta, 
and std.range.primitives. That's because those modules contain templates 
used in other modules as template constraints.

In order to get any additional improvements, you need lazy imports, which 
can also apply to static or selective imports without any syntax changes.

> * These additional imports cost in aggregate under 10ms, bringing the
> average cost of importing a module itself to 54.6 ms.

~10ms is the upper bound of the added cost if you import just one module 
in std that has a template constraint you don't use.

The way you state it implies that every module brings in std.traits, 
std.meta, and std.range.primitives unnecessarily, instead of 26 templates 
across at most 26 modules importing them for a reason.

> * It follows that the average module takes 5.46 more times to import
> alone than the sum of std.traits, std.meta, and std.range.primitives
> (which have a total of 11263 lines, 5x more than the average Phobos
> module).

More like 4.7 on my hardware, but yeah. 11k lines that have to be parsed 
and 0 lines that require semantic analysis. Not terribly surprising.

> I don't see how your claims can be simultaneously true with the findings
> of DIP1005.

You found that the average cost of importing a std module is 54ms or 
thereabouts. std.traits, std.meta, and std.range.primitives are well 
below average. No conflict there. They aren't even the cheapest modules 
in the standard library.

The modules in question are mostly unittests. The compiler doesn't run 
semantic on unittests in a module that wasn't included in the command 
line. (Even if you pass -unittest. Try it out -- you can even have a 
unittest that says `static assert(false);` and it does nothing.)

The parts of the modules that are not unittests are templates. The 
compiler doesn't run semantic analysis on templates until you use them.

So it should be pretty obvious why these modules are so cheap to import 
and not use.

> The scripts that compute those numbers are available with
> the DIP. 

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Sat, 24 Dec 2016 09:20:19 +, John Colvin wrote:
> There are a lot of templates in Phobos that never use the template
> keyword. The proposal doesn't only apply to constraints, it applies to
> the whole declaration.

If you have a better way of estimating the impact, I'd love to see it.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Saturday, 24 December 2016 at 15:44:18 UTC, Andrei 
Alexandrescu wrote:

On 12/24/16 9:20 AM, Stefan Koch wrote:
On Saturday, 24 December 2016 at 14:08:48 UTC, Andrei 
Alexandrescu wrote:

On 12/24/2016 05:54 AM, Stefan Koch wrote:
If that were made more lazy, we could import half of the 
world with

noticing impact.


That is what 1005 will bring. -- Andrei


A compiler enhancement can do this _without_ a language change.


The language addition has additional benefits as described by 
DIP1005. -- Andrei


I just read over the dip, and it is a giant wall of text.
I cannot really make heads or tails of it.
Maybe you could write the advantages you hit at in short 
bullet-point form ?

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/24/16 9:20 AM, Stefan Koch wrote:

On Saturday, 24 December 2016 at 14:08:48 UTC, Andrei Alexandrescu wrote:

On 12/24/2016 05:54 AM, Stefan Koch wrote:

If that were made more lazy, we could import half of the world with
noticing impact.


That is what 1005 will bring. -- Andrei


A compiler enhancement can do this _without_ a language change.


The language addition has additional benefits as described by DIP1005. 
-- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Saturday, 24 December 2016 at 14:08:48 UTC, Andrei 
Alexandrescu wrote:

On 12/24/2016 05:54 AM, Stefan Koch wrote:
If that were made more lazy, we could import half of the world 
with

noticing impact.


That is what 1005 will bring. -- Andrei


A compiler enhancement can do this _without_ a language change.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/24/2016 05:54 AM, Stefan Koch wrote:

If that were made more lazy, we could import half of the world with
noticing impact.


That is what 1005 will bring. -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Saturday, 24 December 2016 at 10:54:08 UTC, Stefan Koch wrote:

300 microseconds, which is not even 0.3% of the time spent.
Lexing them requires additionally also about 300 microseconds.
So Together that makes up 0.6% of the time spent.


Of course in the above 3% and 6% are the right numbers.
(And still conservative.)
Since the are obtained by using a -profile build of dmd.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Stefan Koch via Digitalmars-d]

On Saturday, 24 December 2016 at 09:34:03 UTC, Andrei 
Alexandrescu wrote:
(I confirm that importing std.traits, std.meta, and 
std.range.primitives together takes 10ms.)


while compiling std.traits 6 files are opened and read into 
memory,
taking roughly 300 microseconds, which is not even 0.3% of the 
time spent.

Lexing them requires additionally also about 300 microseconds.
So Together that makes up 0.6% of the time spent.

The real problem here is _NOT_ opening and lexing files.

It is rather eager parsing and sema.

If that were made more lazy, we could import half of the world 
with noticing impact.


(Which espcially in std.traits, would not make that much of a 
difference since every template in there depends on nearly every 
other template in there)

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/23/2016 09:23 PM, Chris Wright wrote:
[abbreviated below]

Upon more investigation, I see a large discrepancy between the findings 
of DIP1005 and yours. So there are a number of claims here, which I'll 
summarize:



tldr: the performance impact of this DIP would be too small to easily
measure and only impacts 26 declarations referencing 14 templates in the
standard library anyway.



Phobos has 26 templates that would use this special syntax, referencing
14 distinct templates and CTFE functions in three modules.



The templates that have to be parsed are std.traits, std.meta, and
std.range.primitives. I wrote a test program that imported them without
using them. It compiled in too little time to accurately measure --
`time` reports 0.01s wall time.

You want to add new language syntax to save me ten milliseconds during
compilation.


The findings of DIP1005 are the following:

* Importing a single std module also imports on average 10.5 other modules.

* Importing a single std module costs on average 64.6 ms.

* (Not stated in the DIP) A majority of std templates would acquire 
inline imports.


According to the DIP, one may estimate that the proposed feature would 
reduce additional imports to 0 and the average time to import a single 
module by a factor of 10 to under 10 ms.


By your estimates:

* 26 templates in std need inline imports.

* Importing a single std module today would only imports 1-3 other 
modules most of the time (one or more of std.traits, std.meta, and 
std.range.primitives).


* These additional imports cost in aggregate under 10ms, bringing the 
average cost of importing a module itself to 54.6 ms.


* It follows that the average module takes 5.46 more times to import 
alone than the sum of std.traits, std.meta, and std.range.primitives 
(which have a total of 11263 lines, 5x more than the average Phobos module).


I don't see how your claims can be simultaneously true with the findings 
of DIP1005. The scripts that compute those numbers are available with 
the DIP. Were you able to reproduce them?


(I confirm that importing std.traits, std.meta, and std.range.primitives 
together takes 10ms.)



Thanks,

Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[John Colvin via Digitalmars-d]

On Saturday, 24 December 2016 at 02:23:53 UTC, Chris Wright wrote:
Appendix A: templates that would receive 'with(import foo)' in 
Phobos.


I used a relatively simple regex to look for this. If someone 
put more than one line between a template and its constraints, 
or comments with certain formatting, I may have missed it. 
However, that would violate the phobos style guide.


This doesn't include templates that use constraints defined in 
their own modules, because that module must already be parsed 
and no efficiency gains could be realized.


It also omits a few cases where the module has a strong reason 
to import the dependency aside from template constraints. I 
believe this was only two constraints defined in std.digest, 
used in one or two other modules.


algorithm/comparison.d:98:template among(values...)
experimental/typecons.d:82:private template 
implementsInterface(Source,

Targets...)
experimental/typecons.d:94:template implementsInterface()
experimental/typecons.d:126:private template 
implementsInterface(Source,

Targets...)
experimental/typecons.d:184:template wrap(Targets...)
experimental/typecons.d:237:template wrap(Source)
range/package.d:2069:template Take(R)
range/package.d:3501:template Cycle(R)
range/interfaces.d:277:template MostDerivedInputRange(R)
range/interfaces.d:336:template InputRangeObject(R)
numeric.d:678:template FPTemporary(F)
conv.d:3894:template octal(alias decimalInteger)
utf.d:1136:package template codeUnitLimit(S)
typecons.d:1779:private mixin template RebindableCommon(T, U, 
alias This)

typecons.d:1838:template Rebindable(T)
typecons.d:4239:template wrap(Targets...)
typecons.d:4252:template wrap(Source)
typecons.d:4412:template wrap(Targets...)
typecons.d:4429:template unwrap(Target)
typecons.d:4461:template unwrap(Target)
format.d:657:template FormatSpec(Char)
algorithm/iteration.d:1055:template filter(alias predicate) if 
(is(typeof

(unaryFun!predicate)))
internal/math/biguintcore.d:81:template maxBigDigits(T) if 
(isIntegral!T)
meta.d:248:package template OldAlias(T) if (!isAggregateType!T 
|| is

(Unqual!T == T))
meta.d:254:package template OldAlias(T) if (isAggregateType!T 
&& !is

(Unqual!T == T))
utf.d:3542:template byUTF(C) if (isSomeChar!C)


Appendix B: templates that would need to be extracted out in 
phobos, if parsing their modules cost a non-negligible amount 
of time.


std.meta:
allSatisfy
anySatisfy
ApplyLeft

std.range.primitives:
hasSlicing
isInputRange
isInfinite

std.traits:
isAggregateType
isAssociativeArray
isDynamicArray
isFloatingPoint
isImplicitlyConvertible
isIntegral
isMutable
isSomeChar


There are a lot of templates in Phobos that never use the 
template keyword. The proposal doesn't only apply to constraints, 
it applies to the whole declaration.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/23/16 9:23 PM, Chris Wright wrote:

The comparison to mach.d is a strawman.


The mach.d is given as an example of the approach of breaking code into 
fine-grained module. No comparison is made or implied.



Then I looked at the code.

Phobos has 26 templates that would use this special syntax, referencing
14 distinct templates and CTFE functions in three modules.


Could you please give more detail about the method you used? What 
special syntax are you referring to - would that be the "with (import 
...)" syntax? If that's the case there's a bunch of stuff missing. 
Consider e.g.:


int cmp(alias pred = "a < b", R1, R2)(R1 r1, R2 r2)
if (isInputRange!R1 && isInputRange!R2 && !(isSomeString!R1 && 
isSomeString!R2));


int cmp(alias pred = "a < b", R1, R2)(R1 r1, R2 r2)
if (isSomeString!R1 && isSomeString!R2);

These and many like them would use the "with import" construct, wouldn't 
they? They are missing from your list.



The templates that have to be parsed are std.traits, std.meta, and
std.range.primitives.


That's where most introspection primitives for the standard library are 
situated indeed.



I wrote a test program that imported them without
using them. It compiled in too little time to accurately measure --
`time` reports 0.01s wall time.

You want to add new language syntax to save me ten milliseconds during
compilation.


DIP1005 enumerates several benefits of the proposed feature. Speed is 
not the most important and is not presented as such, but it takes most 
space because the others are more difficult to experiment with without 
an experimental implementation.



You hope that template constraints become more common.


Well, to the extent that is the way correct code is written, that's more 
than just hope.



If they do, this
change relies on:

* projects defining their own constraints (not just using std.traits)
* those constraints being defined in modules other than where they're used
* those constraints being defined in very large modules
* my code not depending on anything in those constraint-defining modules
* my code depending on something in a file with a template with one of
those constraints
* this whole situation being common enough to give me a death by a
thousand papercuts


It's not only constraints, it's everything in declarations that relies 
on entities (e.g. types) defined in other modules. And again the 
feature's benefits go well beyond making general projects faster to build.


A simple way to look at things is - local imports are The Right Thing, 
for many reasons beyond compilation speed. There needs to be a way to do 
the right thing for the declaration part as well.


(I'm glad we're getting to the point of diminished returns - I recall an 
argument made a while ago was that, on the contrary, our compilation 
model cannot possibly scale for large projects.)



Appendix A: templates that would receive 'with(import foo)' in Phobos.


Could you please provide more detail on how you measured this and what 
assumptions you made? Most templates in Phobos would receive a with 
clause, so there's a disconnect somewhere.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

tldr: the performance impact of this DIP would be too small to easily 
measure and only impacts 26 declarations referencing 14 templates in the 
standard library anyway.

On Fri, 23 Dec 2016 18:55:25 -0500, Andrei Alexandrescu wrote:
>> I grant that everyone uses phobos, and phobos uses template constraints
>> a lot. If it's *just* a problem inside phobos, though, there's another
>> obvious solution: split up modules that tend to house a lot of template
>> constraints. Split up the modules that use a wide variety of template
>> constraints.
> 
> This point is discussed carefully in DIP1005. Please let me know if
> anything needs to be added.

An estimate for the actual impact on phobos, since that's your primary 
driver for the change -- both under the status quo and if we try to split 
modules.

The comparison to mach.d is a strawman. When I thought this might be a 
problem within phobos, I thought we'd probably split std.traits and maybe 
std.meta up, probably into 2-5 modules each. Not 150 lines per module; 
more like 1500 to 4000 lines per module.

Then I looked at the code.

Phobos has 26 templates that would use this special syntax, referencing 
14 distinct templates and CTFE functions in three modules.

If you kept the same ratios as are found in mach.d, you'd have one file 
for every template used as a constraint outside its own module, one for 
everything else, and as many files again with nothing in them.

The templates that have to be parsed are std.traits, std.meta, and 
std.range.primitives. I wrote a test program that imported them without 
using them. It compiled in too little time to accurately measure -- 
`time` reports 0.01s wall time.

You want to add new language syntax to save me ten milliseconds during 
compilation.

You hope that template constraints become more common. If they do, this 
change relies on:

* projects defining their own constraints (not just using std.traits)
* those constraints being defined in modules other than where they're used
* those constraints being defined in very large modules
* my code not depending on anything in those constraint-defining modules
* my code depending on something in a file with a template with one of 
those constraints
* this whole situation being common enough to give me a death by a 
thousand papercuts

If any project aside from the standard library has a 7k line module 
defining things mainly used in template constraints, something is 
seriously weird. But on the plus side, it would only cost me 10 
milliseconds. Now, if *dozens* of projects did that, well, I'd be running 
to the relative simplicity of Java APIs long before I worried about 
compilation speed.

> Lazier compilation is indeed a projected benefit of this DIP. I did not
> want to dilute the thrust of the proposal with a remote promise.

Lazier compilation would *obviate* this DIP. Lazy compilation of 
selective and static imports would not require any parser changes and 
would make a lot of code faster (at the cost of allowing some things that 
don't compile to start compiling, as does your proposal). You can't get 
any performance advantages outside templates without implementing lazy 
imports.



Appendix A: templates that would receive 'with(import foo)' in Phobos.

I used a relatively simple regex to look for this. If someone put more 
than one line between a template and its constraints, or comments with 
certain formatting, I may have missed it. However, that would violate the 
phobos style guide.

This doesn't include templates that use constraints defined in their own 
modules, because that module must already be parsed and no efficiency 
gains could be realized.

It also omits a few cases where the module has a strong reason to import 
the dependency aside from template constraints. I believe this was only 
two constraints defined in std.digest, used in one or two other modules.

algorithm/comparison.d:98:template among(values...)
experimental/typecons.d:82:private template implementsInterface(Source, 
Targets...)
experimental/typecons.d:94:template implementsInterface()
experimental/typecons.d:126:private template implementsInterface(Source, 
Targets...)
experimental/typecons.d:184:template wrap(Targets...)
experimental/typecons.d:237:template wrap(Source)
range/package.d:2069:template Take(R)
range/package.d:3501:template Cycle(R)
range/interfaces.d:277:template MostDerivedInputRange(R)
range/interfaces.d:336:template InputRangeObject(R)
numeric.d:678:template FPTemporary(F)
conv.d:3894:template octal(alias decimalInteger)
utf.d:1136:package template codeUnitLimit(S)
typecons.d:1779:private mixin template RebindableCommon(T, U, alias This)
typecons.d:1838:template Rebindable(T)
typecons.d:4239:template wrap(Targets...)
typecons.d:4252:template wrap(Source)
typecons.d:4412:template wrap(Targets...)
typecons.d:4429:template unwrap(Target)
typecons.d:4461:template unwrap(Target)
format.d:657:template FormatSpec(Char)

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/23/2016 05:33 PM, Chris Wright wrote:

On Fri, 23 Dec 2016 11:25:41 -0500, Andrei Alexandrescu wrote:

Dependency-Carrying Declarations allow scalable template libraries.



template libraries


*This* is the part I hadn't noticed before, and it explains the confusion
I had.

If you *only* apply this to templates, it works. The current situation is
a result of where template constraints are located. If there were a
`static throw` equivalent for indicating that parameters were invalid, or
if template constraints were in an `in` block like contracts, then this
wouldn't even have come up. If phobos went for object oriented code
instead of templates (as an example, not a recommendation), then this
wouldn't be an issue.


I acknowledge that if the language were defined a different way this 
issue wouldn't have come up. But this is a truism - one can say that 
about any issue.



However, at that point, it would be utterly useless to me. I'm looking at
my entire dub package cache, plus the ten-ish most recently updated dub
packages:

* units-d uses allSatisfy. Once.
* vibe has two structs that would benefit, except they're inside a
unittest. I've never compiled dub's unittests.


Fair enough. I reckon a number of traditional ways of designing software 
would not be helped radically by DIP1005.



Template constraints have little adoption outside phobos.


That will change. We definitely need to do all we can to support and 
improve language support for template constraints.



When they *are*
used, they tend to use language facilities instead of templates to
express the condition. And when a template is used, it tends to be
defined in the same module where it's used.


That may be true for some code today but not for future code. std.traits 
gets larger and better with more interesting introspection capabilities. 
I envision introspection as a core differentiating feature of D that 
will put it ahead of all other languages.



I grant that everyone uses phobos, and phobos uses template constraints a
lot. If it's *just* a problem inside phobos, though, there's another
obvious solution: split up modules that tend to house a lot of template
constraints. Split up the modules that use a wide variety of template
constraints.


This point is discussed carefully in DIP1005. Please let me know if 
anything needs to be added.



Now, if you want to apply this to things that are *not* templates, then
you could get a lot further. However, you would end up with code that
compiles when it wouldn't today. Code that compiles because the portions
you use would compile if they were on their own, while other bits
wouldn't. That's the status quo for templates, even no-arg templates, but
a change from what we currently do everywhere else.

And *that* is what would make it equivalent to use static or selective
imports. It would also increase the utility from my perspective from "why
the hell are we even doing this?" to "that's kinda nice".


Lazier compilation is indeed a projected benefit of this DIP. I did not 
want to dilute the thrust of the proposal with a remote promise.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 23 Dec 2016 11:25:41 -0500, Andrei Alexandrescu wrote:
> Dependency-Carrying Declarations allow scalable template libraries.

> template libraries

*This* is the part I hadn't noticed before, and it explains the confusion 
I had.

If you *only* apply this to templates, it works. The current situation is 
a result of where template constraints are located. If there were a 
`static throw` equivalent for indicating that parameters were invalid, or 
if template constraints were in an `in` block like contracts, then this 
wouldn't even have come up. If phobos went for object oriented code 
instead of templates (as an example, not a recommendation), then this 
wouldn't be an issue.

However, at that point, it would be utterly useless to me. I'm looking at 
my entire dub package cache, plus the ten-ish most recently updated dub 
packages:

* units-d uses allSatisfy. Once.
* vibe has two structs that would benefit, except they're inside a 
unittest. I've never compiled dub's unittests.

Template constraints have little adoption outside phobos. When they *are* 
used, they tend to use language facilities instead of templates to 
express the condition. And when a template is used, it tends to be 
defined in the same module where it's used.

I grant that everyone uses phobos, and phobos uses template constraints a 
lot. If it's *just* a problem inside phobos, though, there's another 
obvious solution: split up modules that tend to house a lot of template 
constraints. Split up the modules that use a wide variety of template 
constraints.

Then I can decide whether the convenience of not hunting for narrower 
imports is worth the extra quarter second of compilation.



Now, if you want to apply this to things that are *not* templates, then 
you could get a lot further. However, you would end up with code that 
compiles when it wouldn't today. Code that compiles because the portions 
you use would compile if they were on their own, while other bits 
wouldn't. That's the status quo for templates, even no-arg templates, but 
a change from what we currently do everywhere else.

And *that* is what would make it equivalent to use static or selective 
imports. It would also increase the utility from my perspective from "why 
the hell are we even doing this?" to "that's kinda nice".

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

Major update adding an experiment that shows the cost of top-level imports.

https://github.com/dlang/DIPs/pull/51

https://github.com/dlang/DIPs/blob/a3ef4e25cfb9f884fee29edb5553a3a2b840f679/DIPs/DIP1005.md

Relevant added text:

Another matter we investigated is how reducing top-level imports 
influences build times and the size of the object files produced. We do 
not have an experimental implementation of this DIP, so measuring impact 
directly was not possible. We did the converse experiment---adding 
top-level imports.


In a separate branch of the standard library code, for each module we 
added all nested imports back to the top level. Some hand-editing was 
needed after that because of clashes in symbol names. Also, some imports 
needed to be removed because of circular dependencies and related 
limitations in the language's design and implementation. The resulting 
setup can be seen in 
[PR4992](https://github.com/dlang/phobos/pull/4992). Then for each 
module in the standard library we compiled one file that consists of 
exactly one `import` declaration, monitoring compile time and object 
file size. Appendix B displays build times and size of object files 
produced by this experiment.


|Aggregate|Time (top-level)|Time (nested)|Object size (top-level)|Object 
size (nested)|

|---|---|---|---|---|
|Median|320ms|13788|32ms|4296|
|Average|287.6ms|13437.2|64.6ms|5734.1|

As expected, the experiment shows that both build times and object file 
sizes were improved by moving imports away from the top level. We 
estimate that eliminating the 10.5x slack dependency fan-in will bring 
import costs down to negligible and also bring object file size down.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Fri, 23 Dec 2016 07:48:55 -0500, Andrei Alexandrescu wrote:

> On 12/22/16 10:31 PM, Chris Wright wrote:
>> It's two benefits for one disadvantage.
> 
> One won't make decisions by taking the difference of pros and cons,
> right? -- Andrei

You'd use the weighted difference, naturally.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/22/16 9:53 PM, Chris Wright wrote:

In point of fact, selective and static imports should be *faster* than
Andrei's way. Consider:

  static import myapp.users, std.socket;
  bool isUserOnline(myapp.users.User user, std.socket.Socket socket);

This has to locate a declaration named `User` in myapp.users, and it has
to locate a declaration named `Socket` in std.socket.

But let's look at Andrei's way:

  bool isUserOnline(User user, Socket userSocket)
  import myapp.users, std.socket;


with (static import myapp.users, std.socket)
bool isUserOnline(myapp.users.User user, std.socket.Socket socket);


Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/22/16 9:53 PM, Chris Wright wrote:

  bool isUserOnline(User user, Socket userSocket)
  import myapp.users, std.socket;


Has changed to

with (import myapp.users, std.socket)
bool isUserOnline(User user, Socket userSocket);

Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/22/16 10:31 PM, Chris Wright wrote:

It's two benefits for one disadvantage.


One won't make decisions by taking the difference of pros and cons, 
right? -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

I'm looking at this part:

> The manual conversion of std.array to the "static import" form is shown
> here. It leads to the expected lengthening of the symbols used in
> declarations, which appears to eliminate one disadvantage by
> introducing another.

You get longer declarations when you're reading and writing code. This 
shouldn't impact the docs. Not arguing that there is a disadvantage, just 
pointing out the scope is relatively small.

All your references are entirely unambiguous. If I'm reading the source 
and I'm unfamiliar with phobos, I don't have to wonder if that template 
constraint comes from std.string or std.traits.

It's two benefits for one disadvantage.

You can also have renamed imports to reduce the amount of typing:

  static import trait = std.traits;
  trait.ForeachType!Range[] array(Range)(Range r)
  if (trait.isIterable!Range &&
  trait.isNarrowString!Range &&
  !trait.isInfinite!Range)) {
  }

> The manual conversion of std.array to the "selective import" form is 
> shown here. Conversion was successful but because it collapses all 
> imports at the top, it does not make it much easier to identify e.g. 
> what dependencies would be pulled if a given artifact in std.array were 
> used.

It's still trivial for the compiler to detect which modules a particular 
declaration depends on.

If I'm in vim, I can put my cursor on the declaration, hit '*', navigate 
to the top of the document, and find the next match. It's like five 
keypresses. With VS Code or the like, I can similarly go to the top of 
the file and find the first match without much trouble.

And this process tells me which module defines the symbol. Either the 
first hit is at the top of the file in the import section, and I know 
which file to look in, or it's not, and I know it's defined locally (and 
there's a good chance I'm at the declaration).

With your proposal, when I'm lucky, I know the declaration must be either 
in the current file or in the locally imported file. (When I'm unlucky, 
the file has some top-level imports or multiple local imports.) And since 
the utility of this is mostly for large modules, I'm still going to pull 
out grep. It *does* help when I need to import the same thing elsewhere, 
at least.

> Again the manual process was highly nontrivial.

This is also true for your proposal.

CyberShadow also offered to whip up a tool to automate the conversion to 
selective imports. Presumably they would be able to do something similar 
with your proposal.

So this is kind of irrelevant.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Chris Wright via Digitalmars-d]

On Thu, 22 Dec 2016 10:17:33 +, Joakim wrote:
> Opening a file or 10 is extremely cheap compared to all the other costs
> of the compiler.  Purely from a technical cost perspective,
> I'm not sure even scoped imports were worth it, as my simple
> investigation below suggests.

The compiler doesn't merely have to open the file. It has to run at least 
partial semantic analysis on it in order to locate symbols. When 
templates are involved, this can become quite expensive.

With static and selective imports, this step can be entirely avoided -- 
the current file tells you which files contain which declarations. With 
either, the compiler can tell exactly which file contains which 
declarations, so it can avoid pulling in anything beyond what's strictly 
necessary.

With declaration-scoped imports, you avoid the same step. However, it 
requires the maintainer's discipline to reduce the declaration's imports 
to the minimal possible set of imports.

For instance, I have a declaration:

  import myapp.users, std.socket;
  bool isUserOnline(User user, Socket userSocket);

I decide that this needs compilation time optimizations. The current way:

  static import myapp.users, std.socket;
  bool isUserOnline(myapp.users.User user, std.socket.Socket socket);

And Andrei's way:

  bool isUserOnline(User user, Socket userSocket)
  import myapp.users, std.socket;

I refactor things so that this check finds the user socket on its own. 
The current way:

  static import myapp.users, std.socket;
  bool isUserOnline(myapp.users.User user);

Eh, I forgot I don't need std.socket anymore, but this costs a few 
microseconds of compiler time to add it to the current symbol table. It 
has to allocate a lazily expanded module stub. Shouldn't be a huge deal.

Andrei's way:

  bool isUserOnline(User user)
  import myapp.users, std.socket;

Again, I forgot to update the imports, but this time the compiler has to 
read std.socket from disk, parse it, run semantic on it, and import all 
its top-level symbols into the scope's symbol table. Because there's 
nothing here that says the 'User' type is in myapp.users instead of 
std.socket.

---

In point of fact, selective and static imports should be *faster* than 
Andrei's way. Consider:

  static import myapp.users, std.socket;
  bool isUserOnline(myapp.users.User user, std.socket.Socket socket);

This has to locate a declaration named `User` in myapp.users, and it has 
to locate a declaration named `Socket` in std.socket.

But let's look at Andrei's way:

  bool isUserOnline(User user, Socket userSocket)
  import myapp.users, std.socket;

Here, the compiler has to search *both* myapp.users and std.socket for a 
declaration named `User`, then it has to search both for `Socket`. (Even 
if it finds `User` in the first, it still needs to search the second in 
case both define that symbol.)

You go from O(distinct number of types referenced) lookups to O(types * 
imports).

Granted, you'll usually have between one and three types, between one and 
three imports, so the point is a bit less salient.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/22/2016 05:17 AM, Joakim wrote:

On Wednesday, 21 December 2016 at 15:16:34 UTC, Andrei Alexandrescu wrote:

In first approximation, whether a file gets opened or not makes a
difference (filesystem operations (possibly including networking),
necessity to rebuild if dependent code is changed. The analysis shows
there is significant overhead remaining, on average 10.5 additional
files per unused import.


Opening a file or 10 is extremely cheap compared to all the other costs
of the compiler.  Purely from a technical cost perspective, I'm not sure
even scoped imports were worth it, as my simple investigation below
suggests.


This is a misunderstanding. (I'll make the DIP clearer.) It's not about 
the cost of opening the file per se; I'm using the number of files 
opened as a proxy for all work involved in processing a file. Meaning, 
if you import 10 files you're likely to do roughly 10x the work of 
importing one file.



I just tried compiling phobos and its unittests for dmd 2.066.1 and
2.067.1, the dmd releases from right before and right after Ilya's PRs
linked above (compiling phobos from the older release with the newer dmd
didn't work and I wasn't interested in porting it).  I found they took
about the same amount of time to compile their respective phobos and the
later version consistently took 15 seconds longer to compile the
unittests on a single core.


Unittesting Phobos (whether in separation or together) will instantiate 
everything so they are the case when local imports do _not_ make any 
difference. That's why DIP1005 uses that case as an estimate of 
everything in an imported module being used.


Compiling Phobos in its entirety (one command) and measuring that time 
is also of tenuous relevance; I need to think of it but at first sight 
the cost of unnecessary imports is collapsed, and I suspect (and will 
measure) that either way most modules are in fact imported.


One measurement of interest is: write a module that imports exactly one 
stdlib module, compile it, and measure time. Looking at the generated 
object and executable size would be also of interest. I'll do that.



This suggests that there has been essentially no technical benefit to
scoped imports, that it is largely superficial.  That's fine, I think
it's worth it just from the standpoint of understanding where most
dependencies are coming from.  I don't think an additional syntax change
is necessary for the remaining few module-scope dependencies for
template constraints.

Note on investigation: Of course, some other relevant factors could have
changed in dmd and phobos between those two releases, so the result is
certainly not conclusive.  But the fact that there was no decrease in
compile times while phobos took around the same amount of time is highly
suggestive.

I'm uninterested in investigating further given the consistent
hand-waving justifications for this DIP.  If somebody else had submitted
this DIP, it would have been quickly shot down, and rightly so.


Walter and I have the role of scrutinizing every addition to the 
language (and reject most). It is natural that our own work is met with 
increased scrutiny. Picking to death anything and everything we say or 
do is a staple in this community, and a rite of passage on github. It is 
of course impossible to know what would have happened if the proposal 
were made by someone else. All I can say is Walter knew nothing about it 
and said it is good (except for the initial syntax; he's on board with 
"with").


Anyhow, not to worry. The burden of proof is on the DIP. I'll take a 
look at making some more measurements.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[piotrklos via Digitalmars-d]

On Wednesday, 21 December 2016 at 15:16:34 UTC, Andrei 
Alexandrescu wrote:

On 12/20/2016 11:32 PM, Joakim wrote:

(...)
I don't know how to make matters much clearer than the current 
document. Any suggestions are welcome. The section "Workaround: 
Are Local Imports Good Enough?" discusses the material cost in 
terms of extra files that need to be opened and parsed (some 
unnecessarily) in order to complete a compilation. The 
"Rationale" part of the document discusses the costs in terms 
of maintainability, clarity, and documentation.



Thanks,

Andrei


Stipulation: I think the difference of opinion may be caused by 
working on different sizes of projects in ones career (tens of 
thousands vs millions of LoC).


Suggestion 1: Maybe the DIP should point out that the cost of 
redundant imports (however small) tends to grow quadratically 
with code size (size of import tree times the number of 
compilations).
If this is not the case then maybe the DIP is really in the wrong 
direction.


Suggestion 2:
Implement the DIP, autogenerate millions of lines of D code (2 
versions: with and without DCDs) and see which version of DMD 
compiles them faster. This may also expose other ways to improve 
scalability without implementing this DIP.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timothee Cour via Digitalmars-d]

> I just tried compiling phobos and its unittests for dmd 2.066.1 and
2.067.1

I think it's worth considering compile time for partial recompilation as
opposed to full compilation. The benifit of this DIP should be more
pronounced there since there'll be more opportunities to skip parsing
modules in that case. Partial recompilation is what matters most during
`edit compile debug cycle` anyways

On Thu, Dec 22, 2016 at 2:17 AM, Joakim via Digitalmars-d <
digitalmars-d@puremagic.com> wrote:

> On Wednesday, 21 December 2016 at 15:16:34 UTC, Andrei Alexandrescu wrote:
>
>> On 12/20/2016 11:32 PM, Joakim wrote:
>>
>>> Second, as I noted above, most top-level imports have not been made
>>> selective yet, because of the symbol leak bug that was recently fixed by
>>> Martin.  You will see in my PRs that I only list those symbols as a
>>> comment, because I could not turn those into selective imports yet.  If
>>> the compiler is doing its job right, selective imports should greatly
>>> reduce the cost of importing a module, even if your metric would still
>>> show the module being imported.
>>>
>>
>> That is not relevant to this section, which discusses the effectiveness
>> of using local imports with the current compilation technology. Per the
>> section's opening sentence:
>>
>> A legitimate question to ask is whether consistent use of local
>>> imports wherever possible would be an appropriate approximation of
>>> the Dependency-Carrying Declarations goal with no change in the
>>> language at all.
>>>
>>
> It is relevant because it could further reduce the cost from module-scope
> imports.  Whether your section only chooses to focus on locally scoped
> imports and ignore the impact of selective imports is irrelevant to me.
>
> Finally, while it's nice to know the extent of the dependency graph,
>>> what really matters is the _cost_ of each link of the graph, which is
>>> what I keep hammering on.  If the cost of links is small, it doesn't
>>> matter how entangled it is.  If minimizing the dependency graph through
>>> scoping alone, ie without implementing this DIP, removes most of the
>>> cost, that's all I care about.
>>>
>>
>> In first approximation, whether a file gets opened or not makes a
>> difference (filesystem operations (possibly including networking),
>> necessity to rebuild if dependent code is changed. The analysis shows there
>> is significant overhead remaining, on average 10.5 additional files per
>> unused import.
>>
>
> Opening a file or 10 is extremely cheap compared to all the other costs of
> the compiler.  Purely from a technical cost perspective, I'm not sure even
> scoped imports were worth it, as my simple investigation below suggests.
>
> If the current document could be clearer in explaining costs, please let
>> me know.
>>
>
> Yes, please explain what significant "overhead" was reduced by scoping
> imports and would be futher reduced by this DIP.  Opening files doesn't cut
> it.
>
> My point is that the dependency graph matters, but now that we're
>>> getting down to the last entanglements, we need to know the cost of
>>> those last links.  Your dependency analysis gives us some quantitative
>>> idea of the size of the remaining graph, but tells us nothing about the
>>> cost of those links. That's what I'm looking for.
>>>
>>> I will spend some time now investigating those costs with sample code.
>>> My request all along has been that you give us some idea of those costs,
>>> if you know the answer already.
>>>
>>
>> I don't know how to make matters much clearer than the current document.
>> Any suggestions are welcome. The section "Workaround: Are Local Imports
>> Good Enough?" discusses the material cost in terms of extra files that need
>> to be opened and parsed (some unnecessarily) in order to complete a
>> compilation. The "Rationale" part of the document discusses the costs in
>> terms of maintainability, clarity, and documentation.
>>
>
> I don't know how to make it clearer that that's not good enough.  You seem
> to understand that I want more justification than hand-waving about
> "scalable" and "overhead," which is why you now give the cost of opening
> files as justification, but you don't seem to have anything more
> substantive than that flimsy claim.
>
> I just tried compiling phobos and its unittests for dmd 2.066.1 and
> 2.067.1, the dmd releases from right before and right after Ilya's PRs
> linked above (compiling phobos from the older release with the newer dmd
> didn't work and I wasn't interested in porting it).  I found they took
> about the same amount of time to compile their respective phobos and the
> later version consistently took 15 seconds longer to compile the unittests
> on a single core.
>
> This suggests that there has been essentially no technical benefit to
> scoped imports, that it is largely superficial.  That's fine, I think it's
> worth it just from the standpoint of understanding where most dependencies
> are coming from.  I don't think an 

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Wednesday, 21 December 2016 at 15:16:34 UTC, Andrei 
Alexandrescu wrote:

On 12/20/2016 11:32 PM, Joakim wrote:
Second, as I noted above, most top-level imports have not been 
made
selective yet, because of the symbol leak bug that was 
recently fixed by
Martin.  You will see in my PRs that I only list those symbols 
as a
comment, because I could not turn those into selective imports 
yet.  If
the compiler is doing its job right, selective imports should 
greatly
reduce the cost of importing a module, even if your metric 
would still

show the module being imported.


That is not relevant to this section, which discusses the 
effectiveness of using local imports with the current 
compilation technology. Per the section's opening sentence:



A legitimate question to ask is whether consistent use of local
imports wherever possible would be an appropriate 
approximation of

the Dependency-Carrying Declarations goal with no change in the
language at all.


It is relevant because it could further reduce the cost from 
module-scope imports.  Whether your section only chooses to focus 
on locally scoped imports and ignore the impact of selective 
imports is irrelevant to me.


Finally, while it's nice to know the extent of the dependency 
graph,
what really matters is the _cost_ of each link of the graph, 
which is
what I keep hammering on.  If the cost of links is small, it 
doesn't
matter how entangled it is.  If minimizing the dependency 
graph through
scoping alone, ie without implementing this DIP, removes most 
of the

cost, that's all I care about.


In first approximation, whether a file gets opened or not makes 
a difference (filesystem operations (possibly including 
networking), necessity to rebuild if dependent code is changed. 
The analysis shows there is significant overhead remaining, on 
average 10.5 additional files per unused import.


Opening a file or 10 is extremely cheap compared to all the other 
costs of the compiler.  Purely from a technical cost perspective, 
I'm not sure even scoped imports were worth it, as my simple 
investigation below suggests.


If the current document could be clearer in explaining costs, 
please let me know.


Yes, please explain what significant "overhead" was reduced by 
scoping imports and would be futher reduced by this DIP.  Opening 
files doesn't cut it.


My point is that the dependency graph matters, but now that 
we're
getting down to the last entanglements, we need to know the 
cost of
those last links.  Your dependency analysis gives us some 
quantitative
idea of the size of the remaining graph, but tells us nothing 
about the

cost of those links. That's what I'm looking for.

I will spend some time now investigating those costs with 
sample code.
My request all along has been that you give us some idea of 
those costs,

if you know the answer already.


I don't know how to make matters much clearer than the current 
document. Any suggestions are welcome. The section "Workaround: 
Are Local Imports Good Enough?" discusses the material cost in 
terms of extra files that need to be opened and parsed (some 
unnecessarily) in order to complete a compilation. The 
"Rationale" part of the document discusses the costs in terms 
of maintainability, clarity, and documentation.


I don't know how to make it clearer that that's not good enough.  
You seem to understand that I want more justification than 
hand-waving about "scalable" and "overhead," which is why you now 
give the cost of opening files as justification, but you don't 
seem to have anything more substantive than that flimsy claim.


I just tried compiling phobos and its unittests for dmd 2.066.1 
and 2.067.1, the dmd releases from right before and right after 
Ilya's PRs linked above (compiling phobos from the older release 
with the newer dmd didn't work and I wasn't interested in porting 
it).  I found they took about the same amount of time to compile 
their respective phobos and the later version consistently took 
15 seconds longer to compile the unittests on a single core.


This suggests that there has been essentially no technical 
benefit to scoped imports, that it is largely superficial.  
That's fine, I think it's worth it just from the standpoint of 
understanding where most dependencies are coming from.  I don't 
think an additional syntax change is necessary for the remaining 
few module-scope dependencies for template constraints.


Note on investigation: Of course, some other relevant factors 
could have changed in dmd and phobos between those two releases, 
so the result is certainly not conclusive.  But the fact that 
there was no decrease in compile times while phobos took around 
the same amount of time is highly suggestive.


I'm uninterested in investigating further given the consistent 
hand-waving justifications for this DIP.  If somebody else had 
submitted this DIP, it would have been quickly shot down, and 
rightly so.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/21/16 6:40 PM, Timothee Cour via Digitalmars-d wrote:

Andrei: ping on this? (especially regarding allowing `:`)


I think "lazy" is a bit too cute. "with" is so close to what's actually 
needed, it would be a waste to not use it.


Generally I'm weary of the use of ":" (never liked it - it makes code 
dependent on long-distance context) so I'd rather snatch the opportunity 
to avoid it.



Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timothee Cour via Digitalmars-d]

On Mon, Dec 19, 2016 at 9:33 PM, Timothee Cour 
wrote:

> what about using `lazy` instead of `with`:
>
> `with(import foo)`
> =>
> `lazy(import foo)`
>
> advantages:
> * avoids confusion regarding usual scoping rules of `with` ;
> * conveys that the import is indeed lazy
>
> Furthermore (regardless of which keyword is used), what about allowing `:`
> ```
> // case 1
> lazy(import foo)
> void fun(){}
>
> // case 2
> lazy(import foo) {
>   void fun(){}
> }
>
> // case 3 : this is new
> lazy(import foo):
> void fun1(){}
> void fun2(){}
> ```
>
> advantages:
>
> * same behavior as other constructs which don't introduce a scope:
> ```
> // case 1, 2 3 are allowed:
> version(A):
> static if(true):
> private:
> void fun(){}
> ```
>
> * avoids nesting when case 3 is used (compared to when using `{}`)
>
> * I would argue that grouping lazy imports is actually a common case;
> without case 3, the indentation will increase.
>


Andrei: ping on this? (especially regarding allowing `:`)

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 09:31 AM, Dmitry Olshansky wrote:

On 12/13/16 11:33 PM, Andrei Alexandrescu wrote:

Destroy.

https://github.com/dlang/DIPs/pull/51/files


Andrei


Just a thought but with all of proliferation of imports down to each
declaration comes the pain that e.g. renaming a module cascades to
countless instances of import statements. This is true of local imports
as well but the problem gets bigger.


https://github.com/dlang/DIPs/pull/51/commits/d4ef6826dacedc38f822e48bec2186d93040fb42

Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 11:32 PM, Joakim wrote:

On Tuesday, 20 December 2016 at 20:51:54 UTC, Andrei Alexandrescu wrote:
Thanks for this analysis of the remaining dependency graph, it is worth
looking at.  Allow me to poke some holes in it.

To begin with, the amount of scoping that has been done is overstated,
if you simply count scoped imports and compare it to module-level
imports.  Each module-level import has to be replicated multiple times
for each local scope, especially in unittest blocks.  A better number is
more like 20-30%, as I pointed out 4 out of 13 modules remain at
top-level in std.array. Using that metric, a 3-4X reduction in top-level
imports has led to at least a 2.2x improvement in imported files, so the
effort has been more meaningful than you conclude.


Fixed. I also made a note about the need to duplicate imports as they 
are pushed down.



Second, as I noted above, most top-level imports have not been made
selective yet, because of the symbol leak bug that was recently fixed by
Martin.  You will see in my PRs that I only list those symbols as a
comment, because I could not turn those into selective imports yet.  If
the compiler is doing its job right, selective imports should greatly
reduce the cost of importing a module, even if your metric would still
show the module being imported.


That is not relevant to this section, which discusses the effectiveness 
of using local imports with the current compilation technology. Per the 
section's opening sentence:



A legitimate question to ask is whether consistent use of local
imports wherever possible would be an appropriate approximation of
the Dependency-Carrying Declarations goal with no change in the
language at all.


The section "Alternative: Lazy Imports" discusses how static or local 
imports could be used in conjunction with new compilation technologies. 
If there are improvements to be made there, please advise.



Third, checking some of the output from the commands you ran in your
script shows that up to half of the imported modules are from druntime.
I noted earlier that Ilya usually didn't bother scoping top-level
druntime imports, because he perceived their cost to be low (I scoped
them too in the handful I cleaned up, just for completeness).  As far as
I know, nobody has bothered to spend any time scoping druntime, so it
would be better if you filtered out druntime imports from your analysis.


Fixed to only count imports from std.


Finally, while it's nice to know the extent of the dependency graph,
what really matters is the _cost_ of each link of the graph, which is
what I keep hammering on.  If the cost of links is small, it doesn't
matter how entangled it is.  If minimizing the dependency graph through
scoping alone, ie without implementing this DIP, removes most of the
cost, that's all I care about.


In first approximation, whether a file gets opened or not makes a 
difference (filesystem operations (possibly including networking), 
necessity to rebuild if dependent code is changed. The analysis shows 
there is significant overhead remaining, on average 10.5 additional 
files per unused import.


If the current document could be clearer in explaining costs, please let 
me know.



I have noted one example above, where _a single DCD in phobos_, ie a
scoped, selective import, had gigantic costs in terms of executable
size, where entire modules were included because of it.  If that's the
case more generally, then _no_ amount of dependency disentangling will
matter, because the cost of single DCDs is still huge.  Perhaps that's
just an isolated issue however, it needs to be investigated.


That seems an unrelated matter. Yes, you could pull a large dependency 
in one shot with old or new technology.



My point is that the dependency graph matters, but now that we're
getting down to the last entanglements, we need to know the cost of
those last links.  Your dependency analysis gives us some quantitative
idea of the size of the remaining graph, but tells us nothing about the
cost of those links. That's what I'm looking for.

I will spend some time now investigating those costs with sample code.
My request all along has been that you give us some idea of those costs,
if you know the answer already.


I don't know how to make matters much clearer than the current document. 
Any suggestions are welcome. The section "Workaround: Are Local Imports 
Good Enough?" discusses the material cost in terms of extra files that 
need to be opened and parsed (some unnecessarily) in order to complete a 
compilation. The "Rationale" part of the document discusses the costs in 
terms of maintainability, clarity, and documentation.



Thanks,

Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Tuesday, 20 December 2016 at 20:51:54 UTC, Andrei Alexandrescu 
wrote:
I've asked Joakim about this via email just now, likely other 
folks also know the answer:


1. I found these PRs related to local imports:

https://github.com/dlang/phobos/pull/4361
https://github.com/dlang/phobos/pull/4365
https://github.com/dlang/phobos/pull/4370
https://github.com/dlang/phobos/pull/4373
https://github.com/dlang/phobos/pull/4379
https://github.com/dlang/phobos/pull/4392
https://github.com/dlang/phobos/pull/4467

Are there more? Is there a central point for them (such as a 
bugzilla issue)?


Ilya lists a lot more above, he did most of the work.  No central 
point that I know of.



2. I see you've done a bunch of work in the area. Where, in your
estimate, are we on the spectrum of making imports local 
without a
major reorganization? Any low-hanging fruit to look at, or have 
Joakim,
Ilya, Jack and others made a pass through most/all modules 
already?


There is more to be done, but my guess would be 80-90% done for 
Phobos. Ilya scoped a lot, but usually left druntime imports 
alone.  Top-level module imports mostly don't use selective 
imports yet, because Martin wanted to hold off till he was sure 
the symbol leak bug was fixed.


On Tuesday, 20 December 2016 at 22:08:38 UTC, Andrei Alexandrescu 
wrote:

On 12/20/2016 03:46 AM, Joakim wrote:
I didn't just say "eh:" I gave evidence for why I think the 
problem is
minimal and asked why it's so important to scope those last 
3-4 imported

modules too, which you didn't answer.


You have asked for a smoking gun, and one has been found.

I have just uploaded a major update that carefully analyzes the 
improvements brought about by switching to local imports in the 
D Standard Library. Please refer to the section "Workaround: 
Are Local Imports Good Enough?" and Appendix A:


https://github.com/dlang/DIPs/pull/51/files

https://github.com/dlang/DIPs/blob/91baecedcfe7cb75ac22e66478722ec797ebb901/DIPs/DIP1005.md


Thanks for this analysis of the remaining dependency graph, it is 
worth looking at.  Allow me to poke some holes in it.


To begin with, the amount of scoping that has been done is 
overstated, if you simply count scoped imports and compare it to 
module-level imports.  Each module-level import has to be 
replicated multiple times for each local scope, especially in 
unittest blocks.  A better number is more like 20-30%, as I 
pointed out 4 out of 13 modules remain at top-level in std.array. 
Using that metric, a 3-4X reduction in top-level imports has led 
to at least a 2.2x improvement in imported files, so the effort 
has been more meaningful than you conclude.


Second, as I noted above, most top-level imports have not been 
made selective yet, because of the symbol leak bug that was 
recently fixed by Martin.  You will see in my PRs that I only 
list those symbols as a comment, because I could not turn those 
into selective imports yet.  If the compiler is doing its job 
right, selective imports should greatly reduce the cost of 
importing a module, even if your metric would still show the 
module being imported.


Third, checking some of the output from the commands you ran in 
your script shows that up to half of the imported modules are 
from druntime.  I noted earlier that Ilya usually didn't bother 
scoping top-level druntime imports, because he perceived their 
cost to be low (I scoped them too in the handful I cleaned up, 
just for completeness).  As far as I know, nobody has bothered to 
spend any time scoping druntime, so it would be better if you 
filtered out druntime imports from your analysis.


Finally, while it's nice to know the extent of the dependency 
graph, what really matters is the _cost_ of each link of the 
graph, which is what I keep hammering on.  If the cost of links 
is small, it doesn't matter how entangled it is.  If minimizing 
the dependency graph through scoping alone, ie without 
implementing this DIP, removes most of the cost, that's all I 
care about.


I have noted one example above, where _a single DCD in phobos_, 
ie a scoped, selective import, had gigantic costs in terms of 
executable size, where entire modules were included because of 
it.  If that's the case more generally, then _no_ amount of 
dependency disentangling will matter, because the cost of single 
DCDs is still huge.  Perhaps that's just an isolated issue 
however, it needs to be investigated.


My point is that the dependency graph matters, but now that we're 
getting down to the last entanglements, we need to know the cost 
of those last links.  Your dependency analysis gives us some 
quantitative idea of the size of the remaining graph, but tells 
us nothing about the cost of those links. That's what I'm looking 
for.


I will spend some time now investigating those costs with sample 
code.  My request all along has been that you give us some idea 
of those costs, if you know the answer already.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 05:08 PM, Andrei Alexandrescu wrote:

On 12/20/2016 03:46 AM, Joakim wrote:

I didn't just say "eh:" I gave evidence for why I think the problem is
minimal and asked why it's so important to scope those last 3-4 imported
modules too, which you didn't answer.


You have asked for a smoking gun, and one has been found.

I have just uploaded a major update that carefully analyzes the
improvements brought about by switching to local imports in the D
Standard Library. Please refer to the section "Workaround: Are Local
Imports Good Enough?" and Appendix A:

https://github.com/dlang/DIPs/pull/51/files

https://github.com/dlang/DIPs/blob/91baecedcfe7cb75ac22e66478722ec797ebb901/DIPs/DIP1005.md


Eh, looks like http://dillinger.io/ and github don't agree on table 
rendering... Fixed URL for nice viewing:


https://github.com/dlang/DIPs/blob/249b28ddd784220e44e343e78e5ea7a65c4c7bde/DIPs/DIP1005.md


Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 05:08 PM, Andrei Alexandrescu wrote:

On 12/20/2016 03:46 AM, Joakim wrote:

I didn't just say "eh:" I gave evidence for why I think the problem is
minimal and asked why it's so important to scope those last 3-4 imported
modules too, which you didn't answer.


You have asked for a smoking gun, and one has been found.

I have just uploaded a major update that carefully analyzes the
improvements brought about by switching to local imports in the D
Standard Library. Please refer to the section "Workaround: Are Local
Imports Good Enough?" and Appendix A:

https://github.com/dlang/DIPs/pull/51/files

https://github.com/dlang/DIPs/blob/91baecedcfe7cb75ac22e66478722ec797ebb901/DIPs/DIP1005.md


I've also added a supporting script that creates the table in Appendix 
A: 
https://github.com/andralex/DIPs/blob/9e9ebc8738ce04b2d85a92feafacc7ef81e59811/DIPs/DIP1005-countlines.zsh 
-- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 03:46 AM, Joakim wrote:

I didn't just say "eh:" I gave evidence for why I think the problem is
minimal and asked why it's so important to scope those last 3-4 imported
modules too, which you didn't answer.


You have asked for a smoking gun, and one has been found.

I have just uploaded a major update that carefully analyzes the 
improvements brought about by switching to local imports in the D 
Standard Library. Please refer to the section "Workaround: Are Local 
Imports Good Enough?" and Appendix A:


https://github.com/dlang/DIPs/pull/51/files

https://github.com/dlang/DIPs/blob/91baecedcfe7cb75ac22e66478722ec797ebb901/DIPs/DIP1005.md


Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timon Gehr via Digitalmars-d]

On 19.12.2016 06:31, deadalnix wrote:

On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:

Great, thanks. Please take a look at the accuracy of the discussion. I
expanded the "Workaround" section and moved it near the top.

https://github.com/dlang/DIPs/pull/51

https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md



Andrei


What's wrong with the parentheseless version ? The DIP says it looks
"out of place" but that doesn't strike me as a very good argument. IMO
the version without ";" is the way to go, as it doesn't require to add a
new syntax for imports.

Identical things looking identical is valuable.



+1.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Ilya Yaroshenko via Digitalmars-d]

On Tuesday, 20 December 2016 at 20:51:54 UTC, Andrei Alexandrescu 
wrote:
I've asked Joakim about this via email just now, likely other 
folks also know the answer:


1. I found these PRs related to local imports:

https://github.com/dlang/phobos/pull/4361
https://github.com/dlang/phobos/pull/4365
https://github.com/dlang/phobos/pull/4370
https://github.com/dlang/phobos/pull/4373
https://github.com/dlang/phobos/pull/4379
https://github.com/dlang/phobos/pull/4392
https://github.com/dlang/phobos/pull/4467

Are there more? Is there a central point for them (such as a 
bugzilla issue)?


2. I see you've done a bunch of work in the area. Where, in your
estimate, are we on the spectrum of making imports local 
without a
major reorganization? Any low-hanging fruit to look at, or have 
Joakim,
Ilya, Jack and others made a pass through most/all modules 
already?



Thanks!


This is 95% of my PRs for local imports. Few of them moves codes 
between files.


https://github.com/dlang/phobos/pull/2658
https://github.com/dlang/phobos/pull/2659
https://github.com/dlang/phobos/pull/2661
https://github.com/dlang/phobos/pull/2665
https://github.com/dlang/phobos/pull/2666
https://github.com/dlang/phobos/pull/2667
https://github.com/dlang/phobos/pull/2669
https://github.com/dlang/phobos/pull/2670
https://github.com/dlang/phobos/pull/2671
https://github.com/dlang/phobos/pull/2672
https://github.com/dlang/phobos/pull/2673
https://github.com/dlang/phobos/pull/2675
https://github.com/dlang/phobos/pull/2678
https://github.com/dlang/phobos/pull/2679
https://github.com/dlang/phobos/pull/2680
https://github.com/dlang/phobos/pull/2681
https://github.com/dlang/phobos/pull/2686
https://github.com/dlang/phobos/pull/2691
https://github.com/dlang/phobos/pull/2694
https://github.com/dlang/phobos/pull/2695
https://github.com/dlang/phobos/pull/2696
https://github.com/dlang/phobos/pull/2705
https://github.com/dlang/phobos/pull/2706
https://github.com/dlang/phobos/pull/2707
https://github.com/dlang/phobos/pull/2708
https://github.com/dlang/phobos/pull/2709
https://github.com/dlang/phobos/pull/2710
https://github.com/dlang/phobos/pull/2711
https://github.com/dlang/phobos/pull/2712
https://github.com/dlang/phobos/pull/2713
https://github.com/dlang/phobos/pull/2714
https://github.com/dlang/phobos/pull/2715
https://github.com/dlang/phobos/pull/2716
https://github.com/dlang/phobos/pull/2717
https://github.com/dlang/phobos/pull/2718
https://github.com/dlang/phobos/pull/2719
https://github.com/dlang/phobos/pull/2720
https://github.com/dlang/phobos/pull/2721
https://github.com/dlang/phobos/pull/2722
https://github.com/dlang/phobos/pull/2725
https://github.com/dlang/phobos/pull/2726
https://github.com/dlang/phobos/pull/2727
https://github.com/dlang/phobos/pull/2728
https://github.com/dlang/phobos/pull/2729
https://github.com/dlang/phobos/pull/2754
https://github.com/dlang/phobos/pull/2755

Ilya

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

I've asked Joakim about this via email just now, likely other folks also 
know the answer:


1. I found these PRs related to local imports:

https://github.com/dlang/phobos/pull/4361
https://github.com/dlang/phobos/pull/4365
https://github.com/dlang/phobos/pull/4370
https://github.com/dlang/phobos/pull/4373
https://github.com/dlang/phobos/pull/4379
https://github.com/dlang/phobos/pull/4392
https://github.com/dlang/phobos/pull/4467

Are there more? Is there a central point for them (such as a bugzilla 
issue)?


2. I see you've done a bunch of work in the area. Where, in your
estimate, are we on the spectrum of making imports local without a
major reorganization? Any low-hanging fruit to look at, or have Joakim,
Ilya, Jack and others made a pass through most/all modules already?


Thanks!

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Andrei Alexandrescu via Digitalmars-d]

On 12/20/2016 09:31 AM, Dmitry Olshansky wrote:

On 12/13/16 11:33 PM, Andrei Alexandrescu wrote:

Destroy.

https://github.com/dlang/DIPs/pull/51/files


Andrei


Just a thought but with all of proliferation of imports down to each
declaration comes the pain that e.g. renaming a module cascades to
countless instances of import statements. This is true of local imports
as well but the problem gets bigger.


Good point, I'll integrate it in the document. -- Andrei

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Meta via Digitalmars-d]

On Tuesday, 20 December 2016 at 01:06:01 UTC, Andrei Alexandrescu 
wrote:
Pushed again, now with the syntax using "with" proposed by 
Hatem Oraby and others.


https://github.com/dlang/DIPs/pull/51

https://github.com/dlang/DIPs/blob/71bde077488b566fba7603de6095b45984d9294a/DIPs/DIP1005.md


Andrei


"In addition, we propose the statement and declaration with 
(import ImportList). ImportList is any syntactical construct 
currently accepted by the import declaration. The with (import 
ImportList) obeys the following rules:


- Inside any function, with (Import ImportList) is a statement 
that introduces a scope. Inside the with, lookup considers the 
import local to the declaration (similar to the current handling 
of nested imports).
- Everywhere else, with (Import ImportList) is always a 
declaration and does not introduce a new scope. Lookup of symbols 
is the same as for the statement case."


I must've somehow missed this when reading through for the first 
time after your changes as I thought this was a glaring omission. 
Glad to see that this was covered as this rightly makes it 
turtles all the way down. Actually, you could completely replace 
import statements with this new construct, having the `import 
std.range` at the top level replaced with a module-wide attribute 
`with (import std.range):`. In that light my only real criticism 
is that it's kind of redundant having both this new form *and* 
local imports, and it's not clear which is better and why to a 
new user.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Meta via Digitalmars-d]

On Tuesday, 20 December 2016 at 14:31:38 UTC, Dmitry Olshansky 
wrote:

On 12/13/16 11:33 PM, Andrei Alexandrescu wrote:

Destroy.

https://github.com/dlang/DIPs/pull/51/files


Andrei


Just a thought but with all of proliferation of imports down to 
each declaration comes the pain that e.g. renaming a module 
cascades to countless instances of import statements. This is 
true of local imports as well but the problem gets bigger.



Dmitry Olshansky


Could you not have the old module just be empty and publicly 
import the new one, and also deprecate it so people have time to 
change their code? Also having special syntax makes finding every 
occurrence of an inline import a fairly simple search.

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Dmitry Olshansky via Digitalmars-d]

On 12/13/16 11:33 PM, Andrei Alexandrescu wrote:

Destroy.

https://github.com/dlang/DIPs/pull/51/files


Andrei


Just a thought but with all of proliferation of imports down to each 
declaration comes the pain that e.g. renaming a module cascades to 
countless instances of import statements. This is true of local imports 
as well but the problem gets bigger.



Dmitry Olshansky

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Joakim via Digitalmars-d]

On Tuesday, 20 December 2016 at 04:04:14 UTC, Andrei Alexandrescu 
wrote:

On 12/19/2016 01:47 AM, Joakim wrote:
Why do you care _so_ much about the module dependency graph?  
To make
this question concrete, let's look at an example, the 
std.array module

you keep mentioning.
This is what it looked like before Ilya scoped as
many imports as he could:

https://github.com/dlang/phobos/commit/3fcf723aa498b96de165361b5abb9d3450fdc069#diff-54cf8402b22024ae667d4048a5126f0e

That was a mess, similar to opaque C/C++ code, 13 modules 
imported at
module-scope were whittled down to 4.  You just made those 
more specific

in a commit related to this DIP, by listing the actual symbols
selectively imported from those four modules:

https://github.com/dlang/phobos/commit/e064d5664f92c4b2f0866c08f6d0290ba66825ed#diff-54cf8402b22024ae667d4048a5126f0e


If I'm looking at the template constraints for any particular 
function
and see a couple symbols I don't recognize, I don't think it's 
a big

deal to find the symbols in that list at the top.


It actually is. Some symbols in e.g. std.range are used in 
constraints, some others used in definitions; some in a UFCS 
manner. Some code runs during compilation, with errors gagged 
(as in __traits(compiles)) or not gagged. I don't know of an 
easy manual method to figure out who's who (which makes 
Vladimir's idea of tooling it with brute force awesome and 
scary at the same time).


I'm not sure why you care who's who.  I noted that if I'm looking 
at the constraints for a function, it's easy to check the 
selective imports at the top of the file for any symbols.  That's 
the common scenario.  I don't know why anyone would be checking 
the selective imports at the top to see where each symbol is 
actually used, which is the uncommon scenario you now present.


At any way, I don't see how I can use this tidbit to improve 
the DIP. Anything I could add to it to make it more compelling?


You could answer my question above. We have already scoped 90% of 
the dependencies, why is it so important to remove the remaining 
10% that we need to add new syntax?


In other words, D already allows you to scope most imports.  I 
don't
consider the dozen or two remaining symbols from templaint 
constraints
and function arguments to provide much overhead.  Rather, I 
consider the
weight of this additional syntax, ie the cognitive overhead 
from having
to remember and parse more syntax in my head, to be worse than 
the
remaining dependency reasoning problem you're trying to solve: 
the cost
outweights the benefit.  Perhaps that's subjective and others 
may disagree.


It is subjective and difficult to convert into action. Online 
review among folks who know next to nothing about each other 
does have its challenges. What happens here is I post a 
proposal for a problem that I believe is important for large D 
projects. And I get back "eh, that's not as important to me." 
At a traditional work place we'd know a lot about one another's 
strengths, specialization areas, and design sensibilities. 
Here, all I can do is look at your past PRs (that's why I 
emailed you asking for your github handle; I figure it's 
@joakim-noah). This makes it difficult to act on "I don't buy 
it" feedback.


I didn't just say "eh:" I gave evidence for why I think the 
problem is minimal and asked why it's so important to scope those 
last 3-4 imported modules too, which you didn't answer.


As for looking at my PRs, there were some links above, which show 
that is my handle, but I don't think you'll get much from that, 
as I haven't designed anything in github.com/dlang, only cleaning 
up and porting.


Now, there's also the question of purely technical benefits, 
like
compilation speed or executable bloat.  I looked at the latter 
a little
last summer, after Ilya had cleaned up a lot of the standard 
library:


http://forum.dlang.org/thread/gmjqfjoemwtvgqrtd...@forum.dlang.org

I found that commenting out a single scoped, selective import 
of
"std.string: format" in std.utf led to a 5% decrease in 
executable size
for "hello world."  This is a problem with how dmd compiles or 
appends
these module dependencies and would presumably still be there 
after this

DIP, as you would not remove the dependency.


That might be a related but distinct issue. Can you reproduce 
that experiment?


I went back and looked at that particular import and it appears 
Walter subsequently removed it:


https://github.com/dlang/phobos/pull/3455

I will experiment a bit more with some sample code and see what I 
find.


I think scoped, selective imports have been great at hacking 
away at the
module dependency graph, as you lay out.  It is not clear what 
technical
costs you see from the remaining few dependencies and if this 
DIP is the
best way to remove them.  I think you should explain why you 
want to
untangle the remaining dependency graph, and consider if this 
DIP is

really doing that much.


I've made a few more passes 

Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback

[Timothee Cour via Digitalmars-d]

what about using `lazy` instead of `with`:

`with(import foo)`
=>
`lazy(import foo)`

advantages:
* avoids confusion regarding usual scoping rules of `with` ;
* conveys that the import is indeed lazy

Furthermore (regardless of which keyword is used), what about allowing `:`
```
// case 1
lazy(import foo)
void fun(){}

// case 2
lazy(import foo) {
  void fun(){}
}

// case 3 : this is new
lazy(import foo):
void fun1(){}
void fun2(){}
```

advantages:

* same behavior as other constructs which don't introduce a scope:
```
// case 1, 2 3 are allowed:
version(A):
static if(true):
private:
void fun(){}
```

* avoids nesting when case 3 is used (compared to when using `{}`)

* I would argue that grouping lazy imports is actually a common case;
without case 3, the indentation will increase.






On Mon, Dec 19, 2016 at 8:56 PM, Timothee Cour 
wrote:

> Andrei: you can use blockquotes to make long lines wrap when quoting text:
>
> https://github.com/adam-p/markdown-here/wiki/Markdown-
> Here-Cheatsheet#blockquotes
>
> that way, we don't have to scroll everytime there's a quote, eg:
>
> replace:
> Such scaffolding is of course
> with:
> > Such scaffolding is of course
>
>
>
>
> On Mon, Dec 19, 2016 at 8:04 PM, Andrei Alexandrescu via Digitalmars-d <
> digitalmars-d@puremagic.com> wrote:
>
>> On 12/19/2016 01:47 AM, Joakim wrote:
>>
>>> Why do you care _so_ much about the module dependency graph?  To make
>>> this question concrete, let's look at an example, the std.array module
>>> you keep mentioning.
>>> This is what it looked like before Ilya scoped as
>>> many imports as he could:
>>>
>>> https://github.com/dlang/phobos/commit/3fcf723aa498b96de1653
>>> 61b5abb9d3450fdc069#diff-54cf8402b22024ae667d4048a5126f0e
>>>
>>> That was a mess, similar to opaque C/C++ code, 13 modules imported at
>>> module-scope were whittled down to 4.  You just made those more specific
>>> in a commit related to this DIP, by listing the actual symbols
>>> selectively imported from those four modules:
>>>
>>> https://github.com/dlang/phobos/commit/e064d5664f92c4b2f0866
>>> c08f6d0290ba66825ed#diff-54cf8402b22024ae667d4048a5126f0e
>>>
>>>
>>> If I'm looking at the template constraints for any particular function
>>> and see a couple symbols I don't recognize, I don't think it's a big
>>> deal to find the symbols in that list at the top.
>>>
>>
>> It actually is. Some symbols in e.g. std.range are used in constraints,
>> some others used in definitions; some in a UFCS manner. Some code runs
>> during compilation, with errors gagged (as in __traits(compiles)) or not
>> gagged. I don't know of an easy manual method to figure out who's who
>> (which makes Vladimir's idea of tooling it with brute force awesome and
>> scary at the same time).
>>
>> At any way, I don't see how I can use this tidbit to improve the DIP.
>> Anything I could add to it to make it more compelling?
>>
>> In other words, D already allows you to scope most imports.  I don't
>>> consider the dozen or two remaining symbols from templaint constraints
>>> and function arguments to provide much overhead.  Rather, I consider the
>>> weight of this additional syntax, ie the cognitive overhead from having
>>> to remember and parse more syntax in my head, to be worse than the
>>> remaining dependency reasoning problem you're trying to solve: the cost
>>> outweights the benefit.  Perhaps that's subjective and others may
>>> disagree.
>>>
>>
>> It is subjective and difficult to convert into action. Online review
>> among folks who know next to nothing about each other does have its
>> challenges. What happens here is I post a proposal for a problem that I
>> believe is important for large D projects. And I get back "eh, that's not
>> as important to me." At a traditional work place we'd know a lot about one
>> another's strengths, specialization areas, and design sensibilities. Here,
>> all I can do is look at your past PRs (that's why I emailed you asking for
>> your github handle; I figure it's @joakim-noah). This makes it difficult to
>> act on "I don't buy it" feedback.
>>
>> Now, there's also the question of purely technical benefits, like
>>> compilation speed or executable bloat.  I looked at the latter a little
>>> last summer, after Ilya had cleaned up a lot of the standard library:
>>>
>>> http://forum.dlang.org/thread/gmjqfjoemwtvgqrtd...@forum.dlang.org
>>>
>>> I found that commenting out a single scoped, selective import of
>>> "std.string: format" in std.utf led to a 5% decrease in executable size
>>> for "hello world."  This is a problem with how dmd compiles or appends
>>> these module dependencies and would presumably still be there after this
>>> DIP, as you would not remove the dependency.
>>>
>>
>> That might be a related but distinct issue. Can you reproduce that
>> experiment?
>>
>> I think scoped, selective imports have been great at hacking away at the
>>> module dependency graph, as you lay out.  It is not clear what technical