Re: Operator overloading or alternatives to expression templates

[Timon Gehr via Digitalmars-d]

On 09/15/2015 06:53 PM, Andrei Alexandrescu wrote:

On 09/14/2015 03:35 PM, Timon Gehr wrote:

On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:

On 09/13/2015 10:06 AM, Martin Nowak wrote:

...
- language regularization

   It's surprising to find these "arbitrary" language limitations.
   The non-predictability of what's possible has always been a huge
issue
for me with C++, i.e. you come up with an idea, spend 4 hours
implementing it only to find out that the small detail x isn't
feasible.


This is the case in all powerful languages.


That's an overgeneralization.


Aren't they all :o).
...


I guess we have now established that there are at least two of them. :P


Furthermore, having arbitrary designed-in irregularities is not
comparable to implementing a system whose emergent behavior is not
understood sufficiently well. (D is guilty of both, but the former is
much easier to fix than the latter.)


Martin Odersky told me there's a constant problem with Scala ...


Both C++ and Scala have accidentally Turing-complete type systems.


I think what I'm trying to say is "I'm trying to do this very advanced
thing and the language support is insufficient" is the kind of argument
that needs to be made and taken with caution.
...


This is certainly the case, and I believe the necessary care has been 
applied. The language support required isn't actually very advanced, nor 
are all usage scenarios. Arbitrary restrictions are usually design mistakes.

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/14/2015 03:35 PM, Timon Gehr wrote:

On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:

On 09/13/2015 10:06 AM, Martin Nowak wrote:

...
- language regularization

   It's surprising to find these "arbitrary" language limitations.
   The non-predictability of what's possible has always been a huge
issue
for me with C++, i.e. you come up with an idea, spend 4 hours
implementing it only to find out that the small detail x isn't feasible.


This is the case in all powerful languages.


That's an overgeneralization.


Aren't they all :o).


Furthermore, having arbitrary designed-in irregularities is not
comparable to implementing a system whose emergent behavior is not
understood sufficiently well. (D is guilty of both, but the former is
much easier to fix than the latter.)


Martin Odersky told me there's a constant problem with Scala ...


Both C++ and Scala have accidentally Turing-complete type systems.


I think what I'm trying to say is "I'm trying to do this very advanced 
thing and the language support is insufficient" is the kind of argument 
that needs to be made and taken with caution.



Andrei

Re: Operator overloading or alternatives to expression templates

[burjui via Digitalmars-d]

On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:

int opCmp(Foo rhs)
{
  return (id > rhs.id) - (id < rhs.id);
}


IMO, subtracting boolean values is bad code style, it's better to 
be explicit about your intention:

(id > rhs.id ? 1 : 0) - (id < rhs.id ? 1 : 0)

Re: Operator overloading or alternatives to expression templates

[Sebastiaan Koppe via Digitalmars-d]

On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
Does anyone have a different idea how to make a nice query 
language? db.get!Person.where!(p => p.age > 21 && p.name == 
"Peter")


In our last project we took the following approach:

`auto q = query.builder!Person.age!">"(20).name("Peter");`

In the query-builders there was a lot of ugly concatenating sql 
stuff. But there was just one guy writing that, everybody else 
was using the high-level functions.


It allows for a lot of checking, not only on types but also on 
logic (e.g. age!">"(20).age!"<"(10) was caught at runtime). Plus, 
very unittestable.


It worked so well that once the query-builders were written, all 
the constraints, sorting and ordering parameters could be passed 
straight from the http api down to the query-builders.


Still, you would have to write the query-builders for each Object 
you want query, although that is something you could do in a DSL.

Re: Operator overloading or alternatives to expression templates

[jmh530 via Digitalmars-d]

On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
AFAIK expression templates are the primary choice tom implement 
SIMD and

matrix libraries.
And I still have [this 
idea](http://dpaste.dzfl.pl/cd375ac594cf) of

implementing a nice query language for ORMs.


While expression templates are used in many matrix libraries, 
there has also been some move away from them. Blaze uses 
so-called smart expression templates because of what they view as 
performance limitations of expression templates. I think the 
general idea is to reduce the creation of temporaries and 
ordering the calculations so as to minimize the size of the 
problem (like in the multiplication A*B*c, doing A*(B*c) instead 
of (A*B)*c)

http://arxiv.org/pdf/1104.1729.pdf

At least the comparison operators are really limiting, e.g. 
it's not possible to efficiently implement logical indexing.


vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]



I loves the logical indexing.

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On 09/14/2015 03:47 PM, Sebastiaan Koppe wrote:
> 
> In our last project we took the following approach:
> 
> `auto q = query.builder!Person.age!">"(20).name("Peter");`

Interesting idea.

Re: Operator overloading or alternatives to expression templates

[Sebastiaan Koppe via Digitalmars-d]

On Monday, 14 September 2015 at 18:17:05 UTC, Adam D. Ruppe wrote:
On Monday, 14 September 2015 at 13:47:10 UTC, Sebastiaan Koppe 
wrote:

`auto q = query.builder!Person.age!">"(20).name("Peter");`


I confess that I'm not really paying attention to this thread, 
but I can't help but think plain old literal: `Person.age > 20 
&& Person.name = 'Peter'` is nicer. You can still CT check that 
by parsing the string if you want.


It is definitely nicer, but this is also a contrived use-case. 
What happens when you have a couple of joins, how would you write 
that? Or inner queries?


Suppose we wanted to fetch all users who didn't place an order in 
the last year, it would be as simple as calling 
`.lastOrderDate!"<"(lastYear)`. It would do the join with the 
order table on the appropriate column.


Granted, someone has to write `lastOrderDate()`.

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/13/2015 10:06 AM, Martin Nowak wrote:

On 09/13/2015 05:03 AM, Andrei Alexandrescu wrote:

Yah, understood. Problem here is the approach is bound to run into walls
at every few steps. Say you fix the comparisons to work. Then you have
operators such as LIKE that are necessary yet not representable in D. So
more tricks are in order. This is what I've seen with ET in C++ - an
endless collection of tricks to achieve modest outcomes at enormous costs.


But this is not an argument to rule out `opBinary!"<"`.


Agreed.


To summarize the arguments.

- logical indexing x[x < 20]

   e.g. opBinary!"<" returns a bit mask to select entries of a large vector


Good one. But niche, can be solved with a function, etc.


- faster comparison

   struct Foo
   {
   size_t id;
   int opCmp(Foo rhs)
   {
   if (id < rhs.id) return -1;
   if (id == rhs.id) return 0;
   else return 1;
   }
   bool opBinary(string s:"<")(Foo rhs)
   {
   return id < rhs.id;
   }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with opCmp.

   foos.sort!((a, b) => a.opBinary!"<"(b))(); // 104ms
   foos.sort!((a, b) => a < b)(); // 143ms


I think this is a good candidate for a peephole optimization.


- expression templates

   I'm well aware of the limitations, but still think it will work out
nicely for an ORM b/c there is precedence in other language, e.g. Rails'
(ActiveRecord) query syntax.


Fine, but let's not forget the ET experience in D will be a lot closer 
to that in C++ than that in Rails. And that doesn't bode well.



- language regularization

   It's surprising to find these "arbitrary" language limitations.
   The non-predictability of what's possible has always been a huge issue
for me with C++, i.e. you come up with an idea, spend 4 hours
implementing it only to find out that the small detail x isn't feasible.


This is the case in all powerful languages. Martin Odersky told me 
there's a constant problem with Scala having such a powerful type 
systems, the envelope of what can and cannot be done with it is really 
fuzzy and frustrates its power users.



Andrei

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/13/2015 01:16 PM, Daniel N wrote:

On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

  struct Foo
  {
  size_t id;
  int opCmp(Foo rhs)
  {
  if (id < rhs.id) return -1;
  if (id == rhs.id) return 0;
  else return 1;
  }
  bool opBinary(string s:"<")(Foo rhs)
  {
  return id < rhs.id;
  }
  }

  Sorting a million Foos w/ random ids is 37.5% slower with opCmp.



Could you try this?

int opCmp(Foo rhs)
{
   return (id > rhs.id) - (id < rhs.id);
}


Apparently that does well with gcc: 
http://stackoverflow.com/questions/10996418/efficient-integer-compare-function 
-- Andrei

Re: Operator overloading or alternatives to expression templates

[Adam D. Ruppe via Digitalmars-d]

On Monday, 14 September 2015 at 13:47:10 UTC, Sebastiaan Koppe 
wrote:

`auto q = query.builder!Person.age!">"(20).name("Peter");`


I confess that I'm not really paying attention to this thread, 
but I can't help but think plain old literal: `Person.age > 20 && 
Person.name = 'Peter'` is nicer. You can still CT check that by 
parsing the string if you want.

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/13/2015 03:06 PM, Martin Nowak wrote:

On 09/13/2015 07:16 PM, Daniel N wrote:


Could you try this?

int opCmp(Foo rhs)
{
   return (id > rhs.id) - (id < rhs.id);
}


That's not the point, opCmp requires twice as many comparisons as needed
for <. If they are more expansive, e.g. string comparison, your trick
won't work.


Well for string comparisons, the added cost (if not optimized away) is 
constant. Overall it seems to me that D is to be appreciated for 
requiring only one operator for all ordering comparisons (it causes a 
lot of noise in C++). Reducing comparisons is the job of the optimizer. 
Expression templates are a possible loss, but of a questionable trick. 
-- Andrei

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/14/2015 07:32 AM, burjui wrote:

On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:

int opCmp(Foo rhs)
{
  return (id > rhs.id) - (id < rhs.id);
}


IMO, subtracting boolean values is bad code style


I love it! -- Andrei

Re: Operator overloading or alternatives to expression templates

[Timon Gehr via Digitalmars-d]

On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:

On 09/13/2015 10:06 AM, Martin Nowak wrote:

...
- language regularization

   It's surprising to find these "arbitrary" language limitations.
   The non-predictability of what's possible has always been a huge issue
for me with C++, i.e. you come up with an idea, spend 4 hours
implementing it only to find out that the small detail x isn't feasible.


This is the case in all powerful languages.


That's an overgeneralization.

Furthermore, having arbitrary designed-in irregularities is not 
comparable to implementing a system whose emergent behavior is not 
understood sufficiently well. (D is guilty of both, but the former is 
much easier to fix than the latter.)



Martin Odersky told me there's a constant problem with Scala ...


Both C++ and Scala have accidentally Turing-complete type systems.

Re: Operator overloading or alternatives to expression templates

[Ola Fosheim Grøstad via Digitalmars-d]

On Monday, 14 September 2015 at 19:35:39 UTC, Timon Gehr wrote:
Furthermore, having arbitrary designed-in irregularities is not 
comparable to implementing a system whose emergent behavior is 
not understood sufficiently well. (D is guilty of both, but the 
former is much easier to fix than the latter.)


Well, D1 was a restrictive design that made C++ style programming 
simpler and less error prone (sans compiler bugs). D2 is a 
flexible and complicated package and thereby those restrictions 
cause disharmony. It is basically too incompatible design 
philosophies that are brought together.


Like in D1 it would make sense to say "the language provide the 
essentials, don't design your own pointer types", whereas in D2 
it makes sense to say "all features should be library based". I'm 
a bit miffed that it is difficult to built proper smart pointer 
types in D2.


Both C++ and Scala have accidentally Turing-complete type 
systems.


Yes, not sure how smart that is in terms of maintainable code. 
There is a reaction against overdone type systems that sometimes 
can be more of burden as programs age/grow (e.g. C++/Boost), so 
now we see gradual typing, deliberately unsound type-systems etc.

Re: Operator overloading or alternatives to expression templates

[Timon Gehr via Digitalmars-d]

On 09/14/2015 08:18 PM, Andrei Alexandrescu wrote:

On 09/13/2015 01:16 PM, Daniel N wrote:

On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

  struct Foo
  {
  size_t id;
  int opCmp(Foo rhs)
  {
  if (id < rhs.id) return -1;
  if (id == rhs.id) return 0;
  else return 1;
  }
  bool opBinary(string s:"<")(Foo rhs)
  {
  return id < rhs.id;
  }
  }

  Sorting a million Foos w/ random ids is 37.5% slower with opCmp.



Could you try this?

int opCmp(Foo rhs)
{
   return (id > rhs.id) - (id < rhs.id);
}


Apparently that does well with gcc:
http://stackoverflow.com/questions/10996418/efficient-integer-compare-function
-- Andrei



The first version was actually slightly faster when I tested it with 
std::sort. (i.e. if only "<" is used). I guess it is optimized to a 
simple integer comparison after inlining. In any case, any performance 
differences for such simple equivalent functions are random quirks of 
the respective back-ends.

Re: Operator overloading or alternatives to expression templates

[deadalnix via Digitalmars-d]

On Monday, 14 September 2015 at 18:24:00 UTC, Andrei Alexandrescu 
wrote:

On 09/14/2015 07:32 AM, burjui wrote:

On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:

int opCmp(Foo rhs)
{
  return (id > rhs.id) - (id < rhs.id);
}


IMO, subtracting boolean values is bad code style


I love it! -- Andrei


Beside loving, it, this has good ILP, no branches, and have nice 
codegen on most CPUs.


I was surprised to find that style in jemalloc, and compared to 
various alternatives. The only way I was able to get better is by 
ignoring (and by extension getting invalid results) overflow.

Re: Operator overloading or alternatives to expression templates

[Sönke Ludwig via Digitalmars-d]

Am 11.09.2015 um 21:40 schrieb Martin Nowak:

I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

(...)


The missing support for overloading the individual relational operators 
is something that has often turned out very limiting to me in the past, 
too. The opCmp approach is good as a default, but explicitly disallowing 
implementation of the individual operators seems arbitrary.



Now expression templates make an awful DSL when being used to create a
HTML formatter, but when the original semantic of the operators is
preserved they are really powerful b/c the compiler already handles
typechecking, operator precedence, and captures variables.

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


I had played around with some ideas for a similar project, but didn't 
find a really satisfying solution:

https://github.com/rejectedsoftware/dotter/blob/11ec72325e76c3329a58545526940c1df5328a2d/source/dotter/orm.d#L320

Re: Operator overloading or alternatives to expression templates

[Ola Fosheim Grøstad via Digitalmars-d]

On Sunday, 13 September 2015 at 03:03:44 UTC, Andrei Alexandrescu 
wrote:
Yah, understood. Problem here is the approach is bound to run 
into walls at every few steps. Say you fix the comparisons to 
work. Then you have operators such as LIKE that are necessary


like(Person.Name,"peter%")

Person.Name == pattern("peter%")

yet not representable in D. So more tricks are in order. This 
is what I've seen with ET in C++ - an endless collection of 
tricks to achieve modest outcomes at enormous costs.


FWIW, Guido wrote a similar query language to what Martin 
proposes for Google App Engine called NDB:


https://cloud.google.com/appengine/docs/python/ndb/queries

They already had a string query language called GQL:

https://cloud.google.com/appengine/docs/python/datastore/gqlreference

I never use GQL, because the NDB solution is superior in every 
way.


GQL is only for the REPL.

2. If you want to embed real SQL into D, use string-based 
DSLs.


Strings don't capture context, aren't typechecked, and require 
a complex

CTFE parser.

db.get!Person.where!"age > 21 & name = ?"(name);


Understood as well. I figure from the example you have binding 
in mind, which is good. At some point you need to bite the 
bullet and write a parser for the relational query language you 
want there.


No. D should stop abusing strings. Strings should never contain 
references to type names. It makes it impossible to transpile D 
in the general case.


Walter is unlikely to change his mind on opCmp, because he has 
put a lot of emphasis on that restriction since D1, but I think 
he overreacted on some C++ libraries being less than user 
friendly.


The only thing you achieve is that people will create even uglier 
hacks to get what they want, like using a thread local global AST 
root pointer and build the AST as hidden side effects at runtime 
instead.

Re: Operator overloading or alternatives to expression templates

[deadalnix via Digitalmars-d]

On Sunday, 13 September 2015 at 03:03:44 UTC, Andrei Alexandrescu 
wrote:

On 09/12/2015 04:08 PM, Martin Nowak wrote:
On Friday, 11 September 2015 at 23:47:42 UTC, Andrei 
Alexandrescu wrote:

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input 
range of
Person. Presumably introspection is being used to bind fields 
in the
query such as "age" and "name" to static field names in 
struct Person.

Then good old std.algorithm.filter takes care of the rest.


I'm instantiating the lambda with a fake p to capture the 
expression so
I can translate it to whatever SQL, MongoDB, columnar db is 
used.


Yah, understood. Problem here is the approach is bound to run 
into walls at every few steps. Say you fix the comparisons to 
work. Then you have operators such as LIKE that are necessary 
yet not representable in D. So more tricks are in order. This 
is what I've seen with ET in C++ - an endless collection of 
tricks to achieve modest outcomes at enormous costs.




Once gain, that says more about C++ than anything else. C# have 
such mechnism and users seems very happy with it.


2. If you want to embed real SQL into D, use string-based 
DSLs.


Strings don't capture context, aren't typechecked, and require 
a complex

CTFE parser.

db.get!Person.where!"age > 21 & name = ?"(name);


Understood as well. I figure from the example you have binding 
in mind, which is good. At some point you need to bite the 
bullet and write a parser for the relational query language you 
want there.




That sounds like a bigger problem than whatever you mentioned 
before.

Re: Operator overloading or alternatives to expression templates

[Dicebot via Digitalmars-d]

I am pretty sure I'd prefer hygienic string DSL (with explicit 
alias list for used context symbols, like in Diet) over LINQ-like 
magic.

Re: Operator overloading or alternatives to expression templates

[Jack Stouffer via Digitalmars-d]

On Saturday, 12 September 2015 at 22:38:53 UTC, Bahman Movaqar 
wrote:
Django's approach is, IMO,  the cleverest and least magical one 
while keeping it expressive and efficient:


Person.objects.filter(age__gt=21, name__eq='peter')


As the main examples in this thread are for ORMs, I think the 
best case for more powerful operator overloading is the very 
popular Python library SQLAlchmey. SQLAlchmey is the reason that 
any project of mine that requires a database is going to be in 
Python.


Here is a real code sample from one of my projects from my day 
job:


shipped = session.query(
func.sum(
InvoiceHistoryDetail.QuantityShipped * 
InvoiceHistoryDetail.UnitPrice

).label("sum")
).join(
InvoiceHistoryHeader,
InvoiceHistoryHeader.InvoiceNo == 
InvoiceHistoryDetail.InvoiceNo

).join(
Customer,
and_(
InvoiceHistoryHeader.ARDivisionNo == 
Customer.ARDivisionNo,

InvoiceHistoryHeader.CustomerNo == Customer.CustomerNo
)
).filter(
InvoiceHistoryHeader.ShipDate >= 
fiscal_month.PeriodStartingDate,
InvoiceHistoryHeader.ShipDate == datetime.date.today() - 
datetime.timedelta(days=1),

InvoiceHistoryHeader.SalesOrderNo != None,
Customer.UDF_PARTICIPATION.in_(participation)
).one()

Note how complex operator overloading is used to make very 
readable JOIN and WHERE clauses. One other cool thing to note is 
the func function, which is a generator function that allows 
expressing any SQL function in Python by translating the name 
after the dot to the function name is the final SQL. So 
func.whatever(model.field) would become WHATEVER(`model.field`) 
in the SQL.


I know that this effect is much harder to create in a explicitly 
and strongly typed language, but I just wanted to show a real 
world example of how these could be used to great effect.

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On 09/13/2015 03:15 PM, Jack Stouffer wrote:
> 
> I know that this effect is much harder to create in a explicitly and
> strongly typed language, but I just wanted to show a real world example
> of how these could be used to great effect.

But it is doable in D, and even better it's possible to optimize the
queries b/c we can know in advance what fields are used.
http://dpaste.dzfl.pl/cd375ac594cf

Re: Operator overloading or alternatives to expression templates

[Jack Stouffer via Digitalmars-d]

On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

- logical indexing x[x < 20]

  e.g. opBinary!"<" returns a bit mask to select entries of a 
large vector


Oh yes please! This is one of the more powerful features of numpy.

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On 09/13/2015 05:03 AM, Andrei Alexandrescu wrote:
> Yah, understood. Problem here is the approach is bound to run into walls
> at every few steps. Say you fix the comparisons to work. Then you have
> operators such as LIKE that are necessary yet not representable in D. So
> more tricks are in order. This is what I've seen with ET in C++ - an
> endless collection of tricks to achieve modest outcomes at enormous costs.

But this is not an argument to rule out `opBinary!"<"`.

To summarize the arguments.

- logical indexing x[x < 20]

  e.g. opBinary!"<" returns a bit mask to select entries of a large vector

- faster comparison

  struct Foo
  {
  size_t id;
  int opCmp(Foo rhs)
  {
  if (id < rhs.id) return -1;
  if (id == rhs.id) return 0;
  else return 1;
  }
  bool opBinary(string s:"<")(Foo rhs)
  {
  return id < rhs.id;
  }
  }

  Sorting a million Foos w/ random ids is 37.5% slower with opCmp.

  foos.sort!((a, b) => a.opBinary!"<"(b))(); // 104ms
  foos.sort!((a, b) => a < b)(); // 143ms

- expression templates

  I'm well aware of the limitations, but still think it will work out
nicely for an ORM b/c there is precedence in other language, e.g. Rails'
(ActiveRecord) query syntax.

- language regularization

  It's surprising to find these "arbitrary" language limitations.
  The non-predictability of what's possible has always been a huge issue
for me with C++, i.e. you come up with an idea, spend 4 hours
implementing it only to find out that the small detail x isn't feasible.

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On 09/13/2015 11:00 AM, Sönke Ludwig wrote:
> I had played around with some ideas for a similar project, but didn't
> find a really satisfying solution:
> https://github.com/rejectedsoftware/dotter/blob/11ec72325e76c3329a58545526940c1df5328a2d/source/dotter/orm.d#L320

Yeah, that doesn't look too nice.

I think db.get!Author.where!(a => a.books.count > 10) is very powerful
b/c you can easily use relations for querying.

struct Author
{
  @hasMany // relation in SQL or nested in MongoDB
  Book[] books;
}

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On 09/13/2015 07:16 PM, Daniel N wrote:
> 
> Could you try this?
> 
> int opCmp(Foo rhs)
> {
>   return (id > rhs.id) - (id < rhs.id);
> }

That's not the point, opCmp requires twice as many comparisons as needed
for <. If they are more expansive, e.g. string comparison, your trick
won't work.

Re: Operator overloading or alternatives to expression templates

[Daniel N via Digitalmars-d]

On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

  struct Foo
  {
  size_t id;
  int opCmp(Foo rhs)
  {
  if (id < rhs.id) return -1;
  if (id == rhs.id) return 0;
  else return 1;
  }
  bool opBinary(string s:"<")(Foo rhs)
  {
  return id < rhs.id;
  }
  }

  Sorting a million Foos w/ random ids is 37.5% slower with 
opCmp.




Could you try this?

int opCmp(Foo rhs)
{
  return (id > rhs.id) - (id < rhs.id);
}

Re: Operator overloading or alternatives to expression templates

[Bahman Movaqar via Digitalmars-d]

On Fri, 11 Sep 2015 21:40:58 +0200, Martin Nowak 
 wrote:

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


Django's approach is, IMO,  the cleverest and least magical one while 
keeping it expressive and efficient:


Person.objects.filter(age__gt=21, name__eq='peter')

The input of `filter` is keyword argument style. So, filter can 
access the dynamic argument names, in addition to their values. This 
makes it easy to understand user's intent without the need to 
overloaded operators or black magic (like repeated reflections 
Hibernate uses) and the syntax stays clean from user's perspective as 
well.


--
--
Bahman Movaqar

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu 
wrote:

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input 
range of Person. Presumably introspection is being used to bind 
fields in the query such as "age" and "name" to static field 
names in struct Person. Then good old std.algorithm.filter 
takes care of the rest.


I'm instantiating the lambda with a fake p to capture the 
expression so I can translate it to whatever SQL, MongoDB, 
columnar db is used.



2. If you want to embed real SQL into D, use string-based DSLs.


Strings don't capture context, aren't typechecked, and require a 
complex CTFE parser.


db.get!Person.where!"age > 21 & name = ?"(name);

Re: Operator overloading or alternatives to expression templates

[Dmitry Olshansky via Digitalmars-d]

On 12-Sep-2015 23:08, Martin Nowak wrote:

On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu wrote:

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input range of
Person. Presumably introspection is being used to bind fields in the
query such as "age" and "name" to static field names in struct Person.
Then good old std.algorithm.filter takes care of the rest.


I'm instantiating the lambda with a fake p to capture the expression so
I can translate it to whatever SQL, MongoDB, columnar db is used.


2. If you want to embed real SQL into D, use string-based DSLs.


Strings don't capture context, aren't typechecked, and require a complex
CTFE parser.

db.get!Person.where!"age > 21 & name = ?"(name);




What if we add generic string interpolation a-la:

s"$age > 21 && name = ${someobj.field}"

would translate to:
AliasSeq!("",age," > 21 && name = ", someobj.field, "");

Thoughts? I think it was proposed before by Timothy Cour.

--
Dmitry Olshansky

Re: Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

On Friday, 11 September 2015 at 23:19:54 UTC, Timon Gehr wrote:
Does anyone have a different idea how to make a nice query 
language?

db.get!Person.where!(p => p.age > 21 && p.name == "Peter")



You could give up on operator syntax.


That's what I did in the prototype, p.age.gt(21), but it's 
somewhat ugly.

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/12/2015 04:08 PM, Martin Nowak wrote:

On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu wrote:

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input range of
Person. Presumably introspection is being used to bind fields in the
query such as "age" and "name" to static field names in struct Person.
Then good old std.algorithm.filter takes care of the rest.


I'm instantiating the lambda with a fake p to capture the expression so
I can translate it to whatever SQL, MongoDB, columnar db is used.


Yah, understood. Problem here is the approach is bound to run into walls 
at every few steps. Say you fix the comparisons to work. Then you have 
operators such as LIKE that are necessary yet not representable in D. So 
more tricks are in order. This is what I've seen with ET in C++ - an 
endless collection of tricks to achieve modest outcomes at enormous costs.



2. If you want to embed real SQL into D, use string-based DSLs.


Strings don't capture context, aren't typechecked, and require a complex
CTFE parser.

db.get!Person.where!"age > 21 & name = ?"(name);


Understood as well. I figure from the example you have binding in mind, 
which is good. At some point you need to bite the bullet and write a 
parser for the relational query language you want there.



Andrei

Re: Operator overloading or alternatives to expression templates

[Jacob Carlborg via Digitalmars-d]

On 2015-09-12 01:47, Andrei Alexandrescu wrote:


There's two canonical ways to do that.

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input range of
Person. Presumably introspection is being used to bind fields in the
query such as "age" and "name" to static field names in struct Person.
Then good old std.algorithm.filter takes care of the rest.


A database is really good at what it does. Therefore one wants the 
filter/where statement to be executed by the database, not the 
application code.


It would also be quite wasteful to have to create an instance of Person 
to make the comparison then realizing it doesn't fulfill the condition 
and throw it away.


I'm not sure how database protocols work but one definitely don't want 
to query the database for each row or return all the rows from the database.



2. If you want to embed real SQL into D, use string-based DSLs.


The point is to _not_ use raw SQL.

--
/Jacob Carlborg

Re: Operator overloading or alternatives to expression templates

[Jacob Carlborg via Digitalmars-d]

On 2015-09-11 21:40, Martin Nowak wrote:

I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD and
matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.


I would really like this.


The short circuit operators && and || have a special semantic and can't
be easily, but there is & and | so it's not really required.


Not sure if this would be a problem in D but there's a library in Ruby 
that does exactly this. Due to the operator precedence you end up 
needing to wrap everything in parentheses:


Person.where{ |p| (p.age > 21) & (p.name == "Peter") | (p.name == "Foo") }


Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


AST macros ;)

--
/Jacob Carlborg

Re: Operator overloading or alternatives to expression templates

[via Digitalmars-d]

On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
Does anyone have a different idea how to make a nice query 
language? db.get!Person.where!(p => p.age > 21 && p.name == 
"Peter")


Do it at runtime?

Operator overloading or alternatives to expression templates

[Martin Nowak via Digitalmars-d]

I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD and
matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.

D currently doesn't allow to override some operators like < <= > >= &&
|| !=.

At least the comparison operators are really limiting, e.g. it's not
possible to efficiently implement logical indexing.

vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

Also opCmp is less efficient to implement than opBinary!"<" for many
types. Generally any good implementation of an algorithm should only
require a less operator, not a full ordering opCmp.

The short circuit operators && and || have a special semantic and can't
be easily, but there is & and | so it's not really required.

Now expression templates make an awful DSL when being used to create a
HTML formatter, but when the original semantic of the operators is
preserved they are really powerful b/c the compiler already handles
typechecking, operator precedence, and captures variables.

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")

Re: Operator overloading or alternatives to expression templates

[Jonathan M Davis via Digitalmars-d]

On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
Does anyone have a different idea how to make a nice query 
language? db.get!Person.where!(p => p.age > 21 && p.name == 
"Peter")


Don't the suggestions for DSLs generally revolve around using 
string mixins to use whatever syntax you want?


- Jonathan M Davis

Re: Operator overloading or alternatives to expression templates

[Timon Gehr via Digitalmars-d]

On 09/11/2015 09:40 PM, Martin Nowak wrote:

I find the reasons for turining down my ER a bit moot.
...


+1.


[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD and
matrix libraries.


OTOH, they are a hack.


And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.

D currently doesn't allow to override some operators like < <= > >= &&
|| !=.

At least the comparison operators are really limiting, e.g. it's not
possible to efficiently implement logical indexing.

vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

Also opCmp is less efficient to implement than opBinary!"<" for many
types. Generally any good implementation of an algorithm should only
require a less operator, not a full ordering opCmp.
...


The rewrite a < b => a.opCmp(b)<0 is usually wasteful, but both methods 
can be the most efficient choice depending on the application. Two calls 
to opBinary!"<" will usually not be more efficient than one call to 
"opCmp".


I.e. the precedence should be the other way: try a.opBinary!"<"(b) first 
and then fall back to a.opCmp(b)<0. (Maybe it would also make sense to 
automatically provide opCmp if missing when opBinary!"<" has been 
implemented, such that generic code can use opCmp for all comparable 
types. Built-in comparable types should also provide opCmp.)



The short circuit operators && and || have a special semantic and can't
be easily, but there is & and | so it's not really required.
...


There's the 'lazy' keyword.


...

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")



You could give up on operator syntax.

Re: Operator overloading or alternatives to expression templates

[H. S. Teoh via Digitalmars-d]

On Fri, Sep 11, 2015 at 07:47:42PM -0400, Andrei Alexandrescu via Digitalmars-d 
wrote:
> On 09/11/2015 03:40 PM, Martin Nowak wrote:
> >I find the reasons for turining down my ER a bit moot.
> >
> >[Issue 14593 – operator overloading can't be used with expression
> >templates](https://issues.dlang.org/show_bug.cgi?id=14593)
> >
> >AFAIK expression templates are the primary choice tom implement SIMD
> >and matrix libraries.
> >And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
> >implementing a nice query language for ORMs.
> 
> Expression templates are interesting, but from experience with them in
> C++ they're more trouble than they're worth. They haven't made much
> inroads in C++ outside exotic libraries because (a) they have odd and
> random limitations and corner cases and (b) they have really byzantine
> failure modes. Just look at any non-toy-example C++ use of ETs - it's
> completely bizarre.

A number of years ago I wrote a computational application using Blitz++,
an ET-based matrix / multidimensional array C++ library, and I have to
say that the experience was mostly pleasant.  However, it did push the
limits of operator overloading abuse, such as:

Array<2,int> matrix = Array<2,int>(4, 4);
matrix = 1, 0, 0, 0,
 0, 1, 0, 0,
 0, 0, 1, 0,
 0, 0, 0, 1;

It also had unexpected reference/copying semantics (operator=() copies
by reference, but `=` in a variable declarationi copies by value), as
well as the occasional wrinkle when assigning the result of an
expression template to a variable (sometimes you have to explicitly call
a function to turn it into something storable in a variable).

The error messages however, were completely inscrutable, just as you
said. They generally begin anywhere from 15-16 lines of compiler output
per error, and only grows from there. Needless to say, I did not have
the patience (nor persistence!) to decipher those error messages; most
of my efforts lay in copying textbook examples from the documentation
and modifying them piece by piece, checking their compilability at every
step, until they matched what I ultimately wanted. Writing anything
complex directly was an invitation to be faced with an incomprehensible
screen-filling error message (often more than one), and endless hours of
randomly modifying random bits of syntax in hopes that the error will
somehow, magically, go away.


> I'd say if a language wants to support expression templates properly,
> there's a lot of careful design to get into it, way beyond
> intercepting operators. Like e.g. C#/LINQ. I don't think D is prepared
> for that.

Pluggable syntax modules seem like an attractive idea, though. But
probably outside the scope of D2 at present.


[...]
> >Does anyone have a different idea how to make a nice query language?
> >db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
> 
> There's two canonical ways to do that.
> 
> 1. Use lambdas, which seem to already do what you want:
> 
> db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")
> 
> The way this'd go, the db.get!Person() call returns an input range of
> Person. Presumably introspection is being used to bind fields in the
> query such as "age" and "name" to static field names in struct Person.
> Then good old std.algorithm.filter takes care of the rest.
> 
> 2. If you want to embed real SQL into D, use string-based DSLs.
[...]

Yeah, string-based DSLs make more sense in D: avoid complicating the
core language, thorny issues surrounding operator overloading (and the
abuse thereof), and also allow fully-free syntax of your choice.  With
CTFE, you can parse just about any DSL with any syntax at compile-time,
thus incur none of the performance issues that runtime string-based DSLs
would have.


T

-- 
The only difference between male factor and malefactor is just a little 
emptiness inside.

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/11/2015 08:03 PM, H. S. Teoh via Digitalmars-d wrote:

Needless to say, I did not have
the patience (nor persistence!) to decipher those error messages; most
of my efforts lay in copying textbook examples from the documentation
and modifying them piece by piece, checking their compilability at every
step, until they matched what I ultimately wanted. Writing anything
complex directly was an invitation to be faced with an incomprehensible
screen-filling error message (often more than one), and endless hours of
randomly modifying random bits of syntax in hopes that the error will
somehow, magically, go away.


All the more ironic considering Blitz++ was intended for scientists who 
were not supposed to be programmer experts. -- Andrei

Re: Operator overloading or alternatives to expression templates

[deadalnix via Digitalmars-d]

On Saturday, 12 September 2015 at 00:27:45 UTC, Andrei 
Alexandrescu wrote:

On 09/11/2015 08:03 PM, H. S. Teoh via Digitalmars-d wrote:

Needless to say, I did not have
the patience (nor persistence!) to decipher those error 
messages; most
of my efforts lay in copying textbook examples from the 
documentation
and modifying them piece by piece, checking their 
compilability at every
step, until they matched what I ultimately wanted. Writing 
anything
complex directly was an invitation to be faced with an 
incomprehensible
screen-filling error message (often more than one), and 
endless hours of
randomly modifying random bits of syntax in hopes that the 
error will

somehow, magically, go away.


All the more ironic considering Blitz++ was intended for 
scientists who were not supposed to be programmer experts. -- 
Andrei


I do think it says more about C++ than it does about expression 
templates.

Re: Operator overloading or alternatives to expression templates

[Andrei Alexandrescu via Digitalmars-d]

On 09/11/2015 03:40 PM, Martin Nowak wrote:

I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD and
matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.


Expression templates are interesting, but from experience with them in 
C++ they're more trouble than they're worth. They haven't made much 
inroads in C++ outside exotic libraries because (a) they have odd and 
random limitations and corner cases and (b) they have really byzantine 
failure modes. Just look at any non-toy-example C++ use of ETs - it's 
completely bizarre.


I'd say if a language wants to support expression templates properly, 
there's a lot of careful design to get into it, way beyond intercepting 
operators. Like e.g. C#/LINQ. I don't think D is prepared for that.



D currently doesn't allow to override some operators like < <= > >= &&
|| !=.

At least the comparison operators are really limiting, e.g. it's not
possible to efficiently implement logical indexing.

vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

Also opCmp is less efficient to implement than opBinary!"<" for many
types. Generally any good implementation of an algorithm should only
require a less operator, not a full ordering opCmp.

The short circuit operators && and || have a special semantic and can't
be easily, but there is & and | so it's not really required.

Now expression templates make an awful DSL when being used to create a
HTML formatter, but when the original semantic of the operators is
preserved they are really powerful b/c the compiler already handles
typechecking, operator precedence, and captures variables.

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


There's two canonical ways to do that.

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input range of 
Person. Presumably introspection is being used to bind fields in the 
query such as "age" and "name" to static field names in struct Person. 
Then good old std.algorithm.filter takes care of the rest.


2. If you want to embed real SQL into D, use string-based DSLs.


Andrei