CBOR with Mir Ion
Mir-Ion 2.3.0 [got](https://github.com/libmir/mir-ion/blob/master/source/mir/ser/cbor.d) CBOR serialzation. It is the fastest Mir serialization target. I have no plans for CBOR deserialization. It can be implemented using the MsgPack deserialization target as a draft.
On Mir parts migration to DRuntime
Hi Walter and Dlang Community, Mir libraries or their parts can be moved to Phobos/Druntime under BSL-1.0. I won't be able to help with mir migration except for general review and coordination with copyright holders. I use D for work but am no longer an active open-source contributor. - [mir-ion](http://mir-ion.libmir.org/) - JSON, Amazon Ion, YAML, CSV, and Msgpack serialization library - [mir-algorithm](http://mir-algorithm.libmir.org) - tensors, prices number printing and parsing, interpolation, and algorithms - [mir-core](http://mir-core.libmir.org) - a lot of cool stuff, including algebraic types - [mir-random](http://mir-random.libmir.org) - random engine and algorithms - [mir-cpuid](http://mir-cpuid.libmir.org) - [mir-optim](http://mir-optim.libmir.org) - maybe some other packages as well Requirements - Full backward compatibility up to the namespace until 2025-12-31. - Port as it is with minimal changes, including all Mir dependencies up to the module level. - Port to a separate namespace, something like core.mir/std.mir or core.ext/std.ext. The namespace has to be short. - Mir libraries must be updated in lockstep to depend on Phobos: if a Mir module is ported to Phobos, it must be removed from Mir. - All ported public code documentation must be presented on the dlang.org - If a non-mir module, for example, `std.format`, is reworked with mir, it has to add the credentials. - All credentials have to be presented (authors, and note that it was ported from Mir) - We would need approval from other copyright holders for some modules. Generally, we have to get approval from Symmetry Investments. - Mir code in Phobos has to be frozen from any major rework until 2025-12-31. We may add some exceptions. But the general rule is to freeze the code. Walter, please let me know what you think. ki9ilia on G post Kind regards, Ilya
Re: text based file formats
On Tuesday, 20 December 2022 at 19:46:36 UTC, John Colvin wrote: On Tuesday, 20 December 2022 at 00:40:07 UTC, H. S. Teoh wrote: On Mon, Dec 19, 2022 at 04:16:57PM -0800, Walter Bright via Digitalmars-d-announce wrote: On 12/19/2022 4:35 AM, Adam D Ruppe wrote: > On Monday, 19 December 2022 at 09:55:47 UTC, Walter Bright > wrote: > > Curious why CSV isn't in the list. > > Maybe std.csv is already good enough? LOL, learn something every day! I've even written my own, but it isn't very good. There's also my little experimental csv parser that was designed to be as fast as possible: https://github.com/quickfur/fastcsv However, it can only handle input that fits in memory (using std.mmfile is one possible workaround), has a static limit on field sizes, and does not do validation. T We use this at work with some light tweaks, it’s done a lot work It has already been replaced with [mir.csv](https://github.com/libmir/mir-ion/blob/master/source/mir/csv.d). Mir is faster, SIMD accelerated, and supports numbers and timestamp recognition.
Re: Beta 2.099.0
On Tuesday, 15 February 2022 at 13:06:47 UTC, Martin Nowak wrote: As usual please report any bugs at https://issues.dlang.org The compiler has some mangling or attributes deduction bug when compiling or linking tagged self-referencing algebraic. The GitHub job: https://github.com/libmir/mir-ion/runs/5229969072?check_suite_focus=true I report the bug here because issues.dlang.org doesn't like my email: The e-mail address you entered (@@@my GMAIL@@@) didn't pass our syntax checking for a legal email address. A legal address must contain exactly one '@', and at least one '.' after the @. Currently, registering using Gmail addresses is not allowed due to spam. It also must not contain any illegal characters.
Re: Result handing in D: some|none
On Sunday, 30 January 2022 at 10:58:33 UTC, 9il wrote: [mir.algebraic](http://mir-core.libmir.org/mir_algebraic.html) I like `suit` a lot. It helps to define very clean visitors. ```d import std.traits: isDynamicArray, Unqual; import std.meta: templateNot; alias V = Variant!(long, int, string, long[], int[]); alias autoGetElementType = match!( (string s) => "string", // we override the suit handler below for string suit!(isDynamicArray, a => Unqual!(typeof(a[0])).stringof), suit!(templateNot!isDynamicArray, a => Unqual!(typeof(a)).stringof), ); assert(autoGetElementType(V(string.init)) == "string"); assert(autoGetElementType(V((long[]).init)) == "long"); assert(autoGetElementType(V((int[]).init)) == "int"); assert(autoGetElementType(V(long.init)) == "long"); assert(autoGetElementType(V(int.init)) == "int"); ```
Result handing in D: some|none
[mir.algebraic](http://mir-core.libmir.org/mir_algebraic.html)
additions:
- `some`, `none`, and `suit` second level algebraic visitor
handlers
- `@reflectErr` UDA
- `Err` auto expanded wrapper when `@reflectErr` can't be used
- Required utility API
```d
import mir.algebraic;
import mir.conv: to;
alias orElse(alias fun) = visit!(some!"a", none!fun);
alias convertErrToString = orElse!(to!string);
// can be any other type including integer enums
@reflectErr
static struct ErrorInfo {
string msg;
auto toString() const { return msg; }
}
alias V = Variant!(Err!string, ErrorInfo, long, double);
alias R = typeof(convertErrToString(V.init));
static assert(is(R == Variant!(string, long, double)),
R.stringof);
assert(convertErrToString(V(1)) == 1);
assert(convertErrToString(V(1.0)) == 1.0);
assert(convertErrToString(ErrorInfo("b")) == "b");
assert(convertErrToString("Ш".err) == "Ш");
```
Re: mir.complex
On Friday, 20 August 2021 at 16:55:57 UTC, David Gileadi wrote: On 8/20/21 9:54 AM, Dennis wrote: On Friday, 20 August 2021 at 16:44:53 UTC, 9il wrote: Builtin complex numbers have been replaced with mir.complex in the following packages: Out of curiosity, how did std.complex fall short? Maybe it was too complex? Sorry, I'll see myself out. Yes, it was hard to make std.complex works.
Re: mir.complex
On Friday, 20 August 2021 at 16:54:07 UTC, Dennis wrote: On Friday, 20 August 2021 at 16:44:53 UTC, 9il wrote: Builtin complex numbers have been replaced with mir.complex in the following packages: Out of curiosity, how did std.complex fall short? There are few reasons. The main one is that linking bugs caused by a compiler mangling/generation bugs makes std.complex unusable for us. Initially I have updated all packages with std.complex, however it failed to link in a private project.
mir.complex
mir.complex [1] has been added to mir-core v1.1.80 Builtin complex numbers have been replaced with mir.complex in the following packages: - cblas, - lapack - mir-blas - mir-lapack - lubeck [1] http://mir-core.libmir.org/mir_complex.html MRs are welcome. Kind regards, Ilya
Re: Mir Ion and Asdf benchmakrs
On Sunday, 16 May 2021 at 10:28:24 UTC, Tobias Pankrath wrote: On Sunday, 16 May 2021 at 09:50:21 UTC, 9il wrote: mir-ion and Asdf JSON libraries have been added to the Kostya/benchmarks. https://github.com/kostya/benchmarks#json If we exclude parsers with inaccurate number parsing then the top will be 1. C++, simdjson 2. Rust, Serde 3. Dlang, Mir Amazon's Ion DOM 4. Dlang, Mir Asdf DOM 5. C++ RapidJSON (Precise) Mir Ion has been inspired by simdjson and Amazon's Ion binary format, which is used as DOM. Thus the mir-ion DOM for the 112 MiB file costs 16 MiB comparing with 176 MiB DOM in simdjson. Kind regards, Ilya Great work! What makes simdjson faster? simdjson works with padded single memory chunk and use simple (but large) DOM format. Mir uses buffered input by chunks of 4KB and compresses data to the Ion format on the fly. Ion requires significantly less space but it is more CPU time expensive.
Mir Ion and Asdf benchmakrs
mir-ion and Asdf JSON libraries have been added to the Kostya/benchmarks. https://github.com/kostya/benchmarks#json If we exclude parsers with inaccurate number parsing then the top will be 1. C++, simdjson 2. Rust, Serde 3. Dlang, Mir Amazon's Ion DOM 4. Dlang, Mir Asdf DOM 5. C++ RapidJSON (Precise) Mir Ion has been inspired by simdjson and Amazon's Ion binary format, which is used as DOM. Thus the mir-ion DOM for the 112 MiB file costs 16 MiB comparing with 176 MiB DOM in simdjson. Kind regards, Ilya
Json Algebraics with Mir
Hi all,
As you may know, [Mir](https://github.com/libmir) provides two
JSON libraries: [asdf](http://asdf.libmir.org/) and WIP
[mir-ion](http://mir-ion.libmir.org/). The last one is based on
Amazon's Ion dual format.
Both libraries provide a direct de/serialization API that doesn't
need to have a mutable JSON value. This works awesome for almost
all cases, except when we want to work with JSON tree directly
and dynamically change it. Or sometimes a JSON value may have
different types and we want to check them at runtime.
Looking into JSON value implementations, we can find that they
are _tagged nullable self-referencing algebraic_ types.
- _algebraic_ - type can store a value of a type from a fixed
typeset.
- _self-referencing_ - algebraic typeset types can refer to this
algebraic type
- _nullable_ - `typeof(null)` type is supported, and it is the
default
- _tagged_ - we have an enumeration (`enum`) of the whole
typeset, and value has a property, which returns an enumeration
value that corresponds to the underlying type.
We can define such algebraics with
[mir.algebraic](http://mir-core.libmir.org/mir_algebraic.html):
```d
import mir.algebraic: TaggedVariant, This;
union JsonAlgebraicUnion
{
typeof(null) null_;
bool boolean;
long integer;
double float_;
immutable(char)[] string;
/// Self alias in array
This[] array;
/// Self alias in associative
This[immutable(char)[]] object;
}
alias JsonAlgebraic = TaggedVariant!JsonAlgebraicUnion;
unittest
{
JsonAlgebraic value;
JsonAlgebraic[string] object;
// Default
assert(value.isNull);
assert(value.kind == JsonAlgebraic.Kind.null_);
// Boolean
value = true;
object["key"] = value;
assert(!value.isNull);
assert(value == true);
assert(value.kind == JsonAlgebraic.Kind.boolean);
assert(value.get!bool == true);
assert(value.get!(JsonAlgebraic.Kind.boolean) == true);
...
```
We added serialization of
[mir.algebraic](http://mir-core.libmir.org/mir_algebraic.html) a
few months ago to both JSON libraries. It is effortless to
implement: send a serialization lambda to a visitor. And the
typeset isn't limited to JSON-like types (bool, string, double,
etc.). If all types of the typeset are serializable, then
algebraic is serializable as well.
Deserialization is more complicated. We need to define a rule of
how we want to deserialize a JSON type set to an algebraic
typeset. And more, we don't need to provide a unique
JsonAlgebraic type.
Instead, we provide a common JSON algebraic alias
[mir.algebraic_alias.json](http://mir-algorithm.libmir.org/mir_algebraic_alias_json.html) and support user-provided algebraic aliases as well.
For example, user-provided types can be:
```d
import mir.algebraic: Variant, Nullable, This;
struct Color { ubyte a, r, g, b; }
alias JsonValue0 = Variant!(string, Color[], This[string]);
alias JsonValue1 = Nullable!(This[], long);
```
Asdf match Json types according to the following rules:
- `typeof(null)` can handle JSON `null`
- `bool` can handle JSON `true` and `false`
- `string` can handle JSON strings
- `double` can handle JSON numbers
- `long` can handle JSON integer numbers and has priority for
them comparing to `double`
- `T[]` can handle JSON arrays, where T is a deserializable
type, including the algebraic itself
- `StringMap!T` and `T[string]` can handle JSON objects, where T
is a deserializable type, including the algebraic type itself.
[StringMap](http://mir-algorithm.libmir.org/mir_string_map.html)
is an ordered string-value associative array with fast search
operations. It has a priority over built-in associative arrays.
- Other types of algebraic typeset aren't used for
deserialization.
- If no type can handle the current JSON value, then an
exception is thrown.
Users APIs will be consistent even if they will define different,
their own JSON algebraic aliases. Mir algebraic types:
- are order-independent: `Variant!(A, B)` is the same type as
`Variant!(B, A)`
- can be constructed from their algebraic subset.
- can get their algebraic subset.
mir-ion support for algebraic deserialization will be added later
and extended with Ion types, including Blob, Clob, Timestamp,
Decimal, and 4 bytes floats.
Kind regards,
Ilya
---
This work has been sponsored by Symmetry Investments and Kaleidic
Associates.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 6 January 2021 at 02:30:30 UTC, Walter Bright wrote: On 1/5/2021 2:42 AM, 9il wrote: On Tuesday, 5 January 2021 at 09:47:41 UTC, Walter Bright wrote: On 1/4/2021 11:22 PM, 9il wrote: I can't reproduce the same DMD output as you. I did it on Windows 32 bit. I tried it on Linux 32, which does indeed show the behavior you mentioned. At the moment I don't know why the different behaviors. https://issues.dlang.org/show_bug.cgi?id=21526 It just uses SSE, which I think a good way to go, haha. As I mentioned upthread, it will use SSE when SSE is baseline on the CPU target, and it will always round to precision. Does this mean that DMD Linux 32-bit executables should compile with SSE codes? The baseline Linux target does not have SSE. I ask because DMD compiles Linux 32-bit executables with x87 codes when -O is passed and with SSE if no -O is passed. That is very weird. Example, please? DMD with flag -m32 generates https://cpp.godbolt.org/z/GMGMra assume CS:.text._D7example1fFffZf pushEBP mov EBP,ESP sub ESP,018h movss XMM0,0Ch[EBP] movss XMM1,8[EBP] addss XMM0,XMM1 movss -8[EBP],XMM0 subss XMM0,XMM1 movss -4[EBP],XMM0 movss -018h[EBP],XMM0 fld float ptr -018h[EBP] leave ret 8 add [EAX],AL It has been provided in the thread https://forum.dlang.org/post/gqzdiicrvtlicurxy...@forum.dlang.org
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 5 January 2021 at 09:47:41 UTC, Walter Bright wrote: On 1/4/2021 11:22 PM, 9il wrote: I can't reproduce the same DMD output as you. I did it on Windows 32 bit. I tried it on Linux 32, which does indeed show the behavior you mentioned. At the moment I don't know why the different behaviors. https://issues.dlang.org/show_bug.cgi?id=21526 It just uses SSE, which I think a good way to go, haha. As I mentioned upthread, it will use SSE when SSE is baseline on the CPU target, and it will always round to precision. Does this mean that DMD Linux 32-bit executables should compile with SSE codes? I ask because DMD compiles Linux 32-bit executables with x87 codes when -O is passed and with SSE if no -O is passed. That is very weird.
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 5 January 2021 at 03:20:16 UTC, Walter Bright wrote:
On 1/4/2021 4:11 AM, 9il wrote:
[...]
The reason those switches are provided is because the
write/read is a performance hog.
D provides a couple functions in druntime which guarantee
rounding intermediate values to float/double precision. Those
can be used as required. This is better than a compiler switch
because having compiler switches that influence floating point
results is poor design.
> Since C99 the default x87 behavior is precise.
Not entirely:
float f(float a, float b) {
float d = (a + b) - b;
return d;
}
f:
sub esp, 4
fld DWORD PTR [esp+12]
fld st(0)
faddDWORD PTR [esp+8]
[no write/read to memory here, so no round to float]
fsubrp st(1), st
fstpDWORD PTR [esp]
fld DWORD PTR [esp]
add esp, 4
ret
In any case, let's try your example
https://cpp.godbolt.org/z/7sa8dP with dmd for 32 bits:
pushEAX
pushEAX
fld float ptr 010h[ESP]
faddfloat ptr 0Ch[ESP]
fstpfloat ptr [ESP] // there's the write
fld float ptr [ESP] // there's the read!
fsubfloat ptr 0Ch[ESP]
fstpfloat ptr 4[ESP]// the write
fld float ptr 4[ESP]// the read
add ESP,8
ret 8
It's semantically equivalent to the godbolt asm you posted.
I can't reproduce the same DMD output as you.
DMD with flags -m32 -O generates
https://cpp.godbolt.org/z/9b4e9K
assume CS:.text._D7example1fFffZf
pushEBP
mov EBP,ESP
fld float ptr 0Ch[ESP]
faddfloat ptr 8[EBP]
fsubfloat ptr 8[EBP]
pop EBP
ret 8
add [EAX],AL
add [EAX],AL
As you can see there are no write-read op codes.
DMD with flag -m32 generates
https://cpp.godbolt.org/z/GMGMra
assume CS:.text._D7example1fFffZf
pushEBP
mov EBP,ESP
sub ESP,018h
movss XMM0,0Ch[EBP]
movss XMM1,8[EBP]
addss XMM0,XMM1
movss -8[EBP],XMM0
subss XMM0,XMM1
movss -4[EBP],XMM0
movss -018h[EBP],XMM0
fld float ptr -018h[EBP]
leave
ret 8
add [EAX],AL
It just uses SSE, which I think a good way to go, haha. Probably
if no one has raised this bug then all real-world DMD targets
have at least SSE support.
The only D compiler that uses excess precision is DMD and only if
-O flag is passed. The same example compiled with GDC uses
write-read codes. LDC uses SSE codes.
As for C, it allows an intuitive built-in way to work with exact
precision when an assignment works like a directive to use exact
precision for the expression result, unlike D. It doesn't cover
all cases but an intuitive and very easy way to do things the
right way.
Re: Printing shortest decimal form of floating point number with Mir
On Monday, 4 January 2021 at 05:58:09 UTC, Walter Bright wrote: On 1/3/2021 8:37 PM, 9il wrote: I didn't believe it when I got a similar answer about IEEE floating-point numbers: D doesn't pertinent to be IEEE 754 compatible language and the extended precision bug is declared to be a language feature. The "extended precision bug" is how all x87 code works, C to C++ to Java. The reason is simple - to remove the problem requires all intermediate results to be written to memory and read back in, which is a terrible performance problem. Early Java implementations did this write/read, and were forced to change it. Since C99 the default x87 behavior is precise. https://cpp.godbolt.org/z/7sa8dP For older C versions GCC provides -fexcess-precision=standard flag. Java is going to restore the original behavior. https://bugs.openjdk.java.net/browse/JDK-8175916 C# has an option to control virtual machine behavior https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/control87-controlfp-control87-2?view=msvc-160 Finally, x87 is a deprecated architecture. It likely will be supported for a few decades. From the business's point of view, no one expects the best performance from the code compiled for x87. Instead of speed, numeric correctness is much more important here. I wouldn't explain why extended precision is inaccurate, because Kahan and Darcy already wrote an 80-page essay about it: "How Java’s Floating-Point Hurts Everyone Everywhere" https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
Re: Printing shortest decimal form of floating point number with Mir
On Sunday, 3 January 2021 at 22:50:16 UTC, Ola Fosheim Grøstad wrote: On Sunday, 3 January 2021 at 20:31:41 UTC, welkam wrote: You should put yourself in the boots of Atila. If you accept a change that later turns out to be bad idea you cant just take it out. This is just silly. You don't have to accept a specific solution... ...but... YOU DO HAVE TO ACKNOWLEDGE A TYPE SYSTEM BUG! If an indirection through an alias causes type unification to fail then that is a serious type system failure. No excuses please... "workarounds" are indeed just excuses, telling people "workarounds" they already know about is borderline offensive. I wouldn't call it mocking, but I certainly see why it can be perceived as such. I suppose the answer would be that D doesn't pretend to support all C++ template features and the bug is not a bug because we live with this somehow for years. I didn't believe it when I got a similar answer about IEEE floating-point numbers: D doesn't pertinent to be IEEE 754 compatible language and the extended precision bug is declared to be a language feature. I suppose we shouldn't expect D to pretend to be a robust language for large business projects.
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 29 December 2020 at 19:59:56 UTC, Ola Fosheim Grøstad wrote: On Tuesday, 29 December 2020 at 16:14:59 UTC, Atila Neves wrote: On Thursday, 24 December 2020 at 14:14:33 UTC, 9il wrote: On Thursday, 24 December 2020 at 14:08:32 UTC, welkam wrote: On Wednesday, 23 December 2020 at 18:05:40 UTC, 9il wrote: It was a mockery executed by Atila Read the all comments and didnt saw any mockery Yes, it wasn't explicit. He didn't write bad words, he did a bad decision. Bad for D. I apologise if what I wrote came across as mockery; it certainly wasn't intended that way. How would you have liked for me to have handled it better? I am not speaking for Ilya, but from skimming through the dialogue it struck me that you didn't respond from the perspective of managing the process, but from a pure engineer mindset of providing alternatives. It would've been better if you started by 1. understanding the issue 2. acknowledging that the type system has an obvious bug 3. looking at the issue from the perspective of the person bringing attention to the issue. I don't think anyone was looking for workarounds, but looking for 1. acknowledgment of the issue 2. acknowledgment of what the issue leads to in terms of inconvenience 3. a forward looking vision for future improvements +1
Re: Printing shortest decimal form of floating point number with Mir
On Thursday, 24 December 2020 at 14:08:32 UTC, welkam wrote: On Wednesday, 23 December 2020 at 18:05:40 UTC, 9il wrote: It was a mockery executed by Atila Read the all comments and didnt saw any mockery Yes, it wasn't explicit. He didn't write bad words, he did a bad decision. Bad for D.
Re: Printing shortest decimal form of floating point number with Mir
On Thursday, 24 December 2020 at 11:05:16 UTC, Ola Fosheim
Grøstad wrote:
On Wednesday, 23 December 2020 at 20:56:26 UTC, jmh530 wrote:
[...]
Non-concept version is more verbose, but yeah, works fine in
C++17:
namespace detail {
template class F, class U>
static constexpr void _dummy(const F );
template typename F, class=void>
struct has_outer_template : std::false_type {};
template typename F>
struct
has_outer_template(std::declval()))>>: std::true_type {};
};
template typename F>
inline constexpr bool has_outer_template =
detail::has_outer_template::value;
template
struct Foo{};
static_assert(has_outer_template,Foo>);
Thank you for the examples. They make sense.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 23 December 2020 at 18:23:25 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 18:05:40 UTC, 9il wrote: Have you read the DMD PR thread (not the DIP itself)? It was a mockery executed by Atila accompanied by silent Walter's and Andrei's ignoring. I am not sure if I read the same one, I didn't perceive it as mockery. Atila seemed to imply that a bigger language change is needed and that this DIP would only fix one bit of what has to change? Then he tried to suggest alternatives. Maybe he did not understand that it makes the compiler look buggy. The communication was not very clear though… Looked more like people talking past each other than mockery. Or that was just a very good mockery. But as was said that doesn't really matter for the result we have. I am using D last 11-12 years. This case wasn't an exception, it was a Dlang failure among consequent others.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 23 December 2020 at 17:22:28 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 17:08:26 UTC, 9il wrote: I don't use tensors much, how does it help zipping? Safe optimizations. Mir uses unsafe ugly workarounds somehow in some places and doesn't improve some stuff that can be improved. Lazy tensors are used in https://github.com/typohnebild/numpy-vs-mir which has few kernels. The fastest one is `ndslice`, which uses lazy zipped tensors. Comparing with Phobos zipped ranges, Mir's zipped tensors are mutable.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 23 December 2020 at 17:22:28 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 17:08:26 UTC, 9il wrote: https://issues.dlang.org/show_bug.cgi?id=16486 https://issues.dlang.org/show_bug.cgi?id=16465 https://issues.dlang.org/show_bug.cgi?id=10884 and the oldest one reported in 2008 https://issues.dlang.org/show_bug.cgi?id=1807 C++ templates can be resolved, at least at the level Mir needs this. So, it is a bug in my opinion. But it was said the DIP is required. I can't write DIP well and was very happy that Stefanos wrote the DIP and even the druft. Yes, if something is perceived as bug it becomes a burden to remember that it is isn't. Not sure why anyone resist this improvement. Hm, he seems to be a compiler consultant now, but no longer interested in D? Hi is disappeared from the Dlang after that. Maybe the DIP should have pushed harder on what other languages support (might be viewed as a stronger political argument). Have you read the DMD PR thread (not the DIP itself)? It was a mockery executed by Atila accompanied by silent Walter's and Andrei's ignoring. I know Atila in person. However, it doesn't really matter if Atila really didn't understand the DIP reasons or it was a real mockery. The fact that this behavior including real or seeming mockery and real ignoring is a red flag for any professional cooperation. Atila had been already declared as "new Andrei". Which was noted right in the DIP to define Atila's privileges to make decisions. https://github.com/dlang/dmd/pull/9778#issuecomment-498700369 I see how builtin tuples could be useful for a linalg library. I like how Python allows just using ",". Makes code easier on the eyes x,y = y,x It is also very desired for lazy zipped tensors. I don't use tensors much, how does it help zipping? I sometimes wonder if linalg primitives should be builtin too. Seems like that could allow for better compiler optimization.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 23 December 2020 at 16:51:33 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 16:20:37 UTC, Timon Gehr wrote: On 23.12.20 16:37, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 03:06:51 UTC, 9il wrote: You, Andrey, and Atila don't care about language features that have been requested for Mir or even more: rejecting DIP draft + DMD partial implementation for no real reason. Out of curiosity, which language features would improve Mir? https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1023.md This looks like a bug? https://issues.dlang.org/show_bug.cgi?id=16486 https://issues.dlang.org/show_bug.cgi?id=16465 https://issues.dlang.org/show_bug.cgi?id=10884 and the oldest one reported in 2008 https://issues.dlang.org/show_bug.cgi?id=1807 C++ templates can be resolved, at least at the level Mir needs this. So, it is a bug in my opinion. But it was said the DIP is required. I can't write DIP well and was very happy that Stefanos wrote the DIP and even the druft. I see how builtin tuples could be useful for a linalg library. I like how Python allows just using ",". Makes code easier on the eyes x,y = y,x It is also very desired for lazy zipped tensors.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 23 December 2020 at 15:37:45 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 03:06:51 UTC, 9il wrote: You, Andrey, and Atila don't care about language features that have been requested for Mir or even more: rejecting DIP draft + DMD partial implementation for no real reason. Out of curiosity, which language features would improve Mir? 1. Alias template function parameter resolution https://github.com/dlang/dmd/pull/9778 As a result D and Mir lost Stefanos Baziotis. That is terrible, hi is very talented. 2. Multiple auto ref return values - when function allows returning multiple values.
Re: Printing shortest decimal form of floating point number with Mir
On Wednesday, 23 December 2020 at 16:20:37 UTC, Timon Gehr wrote: On 23.12.20 16:37, Ola Fosheim Grøstad wrote: On Wednesday, 23 December 2020 at 03:06:51 UTC, 9il wrote: You, Andrey, and Atila don't care about language features that have been requested for Mir or even more: rejecting DIP draft + DMD partial implementation for no real reason. Out of curiosity, which language features would improve Mir? https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1023.md https://forum.dlang.org/post/kvcrsoqozrflxibgx...@forum.dlang.org https://forum.dlang.org/thread/gungkvmtrkzcahhij...@forum.dlang.org?page=1 https://forum.dlang.org/post/jwtygeybvfgbosxsb...@forum.dlang.org Thanks, that is a more detailed list.
Re: Truly algebraic Variant and Nullable
On Tuesday, 22 December 2020 at 19:45:43 UTC, Paul Backus wrote: On Tuesday, 22 December 2020 at 16:54:56 UTC, 9il wrote: On Tuesday, 22 December 2020 at 16:43:30 UTC, ag0aep6g wrote: On Tuesday, 22 December 2020 at 16:32:20 UTC, 9il wrote: "Struct non-static methods marked with the return attribute ensure the returned reference will not outlive the struct instance." The issue isn't that the reference outlives the struct. It's that the reference outlives a tag change of the tagged union. If I am correct Dlang doesn't provide an instrument to validate it, isn't it? What alternative is possible? The solution sumtype uses is to make opAssign @system if the union contains any unsafe types. That is interesting, I have missed it. Thanks
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 22 December 2020 at 21:53:20 UTC, Walter Bright wrote: If you don't want the formatting code to be part of Phobos, I respect your choice. Why did you think I may want it? Phobos is almost not used in my work. You, Andrey, and Atila don't care about language features that have been requested for Mir or even more: rejecting DIP draft + DMD partial implementation for no real reason. Why should I care about something important for you while you act like three tsars that care only about their party for years? How come?
Re: Truly algebraic Variant and Nullable
On Tuesday, 22 December 2020 at 17:07:16 UTC, ag0aep6g wrote: On Tuesday, 22 December 2020 at 16:54:56 UTC, 9il wrote: On Tuesday, 22 December 2020 at 16:43:30 UTC, ag0aep6g wrote: [...] [...] If I am correct Dlang doesn't provide an instrument to validate it, isn't it? What alternative is possible? Returning it by value isn't acceptable at least because of performance reasons: the first target of the library is struts with a lot of Mir ref-counted fields. Mark it @system then. [...] This is why it is the market as `trusted`. It is much more convenient than forcing users to wrap each access with @trusted lambda, which feels like masochism. @trusted makes the exact same promise to the user as @safe. If your method doesn't have a safe interface, don't mark it @safe or @trusted. Mark it @system. Hmm, maybe it worth changing the API a bit. Needs to wait for mir-core 1.2.x thought.
Re: Truly algebraic Variant and Nullable
On Tuesday, 22 December 2020 at 16:43:30 UTC, ag0aep6g wrote: On Tuesday, 22 December 2020 at 16:32:20 UTC, 9il wrote: "Struct non-static methods marked with the return attribute ensure the returned reference will not outlive the struct instance." The issue isn't that the reference outlives the struct. It's that the reference outlives a tag change of the tagged union. If I am correct Dlang doesn't provide an instrument to validate it, isn't it? What alternative is possible? Returning it by value isn't acceptable at least because of performance reasons: the first target of the library is struts with a lot of Mir ref-counted fields. The docs maybe not clear enough. `trustedGet` asserts type is matched, while `get` throws an exception if the type doesn't match. You can't rely on an assert for @safe (unless it's `assert(false);`). This is why it is the market as `trusted`. It is much more convenient than forcing users to wrap each access with @trusted lambda, which feels like masochism.
Re: Truly algebraic Variant and Nullable
On Tuesday, 22 December 2020 at 14:27:02 UTC, ag0aep6g wrote: On 22.12.20 04:56, 9il wrote: 6. Algebraic type subsets are supported by `get`, `trustedGet`, `_is`, and `this` primitives. You can operate with algebraic subset as with the type of the original typeset. [1] "trustedGet" - That name smells of a safety violation. And indeed (compile with `-release`): The normal `get` also violates safety by giving out references into the union (compile with `-preview=dip1000`): But that might be an issue with DIP1000. `ref_to_ptr` is a hint that something isn't right in that area. The definitions are ``` auto ref get(E)() @property return inout auto ref trustedGet(E)() @trusted @property return inout nothrow ``` Both market with `return`. According to the spec [1, example 2] "Struct non-static methods marked with the return attribute ensure the returned reference will not outlive the struct instance." The docs maybe not clear enough. `trustedGet` asserts type is matched, while `get` throws an exception if the type doesn't match. [1] https://dlang.org/spec/function.html#return-ref-parameters
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 22 December 2020 at 09:18:25 UTC, Walter Bright wrote: On 12/21/2020 8:33 PM, 9il wrote: These functions in Phobos would make a great advertisement for Mir. How this possible? A lot more people will have Phobos than Phobos+Mir. If they are perusing the source code and see Mir contributed excellent floating point formatting code, they may have never heard of Mir but have now. "If, If Is Good" (Disney Company). From the marketing point of view, this doesn't make real sense. Then they'll be likely to be positively disposed towards using Mir because of the high quality code. Mir doesn't need a Phobos conformity mark. In many designs and implementation questions, Phobos is far behind Mir. The reality is that Phobos asks for 6K+ LOC Mir's code, while Phobos legacy in Mir's codebase is less than a quite well reworked few percentages. It's the same idea as HBO offering the first episode for free in a miniseries. People watch the first episode, like it, and then subscribe to HBO. I don't take payments from people to use Mir. They don't need to dig in Phobos source code to find it. Likely they will search GitHub or code.dlang.org to find a solution they need. Having them in Mir is already a great advertisement for Mir Since they exist in the C standard library (except for DMC :-( ) they by themselves aren't a compelling reason for someone to use Mir. They are, Mir comes with a CTFE/@nogc/nothrow formatting API and these functions are play well inside.
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 22 December 2020 at 04:33:55 UTC, 9il wrote: On Tuesday, 22 December 2020 at 02:02:24 UTC, Walter Bright wrote: On 12/20/2020 9:42 PM, 9il wrote: On Sunday, 20 December 2020 at 22:21:56 UTC, Walter Bright wrote: Can the improved parsing be added to D? (It would need to be Boost licensed.) If I am correct there is open PR that set DMD to use C’s primitives for literals parsing. So, for compiler itself we don’t need Mir. That's not correct for the targets that use Digital Mars C. I thought that DMD is compiled with LDC for release builds, isn't it? If you mean Phobos - one can use Mir instead. These functions in Phobos would make a great advertisement for Mir. How this possible? Having them in Mir is already a great advertisement for Mir and not having them in Phobos is an even more great advertisement for Mir. ... I just have thought maybe I have missed something and DLF helps Mir with advertising at least a bit, maybe at least with two-three tweets per year? The last time @D_Programming tweeted something about Mir was in 2016.
Re: Printing shortest decimal form of floating point number with Mir
On Tuesday, 22 December 2020 at 02:02:24 UTC, Walter Bright wrote: On 12/20/2020 9:42 PM, 9il wrote: On Sunday, 20 December 2020 at 22:21:56 UTC, Walter Bright wrote: Can the improved parsing be added to D? (It would need to be Boost licensed.) If I am correct there is open PR that set DMD to use C’s primitives for literals parsing. So, for compiler itself we don’t need Mir. That's not correct for the targets that use Digital Mars C. I thought that DMD is compiled with LDC for release builds, isn't it? If you mean Phobos - one can use Mir instead. These functions in Phobos would make a great advertisement for Mir. How this possible? Having them in Mir is already a great advertisement for Mir and not having them in Phobos is an even more great advertisement for Mir.
Re: Truly algebraic Variant and Nullable
On Sunday, 20 December 2020 at 12:32:35 UTC, Petar Kirov [ZombineDev] wrote: How does your work compare to sumtype? Would mir.algebraic offer any benefits, which would make it worth switching over? replied at https://forum.dlang.org/post/zlphfxktclgdookqt...@forum.dlang.org If we can work together to consolidate on a single API, I think it would be better for the language ecosystem. Agreed. On the other hand, my public association with a DIP would be a red flag and will increase the chance the DIP would be declined. Cooperation is better to make silently.
Re: Truly algebraic Variant and Nullable
On Sunday, 20 December 2020 at 11:00:05 UTC, Tobias Pankrath wrote: On Sunday, 15 November 2020 at 04:54:19 UTC, 9il wrote: Truly algebraic Variant and Nullable with an order-independent list of types. Thanks for sharing it! Could you give a (very short) explanation on why sumtype could not meet your requirements? I am just starting a new D project and have to choose between sumtype and your solution. Lets users do comparisons between libraries. Both are very good. Some mir.algebraic features: 1. (optionally) Nullable algebraic types. Also serves as buggy Phobos Nullable replacement. 2. (optionally) Tagged algebraic types 3. Type list order-independent declaration 4. Feature-rich visitor handlers. For example, they can form new Algebraic types if the visitors return different types. 5. `void` support. This is an important brick for reflections on the algebra of type sets. 6. Algebraic type subsets are supported by `get`, `trustedGet`, `_is`, and `this` primitives. You can operate with algebraic subset as with the type of the original typeset. [1] 7. Members (fields and methods) reflection. Is more restrictive than in vibe.d implementation. It adds member reflection to an algebraic type if all of the types contain members with the same name. Mir implements Algebra of (type) sets with reflections (functions) on it. [1] https://github.com/libmir/mir-core/issues/33
Re: Printing shortest decimal form of floating point number with Mir
On Sunday, 20 December 2020 at 22:21:56 UTC, Walter Bright wrote: On 12/13/2020 10:47 PM, 9il wrote: Note that D's compiler floating-point literals parsing and Phobos floating-point literals parsing are not precise [5,6,7,8]. It is recommended to use Mir's to!double/float/real to convert floating-point numbers from a string. Can the improved parsing be added to D? (It would need to be Boost licensed.) If I am correct there is open PR that set DMD to use C’s primitives for literals parsing. So, for compiler itself we don’t need Mir. If you mean Phobos - one can use Mir instead.
Re: Printing shortest decimal form of floating point number with Mir
On Monday, 14 December 2020 at 06:47:32 UTC, 9il wrote: Hi all, Generic version of Ryu algorithm [1] was ported to D, well optimized, and adopted to mir packages. [...] Default formatting has been reworked to be more human-friendly: 1.23e1 -> 1.23 https://github.com/ulfjack/ryu/tree/master/ryu Microsoft, Clang++, and others are adopting Ryu as well.
Re: Truly algebraic Variant and Nullable
On Thursday, 17 December 2020 at 15:38:52 UTC, jmh530 wrote: On Thursday, 17 December 2020 at 15:12:12 UTC, 9il wrote: On Wednesday, 16 December 2020 at 16:14:08 UTC, jmh530 wrote: On Wednesday, 16 December 2020 at 15:58:21 UTC, 9il wrote: [...] What about making it into a sub-package, as in here [1]? [1] https://github.com/atilaneves/unit-threaded/tree/master/subpackages It takes 0.1 seconds to compile mir-core with LDC in the release mode. It is almost all generic and quite fast to compile. We can, but dub doesn't always work well with submodules. Is there any other reason except compilation speed? You can put it on code.dlang.org as a subPackage and people can download it without downloading all of mir-core. See below: https://code.dlang.org/packages/unit-threaded dub downloads the whole package if just a subpackage is required. The size of mir-core is less then 0.5 mb and 0.1 mb in Zip archive.
Re: Truly algebraic Variant and Nullable
On Wednesday, 16 December 2020 at 18:14:54 UTC, Oleg B wrote:
On Wednesday, 16 December 2020 at 15:58:21 UTC, 9il wrote:
On Wednesday, 16 December 2020 at 14:54:26 UTC, Oleg B wrote:
On Sunday, 15 November 2020 at 04:54:19 UTC, 9il wrote:
[...]
Great library! Have you any plan to separate it from mir-core
(to mir-algebraic for example)?
Thanks! Maybe, but mir-core is quite small itself and
mir.algebraic is the only part that would be extended or
updated in the near future. Other parts are quite stable. If
there would be a strong reason to split it, we can do it.
That are you planing update? It's will be perfect if you add
`get` overload for kind type and more work with tags [2]
like that:
```
alias TUnion = Algebraic!(
TaggedType!(int, "count"),
TaggedType!(string, "str")
);
auto v = TUnion("hello");
S: final switch (v.kind)
{
static foreach (i, k; EnumMembers!(k.Kind))
case k:
someFunction(v.get!k); // [1] by now
v.get!(TUnion.AllowedTypes[i])
break S;
}
if (v.is_count) // [2]
writeln(v.count);
```
or may be I miss this feature in docs?
Get by Kind cand be added.
You can define TaggedAlgebraic [1].
It has a bit weird API with a separate array of lengths. But we
can add another one definition option I think.
http://mir-core.libmir.org/mir_algebraic.html#TaggedVariant
Re: Truly algebraic Variant and Nullable
On Wednesday, 16 December 2020 at 16:14:08 UTC, jmh530 wrote: On Wednesday, 16 December 2020 at 15:58:21 UTC, 9il wrote: [snip] Thanks! Maybe, but mir-core is quite small itself and mir.algebraic is the only part that would be extended or updated in the near future. Other parts are quite stable. If there would be a strong reason to split it, we can do it. What about making it into a sub-package, as in here [1]? [1] https://github.com/atilaneves/unit-threaded/tree/master/subpackages It takes 0.1 seconds to compile mir-core with LDC in the release mode. It is almost all generic and quite fast to compile. We can, but dub doesn't always work well with submodules. Is there any other reason except compilation speed?
Re: Truly algebraic Variant and Nullable
On Wednesday, 16 December 2020 at 14:54:26 UTC, Oleg B wrote: On Sunday, 15 November 2020 at 04:54:19 UTC, 9il wrote: Truly algebraic Variant and Nullable with an order-independent list of types. Nullable is defined as ``` alias Nullable(T...) = Variant!(typeof(null), T); ``` Variant and Nullable with zero types are allowed. `void` type is supported. Visitors are allowed to return different types. Cyclic referencing between different variant types are supported. More features and API: http://mir-core.libmir.org/mir_algebraic.html Cheers, Ilya The work has been sponsored by Kaleidic Associates and Symmetry Investments. Great library! Have you any plan to separate it from mir-core (to mir-algebraic for example)? Thanks! Maybe, but mir-core is quite small itself and mir.algebraic is the only part that would be extended or updated in the near future. Other parts are quite stable. If there would be a strong reason to split it, we can do it.
Printing shortest decimal form of floating point number with Mir
Hi all,
Generic version of Ryu algorithm [1] was ported to D, well
optimized, and adopted to mir packages.
It allows printing the shortest (scientific) decimal form of a
floating-point number that if it is converted back would produce
the same floating-point number.
The update requires mir-algorithm [2] >=3.10.13
@safe pure nothrow unittest
{
import mir.conv: to;
assert(12.3.to!string == "1.23e1");
assert(12.3456789.to!string == "1.23456789e1");
// CTFE-able
static assert(12.3456789.to!string == "1.23456789e1");
}
@safe pure @nogc unittest
{
// @nogc
import mir.conv: to;
import mir.small_string;
assert(12.3.to!(SmallString!32) == "1.23e1");
assert(12.3456789.to!(SmallString!32) == "1.23456789e1");
}
@safe pure @nogc nothrow unittest
{
// @nogc
import mir.format;
stringBuf buffer;
auto data = buffer << 12.3 << ", " << 12.3456789 << getData;
assert(data == "1.23e1, 1.23456789e1");
}
Floating-point numbers can be converted to stack-allocated
decimal numbers.
@safe pure nothrow @nogc
unittest
{
// float and double can be used to construct Decimal of
any length
auto decimal64 = Decimal!1(-1.235e-7);
assert(decimal64.exponent == -10);
assert(decimal64.coefficient == -1235);
// real number may need Decimal at least length of 2
auto decimal128 = Decimal!2(-1.235e-7L);
assert(decimal128.exponent == -10);
assert(decimal128.coefficient == -1235);
decimal128 = Decimal!2(1234e3f);
assert(decimal128.exponent == 3);
assert(decimal128.coefficient == 1234);
}
Recent releases of ASDF [3] and Mir Ion [4] use this formatting
by default. It allows performing JSON serialization without loss
of precision.
Note that D's compiler floating-point literals parsing and Phobos
floating-point literals parsing are not precise [5,6,7,8]. It is
recommended to use Mir's to!double/float/real to convert
floating-point numbers from a string.
The work has been sponsored by Symmetry Investments and Kaleidic
Associates.
Kind regards,
Ilya
[1] https://github.com/ulfjack/ryu
[2] http://mir-algorithm.libmir.org/
[3] http://asdf.libmir.org/
[4] http://mir-ion.libmir.org/
[5] https://issues.dlang.org/show_bug.cgi?id=20951
[6] https://issues.dlang.org/show_bug.cgi?id=20952
[7] https://issues.dlang.org/show_bug.cgi?id=20953
[8] https://issues.dlang.org/show_bug.cgi?id=20967
Matt Godbolt's Compiler Explorer supports Mir libraries
Hi all, Compiler Explorer [1] is an interactive compiler. The left-hand pane shows the editable code. The right, the assembly output of having compiled the code with a given compiler and settings. HowTo 1. open the site [1] 2. press 'Libraries' bottom on the right window and pick required libraries with all their dependencies. For example, mir-algorithm(trunk) and its dependency mir-core(trunk). 3. Pick LDC compiler 4. Use LDC's -mtriple= or -mcpu= flags to pick the target you want. LDC can do cross-compilation for ARM CPUs. 5. Add compiler flags like -O -release -boundscheck=off -mcpu=native 6. Past you code in the left window. 7. Enjoy [1] https://d.godbolt.org/
Re: Mir vs. Numpy: Reworked!
On Monday, 7 December 2020 at 12:28:39 UTC, data pulverizer wrote: On Monday, 7 December 2020 at 02:14:41 UTC, 9il wrote: I don't know. Tensors aren't so complex. The complex part is a design that allows Mir to construct and iterate various kinds of lazy tensors of any complexity and have quite a universal API, and all of these are boosted by the fact that the user-provided kernel(lambda) function is optimized by the compiler without the overhead. I agree that a basic tensor is not hard to implement, but the specific design to choose is not always obvious. Your benchmarks shows that design choices have a large impact on performance, and performance is certainly a very important consideration in tensor design. For example I had no idea that your ndslice variant was using more than one array internally to achieve its performance - it wasn't obvious to me. ndslice tensor type uses exactly one iterator. However, the iterator is generic and lazy iterators may contain any number of other iterators and pointers.
Re: Mir vs. Numpy: Reworked!
On Sunday, 6 December 2020 at 17:30:13 UTC, data pulverizer wrote: On Saturday, 5 December 2020 at 07:44:33 UTC, 9il wrote: sweep_ndslice uses (2*N - 1) arrays to index U, this allows LDC to unroll the loop. For example, for 2D case, withNeighboursSum [2] will store the pointer to the result, and the pointer at rows above and below. matrix: -- --a--- above iterator --r--- the result --b--- below iterator -- Also, for AVX-512 targets it allows vectorizing the loop [1]. The benchmark has been run on the AVX2 CPU. [1] https://github.com/typohnebild/numpy-vs-mir/issues/4 [2] http://mir-algorithm.libmir.org/mir_ndslice_topology.html#.withNeighboursSum Very interesting, thank you for the explanations. Are there journal/book other implementation references for these approaches to implementing tensor-like multidimensional arrays? I don't know. Tensors aren't so complex. The complex part is a design that allows Mir to construct and iterate various kinds of lazy tensors of any complexity and have quite a universal API, and all of these are boosted by the fact that the user-provided kernel(lambda) function is optimized by the compiler without the overhead.
Re: Mir vs. Numpy: Reworked!
On Friday, 4 December 2020 at 20:26:17 UTC, data pulverizer wrote: On Friday, 4 December 2020 at 14:48:32 UTC, jmh530 wrote: It looks like all the `sweep_XXX` functions are only defined for contiguous slices, as that would be the default if define a Slice!(T, N). How the functions access the data is a big difference. If you compare the `sweep_field` version with the `sweep_naive` version, the `sweep_field` function is able to access through one index, whereas the `sweep_naive` function has to use two in the 2d version and 3 in the 3d version. Also, the main difference in the NDSlice version is that it uses *built-in* MIR functionality, like how `sweep_ndslice` uses the `each` function from MIR, whereas `sweep_field` uses a for loop. I think this is partially to show that the built-in MIR functionality is as fast as if you tried to do it with a for loop yourself. I see, looking at some of the code, field case is literally doing the indexing calculation right there. I guess ndslice is doing the same thing just with "Mir magic" an in the background? sweep_ndslice uses (2*N - 1) arrays to index U, this allows LDC to unroll the loop. More details here https://forum.dlang.org/post/qejwviqovawnuniua...@forum.dlang.org I'm still not sure why slice is so slow. Doesn't that completely rely on the opSlice implementations? The choice of indexing method and underlying data structure? sweep_slice is slower because it iterates data in few loops rather than in a single one. For small matrices this makes JMP/FLOP ratio higher, for large matrices that can't feet into the CPU cache, it is less memory efficient.
Re: Mir vs. Numpy: Reworked!
On Friday, 4 December 2020 at 03:48:15 UTC, Walter Bright wrote: On 12/3/2020 8:27 AM, 9il wrote: Since the first announcement [0] the original benchmark [1] has been boosted [2] with Mir-like implementations. This is really great! Can you write an article about it? Such would be really helpful in letting people know about it. Thanks! The README is really great as the benchmark description. I will do a small article about Mir this year.
Re: Mir vs. Numpy: Reworked!
On Friday, 4 December 2020 at 02:35:49 UTC, data pulverizer wrote: On Thursday, 3 December 2020 at 21:28:04 UTC, jmh530 wrote: Am I correct in assuming that the data in the NDSlice is also a single array? sweep_ndslice uses (2*N - 1) arrays to index U, this allows LDC to unroll the loop. For example, for 2D case, withNeighboursSum [2] will store the pointer to the result, and the pointer at rows above and below. matrix: -- --a--- above iterator --r--- the result --b--- below iterator -- Also, for AVX-512 targets it allows vectorizing the loop [1]. The benchmark has been run on the AVX2 CPU. [1] https://github.com/typohnebild/numpy-vs-mir/issues/4 [2] http://mir-algorithm.libmir.org/mir_ndslice_topology.html#.withNeighboursSum
Re: Mir vs. Numpy: Reworked!
On Thursday, 3 December 2020 at 17:08:58 UTC, jmh530 wrote: On Thursday, 3 December 2020 at 16:27:59 UTC, 9il wrote: Looks good, but a few typos: Thanks!
Re: Mir vs. Numpy: Reworked!
On Thursday, 3 December 2020 at 16:50:39 UTC, Andre Pany wrote: On Thursday, 3 December 2020 at 16:27:59 UTC, 9il wrote: Hi all, Since the first announcement [0] the original benchmark [1] has been boosted [2] with Mir-like implementations. D+Mir: 1. is more abstract than NumPy 2. requires less code for multidimensional algorithms 3. doesn't require indexing 4. uses recursion across dimensions 5. a few times faster than NumPy for non-trivial real-world applications. Why Mir is faster than NumPy? 1. Mir allows the compiler to generate specialized kernels while NumPy constraints a user to write code that needs to access memory twice or more times. Another Mir killer feature is the ability to write generalized N-dimensional implementations, while Numpy code needs to have separate implementations for 1D, 2D, and 3D cases. For example, the main D loop in the benchmark can compile for 4D, 5D, and higher dimensional optimizations. 2. @nogc iteration loop. @nogc helps when you need to control what is going on with your memory allocations in the critical code part. [0] https://forum.dlang.org/post/pemharpztorlqkxdo...@forum.dlang.org [1] https://github.com/typohnebild/numpy-vs-mir [2] https://github.com/typohnebild/numpy-vs-mir/pull/1 The benchmark [1] has been created by Christoph Alt and Tobias Schmidt. Kind regards, Ilya Hi Ilya, Thanks a lot for sharing the update. I am currently working on porting a python package called FMPY to D. This package makes usage of numpy and I hope I can use MIR here. Probably you may want to express FMI entities as Algebraic types rather than classes. mir.algebraic can be really helpful here http://mir-core.libmir.org/mir_algebraic.html Somehow it is hard to get started to learn MIR. What maybe could help python developers is to have some articles showing numpy coding and side by side the equivalent MIR coding. It is hard for me to write articles. I will try to write a small one this year, but it would be Mir only. Maybe this benchmark can be used as an example and if one wishes to write a side-by-side comparison with NumPy I would be happy to comment and explain the D implementation and what it is doing internally. What I miss in MIR is a function to read and write CSV files. Is s.th. like numpy.genfromtxt planned? Unlikely I would add it but can do a code review. Currently, we can load/safe NumPy binary data with numir https://libmir.github.io/numir/io.html Kind regards, Ilya
Mir vs. Numpy: Reworked!
Hi all, Since the first announcement [0] the original benchmark [1] has been boosted [2] with Mir-like implementations. D+Mir: 1. is more abstract than NumPy 2. requires less code for multidimensional algorithms 3. doesn't require indexing 4. uses recursion across dimensions 5. a few times faster than NumPy for non-trivial real-world applications. Why Mir is faster than NumPy? 1. Mir allows the compiler to generate specialized kernels while NumPy constraints a user to write code that needs to access memory twice or more times. Another Mir killer feature is the ability to write generalized N-dimensional implementations, while Numpy code needs to have separate implementations for 1D, 2D, and 3D cases. For example, the main D loop in the benchmark can compile for 4D, 5D, and higher dimensional optimizations. 2. @nogc iteration loop. @nogc helps when you need to control what is going on with your memory allocations in the critical code part. [0] https://forum.dlang.org/post/pemharpztorlqkxdo...@forum.dlang.org [1] https://github.com/typohnebild/numpy-vs-mir [2] https://github.com/typohnebild/numpy-vs-mir/pull/1 The benchmark [1] has been created by Christoph Alt and Tobias Schmidt. Kind regards, Ilya
Re: MIR vs. Numpy
On Wednesday, 18 November 2020 at 10:05:06 UTC, Tobias Schmidt wrote: Dear all, to compare MIR and Numpy in the HPC context, we implemented a multigrid solver in Python using Numpy and in D using Mir and perforemd some benchmarks with them. You can find our code and results here: https://github.com/typohnebild/numpy-vs-mir Feedback is very welcome. Please feel free to open issues, pull requests or simply post your thoughts below. Kind regards, Tobias Thank you a lot! It is a huge benefit for Mir and D to have so quality benchmarks. Python's sweep_3D access memory only once for one element computation, while old D's sweep_slice access it 7 times. A PR [1] for new version of sweep_slice was added, I expect it will be at least twice faster. The new sweep_slice uses a more D'sh approach and single memory access to the computation element. [1] https://github.com/typohnebild/numpy-vs-mir/pull/1 Cheers, Ilya
Re: MIR vs. Numpy
On Wednesday, 18 November 2020 at 13:14:37 UTC, jmh530 wrote: On Wednesday, 18 November 2020 at 10:05:06 UTC, Tobias Schmidt wrote: It also looks like you are compiling on ldc with -mcpu=native --boundscheck=off. Why not -O as well? -O is added by DUB
Truly algebraic Variant and Nullable
Truly algebraic Variant and Nullable with an order-independent list of types. Nullable is defined as ``` alias Nullable(T...) = Variant!(typeof(null), T); ``` Variant and Nullable with zero types are allowed. `void` type is supported. Visitors are allowed to return different types. Cyclic referencing between different variant types are supported. More features and API: http://mir-core.libmir.org/mir_algebraic.html Cheers, Ilya The work has been sponsored by Kaleidic Associates and Symmetry Investments.
Re: mir-stat
On Sunday, 11 October 2020 at 17:10:19 UTC, jmh530 wrote: On Sunday, 11 October 2020 at 10:14:04 UTC, tastyminerals wrote: On Thursday, 8 October 2020 at 16:40:01 UTC, 9il wrote: It is a pleasure to announce the Dlang Statistical Package by John Michael Hall. [...] Awesome! Are there any plans to add functions for inferential stats? Next thing I want to add is histogram (influenced by Boost histogram), but I have been a bit busy lately and haven't finished it. After histogram, the next step would probably be pdfs/cdfs/icdfs, but I was thinking about just borrowing from what is in dstats (I'll need to look into the license compatibility). With those functions in there, then t-test and similar functions would be straightforward. Maybe we should replace Boost with MIT for most of the Mir packages. What do you think?
mir-stat
It is a pleasure to announce the Dlang Statistical Package by John Michael Hall. API http://mir-stat.libmir.org/ GitHub http://github.com/libmir/mir-stat DUB https://code.dlang.org/packages/mir-stat The initial release provides descriptive statistics and algorithms for transforming data that are useful in statistical applications. The very basic stuff like `gmean` [1] is located in the mir-algorithm package, it will be downloaded automatically. The generation of random numbers of various distributions is provided by mir-random package [2]. --- libmir.org infrastructure supports cross-site links now. New documentation macro set: GREF GREF1 GREF_ALTTEXT GREF1_ALTTEXT The G* macros add an argument to the first position that should refer to the mir package name. Example: /++ Module header Macros: NDSLICEREF = $(GREF_ALTTEXT mir-algorithm, $(TT $2), $2, mir, ndslice, $1)$(NBSP) +/ /++ See_also: $(NDSLICEREF slice, Slice) +/ Slice!(double, 2) eye(size_t n); --- Have a good day! Ilya [1] http://mir-algorithm.libmir.org/mir_math_stat.html [2] http://mir-random.libmir.org/mir_random_variable.html
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 23:56:45 UTC, kinke wrote: On Tuesday, 7 July 2020 at 23:52:05 UTC, 9il wrote: On Tuesday, 7 July 2020 at 16:38:39 UTC, kinke wrote: So wouldn't the trivial 'fix' be using `strtod` for double literals and `strtof` for floats? [For LDC, we wouldn't rely on the host C runtime or a mir implementation, but use LLVM facilities anyway.] This should work if the C runtime handles the values correctly. I've just opened a PR for DMD (and LDC too): https://github.com/dlang/dmd/pull/11387 Does this actually mean DMD wouldn't be able to compile itself with DigigtalMars C runtime? Sorry, I don't understand. - I think this excess precision was at some point considered a feature (but probably only for D, not for DigitalMars C), there's even a test making sure 0.9L and 0.9 are parsed to the same compile-time value. DMC strtod [1] isn't IEEE compatible. Just nitpick. Unlikely it is used to compile DMD thought. [1] https://github.com/DigitalMars/dmc/blob/9a774f3f2b3227fd416ec3a83cb9eb8f8751425f/src/core/strtod.c
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 16:38:39 UTC, kinke wrote: On Saturday, 4 July 2020 at 20:35:48 UTC, Walter Bright wrote: On 6/21/2020 8:24 AM, 9il wrote: [...] Great work! Would you like to add it to dmd? AFAIU, the 'problem' is that *all* floating-point literals are parsed as real_t values, which for DMD is x87 real (usually using the host C runtime's `strtold`). When emitting them as double or float literals, the compiler converts these values to a lower precision, where the increased intermediate precision might break the 'banker's rule'. So wouldn't the trivial 'fix' be using `strtod` for double literals and `strtof` for floats? [For LDC, we wouldn't rely on the host C runtime or a mir implementation, but use LLVM facilities anyway.] This should work if the C runtime handles the values correctly. Does this actually mean DMD wouldn't be able to compile itself with DigigtalMars C runtime?
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 12:52:35 UTC, Adam D. Ruppe wrote: On Tuesday, 7 July 2020 at 07:49:02 UTC, Walter Bright wrote: Businesses will not want to commit to a balkanized project. It's been ages since I worked on a software project for a business that didn't have many random third (and fourth and fifth and sixth and seventh.) party dependencies. Trying to remove or avoid them would universally encounter pushback from management. "Don't reinvent the wheel" they say. It is really absurd. But anyway this whole debate is moot because if you like the code, you can simply copy/paste it (with attribution as required by Boost copyright of course) into your own files. You keep full control and get all the benefits of using it. This would be good advertising for DFL, haha.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 12:04:43 UTC, Steven Schveighoffer wrote: On 7/7/20 7:13 AM, 9il wrote: [...] Guys, this is all open source, all licensed identically. There are ways to solve this. Practically speaking, just because DMD depends on Mir, doesn't mean that Mir has control over how the dependency works. DMD can depend on a specific version of Mir, upgraded when reasonable (i.e. it should take a PR change to DMD for upgrading which code exactly is depended on) and if something changes in the future, you can fork it, or move back to using libc. This way, the code is only maintained in one place unless something catastrophic happens. In this sense, the DLF *does* control which code is used, as well as if it were in the DMD repository itself. We have a boost license for a reason. -Steve Exactly. Thank you
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 12:14:16 UTC, Guillaume Piolat wrote: On Tuesday, 7 July 2020 at 10:58:25 UTC, 9il wrote: From a business point of view, having slightly more correct string to float conversion holds very little value. I'll stick with sscanf thanks... For a high tech real markets (airspace, automotive, science, military-industrial complex) having a correct decimal literal parsing... Phobos is the stdlib of the language. Mir is not. Likewise, you've made the std.experimental.allocator on DUB depends on mir-core... stdlib shouldn't depend on non-stdlib, there isn't anything to debate on this point. Mir is stdlib for my business. Phobos is not. It wasn't me who proposed to use Mir code in DMD. I have heard Walter. There isn't anything to debate on this point.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 12:35:57 UTC, jmh530 wrote: On Tuesday, 7 July 2020 at 12:33:40 UTC, jmh530 wrote: On Tuesday, 7 July 2020 at 12:14:16 UTC, Guillaume Piolat wrote: [...] Is that not an example of how Steve thinks it should work? Both are Boost licensed. mir-core has no external dependencies. Ilya could split the parse/bignum packages to a separate repo with no other dependencies and then public import them in mir-algorithm and just normally import them elsewhere. [...] Walter's original comment was about adding it to DMD. He may have intended Phobos, but I read it as DMD. Eh, it looks like parse.d depends on std. So it wouldn't make sense to add to DMD since then the compiler would depend on the standard library. All std.* dependencies in Mir are minor and can be replaced in a day.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 07:49:02 UTC, Walter Bright wrote: On 7/5/2020 5:46 AM, Joseph Rushton Wakeling wrote: On Sunday, 5 July 2020 at 11:07:55 UTC, 9il wrote: There is no risk for DMD and DFL to depend on a Mir's Boost licensed library. If something happens with Mir or Mir change the license, DFL will be able to fork the required code at any point in the Boost licensed part of git history. Can't speak for Walter or the D foundation here, but I'm not sure the concern is really about licensing. It's about putting in place a required dependency on code where maintenance decisions are outside the hands of the D Foundation. That's right, it's not about the licensing. It's that the DLF should control the code it distributes. Businesses will not want to commit to a balkanized project. The proposal is for Mir to become a central required component of DMD and Phobos. This means it needs to become part of the D Language Foundation. These don't serve my business needs. DLF doesn't serve my business needs. DLF blocks the initiatives my business needs. For the current state of things being a part of DLF codebase for Mir is nonsense.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Tuesday, 7 July 2020 at 09:28:26 UTC, Guillaume Piolat wrote: On Tuesday, 7 July 2020 at 07:49:02 UTC, Walter Bright wrote: That's right, it's not about the licensing. It's that the DLF should control the code it distributes. Businesses will not want to commit to a balkanized project. From a business point of view, having slightly more correct string to float conversion holds very little value. I'll stick with sscanf thanks... For a high tech real markets (airspace, automotive, science, military-industrial complex) having a correct decimal literal parsing has a little but absolutely mandatory value. If SpaceX is lending a rocket, they want it located on the platform, something around wouldn't make sense. Note, that these companies hold a huge amount of the legacy C/C++ code and they are potential Dlang markets. But only if Dlang will be able to match C exactly for numeric code. Otherwise merging C/C++ code would have a huge negative impact on them.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Sunday, 5 July 2020 at 10:39:49 UTC, Walter Bright wrote: On 7/5/2020 3:35 AM, Walter Bright wrote: All of DMD, Druntime, and Phobos use Boost, except for Curl and the zip library (which we probably shouldn't have added). Also, there are no dependencies on Curl and zip. We don't distribute the C libraries, we use whatever is on the user's system. DMD statically links the C standard library (and maybe something else). There is no risk for DMD and DFL to depend on a Mir's Boost licensed library. If something happens with Mir or Mir change the license, DFL will be able to fork the required code at any point in the Boost licensed part of git history.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Sunday, 5 July 2020 at 08:15:53 UTC, Walter Bright wrote: It doesn't work quite like that. The D Language Foundation controls it. Andrei, Atila, and myself control it only as far as we DLF empowers us to, which can change. Official parts of the DMD distribution have to be controlled by the DLF. It's unworkable otherwise. If I remember correctly some time ago DMD hasn't been even Boost licensed. Also, DMD uses C libraries at least. I can't see why adding an open-source Boost licensed dependency is unworkable then.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Sunday, 5 July 2020 at 08:15:53 UTC, Walter Bright wrote: On 7/5/2020 12:24 AM, 9il wrote: On Sunday, 5 July 2020 at 06:23:35 UTC, Walter Bright wrote: On 7/4/2020 8:09 PM, 9il wrote: Does the float parsing code require bignum? Yes. The decimal float parsing requires big integer arithmetic and software Arbitrary precision or simply a fixed amount of more precision? Up to 2^^16384 - 1.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Sunday, 5 July 2020 at 06:23:35 UTC, Walter Bright wrote: On 7/4/2020 8:09 PM, 9il wrote: On Saturday, 4 July 2020 at 20:35:48 UTC, Walter Bright wrote: On 6/21/2020 8:24 AM, 9il wrote: So excited to finally announce we can correctly parse floating-point numbers according to IEEE round half-to-even (bankers) rule like in C/C++, Rust, and others. Great work! Would you like to add it to dmd? Thank you! Yes. It would be very much appreciated to preserve the `mir.` namespace for the `parse` module, the required `bignum` package*. Also, whenever the code will be located I would like to have control over it. Will you agree? * - `mir.bignum` is 6K LOC and it is expected to grow up to 20K LOC if finished. The package includes abstract views for big integers, decimal, and binary FP numbers; stack-allocated big integers; midsize unsigned integers; software FP numbers with extended precision. Does the float parsing code require bignum? Yes. The decimal float parsing requires big integer arithmetic and software floating-point multiplication with extended precision (128-bit mantissa). I'm also not sure I know what you mean by control. Contributions to dmd would need to be Boost Licensed, which means anyone can do what they like with them. The code is already Boost licensed. We need not only literals parsing but also library text parsing. So the code should be available for users and for the compiler. I see two possible solutions that look good to me. The first one is to add mir-algorithm package or its part as an external dependency for DMD. It is preferable and either way. If you will accept the PR, I will do it. The second solution is to move `mir.bignum` and `mir.parse` to DRuntime/Phobos. In this case, I would like to preserve the `mir.` namespace and the same authority and veto right for this part of the codebase as I have at Mir Org. I mean the following. Your voice has a veto right for DMD and Dlang evaluation. Andrei has a veto right for Phobos. Atila seems to have almost the same veto right as Andrei and you and blocks required Dlang features for Mir [1, 2]. Furthermore, if you and Andrei really want to add or change something you will force it to happen. I want the same veto right for evaluation of the Mir parts in case you think they should be moved to DRuntime/Phobos. Also, the code under `mir.` namespace should be less constrained then `core`/`std` code in terms of API changes. [1] https://github.com/dlang/dmd/pull/9778 [2] https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1023.md
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Saturday, 4 July 2020 at 20:35:48 UTC, Walter Bright wrote: On 6/21/2020 8:24 AM, 9il wrote: So excited to finally announce we can correctly parse floating-point numbers according to IEEE round half-to-even (bankers) rule like in C/C++, Rust, and others. Great work! Would you like to add it to dmd? Thank you! Yes. It would be very much appreciated to preserve the `mir.` namespace for the `parse` module, the required `bignum` package*. Also, whenever the code will be located I would like to have control over it. Will you agree? * - `mir.bignum` is 6K LOC and it is expected to grow up to 20K LOC if finished. The package includes abstract views for big integers, decimal, and binary FP numbers; stack-allocated big integers; midsize unsigned integers; software FP numbers with extended precision.
Re: From the D Blog: A Pattern for Head-mutable Structures
On Thursday, 25 June 2020 at 11:55:14 UTC, Mike Parker wrote: Simen Kjærås outlines an approach to supporting head-mutable types in D without the need for compiler or language changes. The blog: https://dlang.org/blog/2020/06/25/a-pattern-for-head-mutable-structures/ Reddit: https://www.reddit.com/r/programming/comments/hfkq5e/a_pattern_for_headmutable_structures_in_d/ I've also submitted it to HN (please use the search box): https://news.ycombinator.com/newest Good stuff. If this pattern will be accepted to language design the same way as Ranges API then Mir will adopt it to extend the current lightConst/lightImmutable features. Ilya
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Monday, 22 June 2020 at 12:04:13 UTC, 9il wrote: On Monday, 22 June 2020 at 10:53:02 UTC, Dukc wrote: On Sunday, 21 June 2020 at 15:24:14 UTC, 9il wrote: Can mir_parse handle other bases than decimal? No, only the decimal basis is supported for now. Support for hexadecimal FP/integer parsing can be added though. The basic stuff for correct FP hexadecimal parsing is done: we can convert a big integer view to FP number with half-to-even rounding. So the algorithm would look like: 1. Parse hexadecimal big integer 2. Parse exponent 3. Cast big integer to `Fp` with a specific number of meaningful bits (its already implemented) 4. Add exponent to `Fp`'s exponent, and cast the result to a hardware floating point type. My bad, the hexadecimal parsing is already implemented for big integers! http://mir-algorithm.libmir.org/mir_bignum_low_level_view.html#.BigUIntView.fromHexStringImpl So, each part of the algorithm above is implemented. Maybe we need to rework fromHexStringImpl to make it return a boolean value.
Re: Decimal string to floating point conversion with correct half-to-even rounding
On Monday, 22 June 2020 at 10:53:02 UTC, Dukc wrote: On Sunday, 21 June 2020 at 15:24:14 UTC, 9il wrote: Can mir_parse handle other bases than decimal? No, only the decimal basis is supported for now. Support for hexadecimal FP/integer parsing can be added though. The basic stuff for correct FP hexadecimal parsing is done: we can convert a big integer view to FP number with half-to-even rounding. So the algorithm would look like: 1. Parse hexadecimal big integer 2. Parse exponent 3. Cast big integer to `Fp` with a specific number of meaningful bits (its already implemented) 4. Add exponent to `Fp`'s exponent, and cast the result to a hardware floating point type.
Decimal string to floating point conversion with correct half-to-even rounding
Hey everyone, So excited to finally announce we can correctly parse floating-point numbers according to IEEE round half-to-even (bankers) rule like in C/C++, Rust, and others. @nogc, optionally nothrow API is provided as part of Mir Algorithm v3.9.0 [0]. The documentation is available [1]. In case you are surprised, you can be sure that neither D compilers can correctly parse decimal FP literals [2, 3, 4], nor Phobos can correctly parse decimal FP strings [6]. Mir decimal parsing supports up-to quadruple precision. The conversion error is 0 ULP for normal numbers. Subnormal numbers with a decimal exponent greater than or equal to -512 have upper error bound equal to 1 ULP. Zero error bound for subnormal numbers can be supported in the future when Mir Ion, the ASDF successor, is ready. The implementation is based on the paper [7]. The error bounds above are valid for LDC. DMD may have slightly larger errors because of the wrong code generation for ulong to double conversion [5]. This work has been sponsored by Symmetry Investments and Kaleidic Associates. Best regards, Ilya [0] https://github.com/libmir/mir-algorithm [1] http://mir-algorithm.libmir.org/mir_parse.html#.fromString[2] https://issues.dlang.org/show_bug.cgi?id=20951 [3] https://issues.dlang.org/show_bug.cgi?id=20952 [4] https://issues.dlang.org/show_bug.cgi?id=20953 [5] https://issues.dlang.org/show_bug.cgi?id=20963 [6] https://issues.dlang.org/show_bug.cgi?id=20967 [7] https://www.researchgate.net/publication/2295884_How_to_Read_Floating_Point_Numbers_Accurately
Re: On the D Blog: A Looat at Chapel, D, and Julia Using Kernel Matrix Calculations
On Wednesday, 3 June 2020 at 16:15:41 UTC, jmh530 wrote: This is unclear: The chart below shows matrix implementation times minus ndslice times; negative means that ndslice is slower, indicating that the implementation used here does not negatively represent D’s performance. hmm, ndslice isn't slower on my machine. https://github.com/dataPulverizer/KernelMatrixBenchmark/pull/2 https://github.com/dataPulverizer/KernelMatrixBenchmark/issues/3
Re: On the D Blog: A Looat at Chapel, D, and Julia Using Kernel Matrix Calculations
On Wednesday, 3 June 2020 at 16:15:41 UTC, jmh530 wrote: This is unclear: The chart below shows matrix implementation times minus ndslice times; negative means that ndslice is slower, indicating that the implementation used here does not negatively represent D’s performance. This means that ndslice (in the way how it was used here) is slower than the custom matrix type used.
Re: On the D Blog: A Looat at Chapel, D, and Julia Using Kernel Matrix Calculations
On Wednesday, 3 June 2020 at 14:34:02 UTC, Mike Parker wrote: Some of you may have seen a draft of this post from user "data pulverizer" elsewhere on the forums. The final draft is now on the D Blog under his real name and ready for your perusal. The blog: https://dlang.org/blog/2020/06/03/a-look-at-chapel-d-and-julia-using-kernel-matrix-calculations/ Reddit: https://www.reddit.com/r/programming/comments/gvuy59/a_look_at_chapel_d_and_julia_using_kernel_matrix/ I'll be posting on HN, too, but please don't share a direct link. I did some digging around and it really does affect the ranking -- your upvotes won't count. Please fix the link http://docs.algorithm.dlang.io/latest/mir_ndslice.html to http://mir-algorithm.libmir.org/ docs.algorithm.dlang.io is outdated.
Tensorflow wrapper for D
by Shigeki Karita https://github.com/ShigekiKarita/tfd
Re: DDOC generator Harbored-Mod - v0.3.4
On Saturday, 25 April 2020 at 06:50:07 UTC, Basile B. wrote: [1] https://gitlab.com/basile.b/harbored-mod Note that I don't have access to the DUB registry to update the location so Ilya Y. if you read this maybe you can do that [2] ;). [2] https://code.dlang.org/packages/harbored-mod done
Mir Ref-Counted Type System for .NET
mir.net https://github.com/libmir/mir.net Without docs for now. Feel free to open an issue if you have a question. Ilya --- This work has been sponsored by Symmetry Investments and Kaleidic Associates
mir-algorithm v3.7.30: mir.date, squeeze, unsqueeze
mir.date [1] is a rework of the Phobos Date type with the following features: 1. ABI and mangling compatibility with Boost's date type 2. BetterC compatible 3. ASDF (JSON) (de)serialization support 4. reworked all (to/from)String methods, added nothrow and @nogc versions 5. `toString` is the alias for `toISOExtString` instead of `toSimpleString` 6. Added universal `fromString` that parses ISO, Extended ISO, and Simple date strings. `squeeze` and `unsqueeze` has been added to mir.ndslice.topology [2] [1] http://mir-algorithm.libmir.org/mir_date.html [2] http://mir-algorithm.libmir.org/mir_ndslice_topology.html --- This work has been sponsored by Symmetry Investments and Kaleidic Associates.
mir-algorithm v3.7.30: mir.date, squeeze, unsqueeze
mir.date [1] is a rework of the Phobos Date type with the following features: 1. ABI and mangling compatibility with Boost's date type 2. BetterC compatible 3. ASDF (JSON) (de)serialization support 4. reworked all (to/from)String methods, added nothrow and @nogc versions 5. `toString` is the alias for `toISOExtString` instead of `toSimpleString` 6. Added universal `fromString` that parses ISO, Extended ISO, and Simple date strings. `squeeze` and `unsqueeze` has been added to mir.ndslice.topology [2] [1] http://mir-algorithm.libmir.org/mir_date.html [2] http://mir-algorithm.libmir.org/mir_ndslice_topology.html --- This work has been sponsored by Symmetry Investments and Kaleidic Associates.
Re: Mir updates
On Thursday, 2 April 2020 at 13:39:41 UTC, jmh530 wrote: On Thursday, 2 April 2020 at 13:16:46 UTC, 9il wrote: [snip] For my work, I use `mir-blas` - ndslice bindings to CBLAS API. It is faster to write binding rather than finish BLAS implementation. It is useful. And I think it is very promising for Dlang promotion and can really involve new companies. I will be happy to finish it if a company sponsors the work. Companies like CPU/GPU vendors or companies that require slim and fast BLAS implementation will win from this work. However, I have no idea how to sell mir-glas, it is more complex to find a client rather than finish it. Can mir-glas be used as a replacement blas implementation for mir-lapack? Yes (when finished). However, mir-lapack would need to link with a LAPACK library. For example, OpenBLAS can be compiled with LAPACK as a solid library. We can do the same. Actually, when the library is finished it can be available in the dub registry as well as it should be packed for Linux and windows as a standalone BLAS/LAPACK library with common API.
Re: Mir updates
On Thursday, 2 April 2020 at 06:20:22 UTC, Manu wrote: On Tue, Mar 31, 2020 at 12:15 AM 9il via Digitalmars-d-announce < digitalmars-d-announce@puremagic.com> wrote: On Monday, 30 March 2020 at 12:23:03 UTC, jmh530 wrote: > On Monday, 30 March 2020 at 06:33:13 UTC, 9il wrote: >> [snip] > > Thanks, I like 'em. > > I noticed that the little icon in the tabs has changed from > most of them. However, the mir random is unchanged from > before. Probably it is because of your browser cache, likely will be updated after a while. > Also, on the mir.glas page, one of the lines says > "matrix-vector operations %3 done, partially optimized for > now". Another line says "l3 was moved to mir-glas", which is > confusing because it should be at the mir-glas documentation > page anyway. We don't have documentation for mir-glas library, only for mir (backports) package, which has mir.glas package. I don't know what to do with mir-glas, it is too good to be forgotten, but I don't see a commercial perspective in it. Why not? Where does it fall short of being useful? For my work, I use `mir-blas` - ndslice bindings to CBLAS API. It is faster to write binding rather than finish BLAS implementation. It is useful. And I think it is very promising for Dlang promotion and can really involve new companies. I will be happy to finish it if a company sponsors the work. Companies like CPU/GPU vendors or companies that require slim and fast BLAS implementation will win from this work. However, I have no idea how to sell mir-glas, it is more complex to find a client rather than finish it.
Re: Mir simple linear
On Monday, 30 March 2020 at 17:21:57 UTC, Vino wrote: Hi All, Can anyone guide me what is the problem with below code as it throws error. import std.stdio, mir.math.common: approxEqual; static immutable x = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]; static immutable y = [1, 3, 2, 5, 7, 8, 8, 9, 10, 12]; auto params = x.simpleLinearRegression(y); writeln(params); static immutable y = [1.0, 3, 2, 5, 7, 8, 8, 9, 10, 12]; `.0` will change the sum type of y to double.
Re: Mir updates
On Monday, 30 March 2020 at 12:23:03 UTC, jmh530 wrote: On Monday, 30 March 2020 at 06:33:13 UTC, 9il wrote: [snip] Thanks, I like 'em. I noticed that the little icon in the tabs has changed from most of them. However, the mir random is unchanged from before. Probably it is because of your browser cache, likely will be updated after a while. Also, on the mir.glas page, one of the lines says "matrix-vector operations %3 done, partially optimized for now". Another line says "l3 was moved to mir-glas", which is confusing because it should be at the mir-glas documentation page anyway. We don't have documentation for mir-glas library, only for mir (backports) package, which has mir.glas package. I don't know what to do with mir-glas, it is too good to be forgotten, but I don't see a commercial perspective in it.
Re: Mir updates
On Monday, 30 March 2020 at 06:33:13 UTC, 9il wrote: Docs and links have been updated for http://mir-algorithm.libmir.org, Mir Algorithm http://mir-core.libmir.org, Mir Core http://mir-random.libmir.org, Mir Random http://mir-optim.libmir.org, Mir Optim http://mir-cpuid.libmir.org, Mir CPUID http://mir-blas.libmir.org, Mir BLAS http://mir-lapack.libmir.org, Mir LAPACK http://mir.libmir.org, Mir Backport http://asdf.libmir.org, ASDF - JSON library and http://mir-integral.libmir.org/
Mir updates
Docs and links have been updated for http://mir-algorithm.libmir.org, Mir Algorithm http://mir-core.libmir.org, Mir Core http://mir-random.libmir.org, Mir Random http://mir-optim.libmir.org, Mir Optim http://mir-cpuid.libmir.org, Mir CPUID http://mir-blas.libmir.org, Mir BLAS http://mir-lapack.libmir.org, Mir LAPACK http://mir.libmir.org, Mir Backport http://asdf.libmir.org, ASDF - JSON library I have started this a year ago and recently I got time to finish this work. All of the libraries above have: 1. Meson and dub support 2. Imports validation for documented examples 3. AWS S3 + Circle CI automated documentation updates Mir Optim was completely reworked and got Box Constrained QP Solver ASDF, mir-algorithm, mir-lapack, mir-random, mir-core have been extended --- This work has been sponsored by Symmetry Investments and Kaleidic Associates. http://symmetryinvestments.com/ https://github.com/kaleidicassociates
mir-integral: gauss quadratures for integration
Hey folks, Just a small betterC mir.quadrature module - Gauss-Hermite Quadrature - Gauss-Jacobi Quadrature - Gauss-Laguerre Quadrature - Gauss-Legendre Quadrature They should be enough to implement a dozen integration algorithms. MRs, feature requests, and GSoC projects with integration related stuff are welcome. https://github.com/libmir/mir-integral - This work has been sponsored by Symmetry Investment and Kaleidic Associates
Re: DMD 32 bit Linux code gen now uses XMM for float & double
On Tuesday, 28 May 2019 at 01:23:18 UTC, Walter Bright wrote: 32 Bit Linux now uses XMM registers for float and double rather than the x87. This should substantially speed up routine float and double processing. SIMD vector operations, however, are still not support on 32 bit Linux code because of issues with 16 byte stack alignment. This means that generated code will no longer work on older x86 processors that do not have XMM registers. If this is an issue, please file a bug report. https://github.com/dlang/dmd/pull/9895 DMD always had that capability, it just wasn't enabled for backwards compatibility with older processors. Thanks a lot. I expect it resolves some ancient issues I had with math functions when compiled with DMD. Does DMD still use x87 for float/double on other targets?
Re: dlang.ru is updated
On Saturday, 18 May 2019 at 09:19:24 UTC, 9il wrote: On Saturday, 18 May 2019 at 06:26:10 UTC, Suliman wrote: On Saturday, 18 May 2019 at 03:01:33 UTC, 9il wrote: On Thursday, 16 May 2019 at 12:25:52 UTC, Suliman wrote: [...] Bullshit! I have checked the dlang.ru at RKN [1] registry and there are no dlang.ru in it. I checked with two providers from Russia and access works. Don't bug the forum with provocations or simply wait for DNS update. [1] - https://eais.rkn.gov.ru/ It was banned by IP. I checked it two weeks ago. It work very strange even from single mobile providers. https://imgur.com/a/NN3Apfo Well, yes this IP 37.252.127.244 is blocked for me and the checker [1] shows it belongs to dlang.ru (however the site itself works well). This was done by the court's decision [2] at 13.04.2018, more than a year ago. The information can be checked at [3]. You picked an already blocked IP. [1] https://ipinfo.info/html/ip_checker.php [2] https://www.mos-gorsud.ru/rs/taganskij/services/cases/civil/details/2cc72aea-39e7-4f8e-adc9-37d170966efa [3] http://blocklist.rkn.gov.ru/ Or, if you picked it a long time ago, then probably they blocked a Europe's subnet with this IP (because of telegram, games or drugs). However, nothing in this information says that they wanted to block dlang.ru. Placing the site at a new IP will solve the problem.
Re: dlang.ru is updated
On Saturday, 18 May 2019 at 06:26:10 UTC, Suliman wrote: On Saturday, 18 May 2019 at 03:01:33 UTC, 9il wrote: On Thursday, 16 May 2019 at 12:25:52 UTC, Suliman wrote: After 2 years dlang.ru was update. Content did not change. Main improves was is technology stack and design (still not perfect, but better than was). http://dlang.ru P.S. site is blocked by most of russian internet-providers by RKN Bullshit! I have checked the dlang.ru at RKN [1] registry and there are no dlang.ru in it. I checked with two providers from Russia and access works. Don't bug the forum with provocations or simply wait for DNS update. [1] - https://eais.rkn.gov.ru/ It was banned by IP. I checked it two weeks ago. It work very strange even from single mobile providers. https://imgur.com/a/NN3Apfo Well, yes this IP 37.252.127.244 is blocked for me and the checker [1] shows it belongs to dlang.ru (however the site itself works well). This was done by the court's decision [2] at 13.04.2018, more than a year ago. The information can be checked at [3]. You picked an already blocked IP. [1] https://ipinfo.info/html/ip_checker.php [2] https://www.mos-gorsud.ru/rs/taganskij/services/cases/civil/details/2cc72aea-39e7-4f8e-adc9-37d170966efa [3] http://blocklist.rkn.gov.ru/
Re: LDC 1.16.0-beta1
On Thursday, 9 May 2019 at 21:14:02 UTC, kinke wrote: Glad to announce the first beta for LDC 1.16; mainly just an upgrade to D 2.086.0. Full release log and downloads: https://github.com/ldc-developers/ldc/releases/tag/v1.16.0-beta1 Please help test, and thanks to all contributors! Many thanks!
Re: Mir Algorithm 3.4.1 - RCArray and RCPtr
On Wednesday, 24 April 2019 at 10:52:14 UTC, jmh530 wrote: On Wednesday, 24 April 2019 at 01:34:58 UTC, 9il wrote: Thread safe RC Array and Ptr. Plus C++ headers for code integration. [snip] Cool. Does this make any use of DIP1000? How is the run-time/memory performance vs. the GC versions? RC types are created to be used with DIP1000. Plus, Mir Algorithm used in production with this DIP. See configuration "dips" [1] Well, the allocator support is not ready yet. But the mir_rc_context already contains `void* allocator` that will be replaced in the future with nothrow @nogc allocator interface (maybe this or next year). For now, Mir RC types are like a C++ shared_ptr without allocators and with the same performance. 1. https://github.com/libmir/mir-algorithm/blob/master/dub.sdl#L26
Mir Algorithm 3.4.1 - RCArray and RCPtr
Thread safe RC Array and Ptr. Plus C++ headers for code integration. https://github.com/libmir/mir-algorithm Docs: http://mir-algorithm.libmir.org http://mir-algorithm.libmir.org/mir_rc_array.html http://mir-algorithm.libmir.org/mir_rc_ptr.html Also, Ndslice and Series RC wrappers are available: http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#.rcslice http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#.mininitRcslice http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#.bitRcslice http://mir-algorithm.libmir.org/mir_series.html#.rcseries http://mir-algorithm.libmir.org/mir_series.html#.rcTroykaSeries http://mir-algorithm.libmir.org/mir_series.html#.rcUnionSeries mir.qualifier from mir-core library contains useful utilities to work with RC and/or const data. 1. lightScope - returns a rc-free view of a Slice/Series. For example, for Slice!(RCI!double) it returns Slice!(double*) 2. lightConst - returns a mutable Slice/Series view to constant data. In terms of Phobos: it converts a constant range structure to a mutable range structure with constant elements. 3. lightImmutable - the same for immutable qualifier DMD related regression: https://issues.dlang.org/show_bug.cgi?id=19774 and a new bug: https://issues.dlang.org/show_bug.cgi?id=19793 This work has been sponsored by Symmetry Investments and Kaleidic Associates. http://symmetryinvestments.com/ https://github.com/kaleidicassociates/ Best, Ilya
Re: ldexp and frexp benchmark between Mir, C and Phobos
On Saturday, 29 December 2018 at 15:15:48 UTC, Iain Buclaw wrote: On Fri, 28 Dec 2018 at 20:50, 9il via Digitalmars-d-announce wrote: ldexp and frexp are base building blocks for a lot of math functions. Here is a small benchmark that compares Mir, C and Phobos implementations: https://github.com/libmir/mir-core/blob/master/bench_ldexp_frexp.d Mir ldexp is 2.5 (5.5 - dmd) times faster for double and float. You could double the speed of ldexp if you actually used the checkedint compiler intrinsics rather than implementing it yourself. Using libm's ldexp() is also likely going to be 2-5x slower than using the implementation you've written for mir.ldexp(). For one, your version will be inlined! Mir has support for LLVM checkedint intrinsics. GDC checkedint intrinsics are not yet integrated in Mir. https://github.com/libmir/mir-core/blob/master/source/mir/checkedint.d
Re: ldexp and frexp benchmark between Mir, C and Phobos
On Saturday, 29 December 2018 at 12:35:03 UTC, kinke wrote: On Friday, 28 December 2018 at 19:48:28 UTC, 9il wrote: Any chance the multi-precision ldexp can be upstreamed to Phobos (which currently uses real precision for the float/double overloads, which explains the suboptimal performance)? It'd make a *lot* more sense there, instead of having it in a separate library. It's well-known that there's a lot of remaining std.math functions which need proper single/double precision implementations, and ldexp is one of them. Yes, Mir is Boost licensed, but I don't work on Phobos anymore. Mir libraries are going to be independent of Phobos.
ldexp and frexp benchmark between Mir, C and Phobos
ldexp and frexp are base building blocks for a lot of math functions. Here is a small benchmark that compares Mir, C and Phobos implementations: https://github.com/libmir/mir-core/blob/master/bench_ldexp_frexp.d Mir ldexp is 2.5 (5.5 - dmd) times faster for double and float. = LDC, macos x64: --- float ldexp (Phobos time / Mir time) = 2.55584 ldexp ( stdc time / Mir time) = 0.773019 frexp (Phobos time / Mir time) = 1.04093 frexp ( stdc time / Mir time) = 1.748 --- double ldexp (Phobos time / Mir time) = 2.49162 ldexp ( stdc time / Mir time) = 1.31868 frexp (Phobos time / Mir time) = 0.937906 frexp ( stdc time / Mir time) = 1.82241 --- real ldexp (Phobos time / Mir time) = 0.999327 (LDC Phobos uses C func for real) ldexp ( stdc time / Mir time) = 0.969467 (LDC Mir uses C func for real) frexp (Phobos time / Mir time) = 1.02512 frexp ( stdc time / Mir time) = 1.77901 = DMD, macos x64: --- float ldexp (Phobos time / Mir time) = 5.53172 ldexp ( stdc time / Mir time) = 0.535711 frexp (Phobos time / Mir time) = 2.06024 frexp ( stdc time / Mir time) = 0.739571 --- double ldexp (Phobos time / Mir time) = 5.32189 ldexp ( stdc time / Mir time) = 0.772949 frexp (Phobos time / Mir time) = 2.02758 frexp ( stdc time / Mir time) = 0.637328 --- real ldexp (Phobos time / Mir time) = 2.61905 ldexp ( stdc time / Mir time) = 0.803806 frexp (Phobos time / Mir time) = 1.22398 frexp ( stdc time / Mir time) = 1.08659 Best, Ilya This work has been sponsored by Symmetry Investments and Kaleidic Associates. http://symmetryinvestments.com/ https://github.com/kaleidicassociates/
RCArray for D & C++ - mir-algorithm v3.1.12
Thread safe reference counted RCArray with DIP1000 in mind has been released with mir-algorithm v3.1.12. It has well defined C++ header. You can use it from C++ if the same type is used in linked D object file. D's RCArray (1) (mir_rcarray) does not have range primitives, only length, [], and [i] primitives. [] returns a common scoped slice view (common D array). C++'s mir_rcarray (2) defines [i], at, begin, cbegin, end, cend, and data primitives. Both D and C++ versions define asSlice method that returns ndslice view on top of reference counted iterator. See also the example D++ interaction (5) D++ interaction now tested in Travis CI. GCC may not be compatible with LDC, CLANG interacts well. This work has been sponsored by Symmetry Investments (3) and Kaleidic Associates (4). 1. https://github.com/libmir/mir-algorithm/blob/master/source/mir/rcarray.d 2. https://github.com/libmir/mir-algorithm/blob/master/include/mir/rcarray.h 3. http://symmetryinvestments.com/ 4. https://github.com/kaleidicassociates/ 5. https://github.com/libmir/mir-algorithm/tree/master/cpp_example Ilya
Re: usable @nogc Exceptions with Mir Runtime
On Friday, 2 November 2018 at 07:00:49 UTC, Manu wrote:
On Tue, Oct 30, 2018 at 9:30 AM Oleg via Digitalmars-d-announce
wrote:
Thanks for your work!
> Example
> ===
> ///
> @safe pure nothrow @nogc
> unittest
> {
> import mir.exception;
> import mir.format;
> try throw new MirException(stringBuf() << "Hi D" << 2 <<
> "!" << getData);
> catch(Exception e) assert(e.msg == "Hi D2!");
> }
>
> ===
I don't understand why you choose C++ format style instead of
D-style format?
Perhaps this is a stupid question... but there's clearly `new
MirException` right there in that code.
How is this @nogc?
The code requires -dip1008 flag. Take a look into the DIP
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1008.md
