Re: fft and isPowerOf2?
On Thursday, 17 May 2018 at 13:36:39 UTC, kdevel wrote: On Thursday, 17 May 2018 at 12:34:25 UTC, Andre Pany wrote: this applications throws an error in std.numeric (Line 2826). => assert(isPowerOf2(range.length)); Isn't it possible to give an arbitrary length of data to fft like in numpy? There is a mathematical background which is well explained here: https://math.stackexchange.com/questions/77118/non-power-of-2-ffts#77152 Thanks a lot for the details link. Kind regards André
Re: Splitting up large dirty file
On Thursday, May 17, 2018 21:10:35 Dennis via Digitalmars-d-learn wrote: > On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: > > For various reasons, that doesn't always hold true like it > > should, but pretty much all of Phobos is written with that > > assumption and will generally throw an exception if it isn't. > > It's unfortunate that Phobos tells you 'there's problems with the > encoding' without providing any means to fix it or even diagnose > it. The UTFException doesn't contain what the character in > question was. You just have to abort whatever you were trying to > do. UTFException has a sequence member and a len member (which appear to be public but undocumented) which should contain the invalid sequence of code units. In general though, exceptions aren't a great way to deal with this problem. I think that you either want to be calling decode manually (in which case, you have direct access to where the invalid Unicode is and have the freedom to deal with it however is appropriate), or using the Unicode replacement character would be better (which std.utf.decode supports, but it's not what's used by default). Really, what's byting you here is the auto-decoding. With Phobos, you have to fight to have it not happen by doing stuff like special-casing your code for strings or using std.string.representation or std.utf.byCodeUnit. In principle, the way that Unicode would ideally be handled would be to validate all character data when it enters the program (soing whatever is appropriate with invalid Unicode at that point), and then the rest of the program then either is always dealing with valid Unicode, or it's dealing with integral values that it doesn't treat as Unicode (e.g. ubyte[]). But the way that Phobos is written, it ends up decoding and validating all over the place. > On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: > > If you're ever dealing with a different encoding (or with > > invalid Unicode), you really need to use integral types like > > ubyte > > I tried something like byChunk(4096).joiner.splitter(cast(ubyte) > '\n') but it turns out splitter wants at least a forward range, > even when the separator is a single element. Actually, I'm pretty sure that splitter curently requires a random-access range (even though it should theoretically work with a forward range). I don't think that it can be made to work with an input range though given how the range API works - or at least, it were made to work with it, you'd have to deal with the fact that popping front on the spitter range would invalidate anything that had been returned from front. And it would be difficult to implement it @safely if what gets returned by front is not completely independent of the splitter range (which means that it needs save). Basic input ranges in general tend to be extremely limited in what they can do, which can get really annoying when you deal with stuff like files or sockets where making it a forward range likely means either reading it all into memory or having buffers that potentially have to be dup-ed by each call to save. - Jonathan M Davis
Re: Template instantiation fails on Linux, succeeds on Windows
On Thursday, 17 May 2018 at 23:18:32 UTC, Basile B. wrote: On Thursday, 17 May 2018 at 22:07:46 UTC, Bastiaan Veelo wrote: Hi! The code in [1] compiles and runs flawlessly on Windows, but not on Linux (neither run.dlang nor Travis docker image). Any idea what can be done? Hello. Yes, add `import core.stdc.stdarg;` in your module and it works. I don't know why it's not required on windows, this is strange. Great! Thanks a lot, Basile! Seems there is room for improvement somewhere, a better error message at the least.
Re: Splitting up large dirty file
On Thursday, 17 May 2018 at 20:08:09 UTC, Dennis wrote:
On Wednesday, 16 May 2018 at 15:47:29 UTC, Jon Degenhardt wrote:
If you write it in the style of my earlier example and use
counters and if-tests it will work. byLine by itself won't try
to interpret the characters (won't auto-decode them), so it
won't trigger an exception if there are invalid utf-8
characters.
When printing to stdout it seems to skip any validation, but
writing to a file does give an exception:
```
auto inputStream = (args.length < 2 || args[1] == "-") ?
stdin : args[1].File;
auto outputFile = new File("output.txt");
foreach (line; inputStream.byLine(KeepTerminator.yes))
outputFile.write(line);
```
std.exception.ErrnoException@C:\D\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(2877):
(No error)
According to the documentation, byLine can throw an
UTFException so relying on the fact that it doesn't in some
cases doesn't seem like a good idea.
Instead of:
auto outputFile = new File("output.txt");
try:
auto outputFile = File("output.txt", "w");
That works for me. The second arg ("w") opens the file for write.
When I omit it, I also get an exception, as the default open mode
is for read:
* If file does not exist: Cannot open file `output.txt' in mode
`rb' (No such file or directory)
* If file does exist: (Bad file descriptor)
The second error presumably occurs when writing.
As an aside - I agree with one of your bigger picture
observations: It would be preferable to have more control over
utf-8 error handling behavior at the application level.
Re: Template instantiation fails on Linux, succeeds on Windows
On Thursday, 17 May 2018 at 22:07:46 UTC, Bastiaan Veelo wrote: Hi! The code in [1] compiles and runs flawlessly on Windows, but not on Linux (neither run.dlang nor Travis docker image). Any idea what can be done? Hello. Yes, add `import core.stdc.stdarg;` in your module and it works. I don't know why it's not required on windows, this is strange.
Re: Splitting up large dirty file
On 05/17/2018 11:40 PM, Neia Neutuladh wrote: 0b1100_ through 0b_1110 is the start of a multibyte character Nitpick: It only goes up to 0b_0100. The highest code point is U+10. There are no sequences with more than four bytes.
Template instantiation fails on Linux, succeeds on Windows
Hi! The code in [1] compiles and runs flawlessly on Windows, but not on Linux (neither run.dlang nor Travis docker image). Any idea what can be done? errors: Error: undefined identifier __va_list_tag onlineapp.d(282): Error: template instance `onlineapp.SetFactory!byte` error instantiating Error: undefined identifier __va_list_tag onlineapp.d(288): Error: template instance `onlineapp.SetFactory!(Count)` error instantiating Error: undefined identifier __va_list_tag onlineapp.d(290): Error: template instance `onlineapp.SetFactory!char` error instantiating SetFactory is defined on row 233, and it seems that the problem relates to the vararg opIndex. [1] https://run.dlang.io/gist/6dcbe4e82846cbeb3bf478e92a2b6e6f
Re: Splitting up large dirty file
On Tuesday, 15 May 2018 at 20:36:21 UTC, Dennis wrote: I have a file with two problems: - It's too big to fit in memory (apparently, I thought 1.5 Gb would fit but I get an out of memory error when using std.file.read) Memory mapping should work. That's in core.sys.posix.sys.mman for Posix systems, and Windows has some equivalent probably. (But nobody uses Windows, right?) - It is dirty (contains invalid Unicode characters, null bytes in the middle of lines) std.algorithm should generally work with sequences of anything, not just strings. So memory map, cast to ubyte[], and deal with it that way? - When you convert chunks to arrays, you have the risk of a split being in the middle of a character with multiple code units It's straightforward to scan for the start of a Unicode character; you just skip past characters where the highest bit is set and the next-highest is not. (0b1100_ through 0b_1110 is the start of a multibyte character; 0b_ through 0b0111_ is a single-byte character.) That said, you seem to only need to split based on a newline character, so you might be able to ignore this entirely, even if you go by chunks.
Re: Splitting up large dirty file
On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: For various reasons, that doesn't always hold true like it should, but pretty much all of Phobos is written with that assumption and will generally throw an exception if it isn't. It's unfortunate that Phobos tells you 'there's problems with the encoding' without providing any means to fix it or even diagnose it. The UTFException doesn't contain what the character in question was. You just have to abort whatever you were trying to do. On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: If you're ever dealing with a different encoding (or with invalid Unicode), you really need to use integral types like ubyte I tried something like byChunk(4096).joiner.splitter(cast(ubyte) '\n') but it turns out splitter wants at least a forward range, even when the separator is a single element.
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 20:38:13 UTC, Sjoerd Nijboer wrote: But then how do you put this into a mixin template so I can ... mixin castingRules(typeof(this) T); I guess I can refine my question to "How do you let a mixin template detect the template name it is instantiated with and return its type whitout the template arguments.
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 16:27:48 UTC, Paul Backus wrote:
On Thursday, 17 May 2018 at 15:25:37 UTC, Sjoerd Nijboer wrote:
I want to make a template mixin that is able to cast one of
these generic structs to the other explicitly. I have a bunch
of these structs and therefore I thought it would make sense
to do it by template mixin. I just don't know what language or
library features I need to use and how to apply them.
It sounds like you want to overload opCast:
https://dlang.org/spec/operatoroverloading.html#cast
Something like:
`
Foo!T opCast(U)(Foo!U old)
if (isImplicitlyConvertible!(T, U))
{
return Foo!T(old.t);
}
Foo!T opCast(UTemplate, U)(UTemplate!U old)
if (isImplicitlyConvertible!(T, U) && isNumerical!U)
{
return Foo!T(old.t);
}
`
But then how do you put this into a mixin template so I can write
something like
`
struct Foo(T)
if(isNumerical(T))
{
mixin castingRules(typeof(this) T);
}
`
Re: there's no gdc for windows?
On Tuesday, 15 May 2018 at 14:25:31 UTC, Dr.No wrote: Has gdc been supported for Windows? if so, where can I find it? I've only find Linux versions so far... ldc has good Windows support: https://github.com/ldc-developers/ldc/releases
Re: Splitting up large dirty file
On Wednesday, 16 May 2018 at 15:47:29 UTC, Jon Degenhardt wrote:
If you write it in the style of my earlier example and use
counters and if-tests it will work. byLine by itself won't try
to interpret the characters (won't auto-decode them), so it
won't trigger an exception if there are invalid utf-8
characters.
When printing to stdout it seems to skip any validation, but
writing to a file does give an exception:
```
auto inputStream = (args.length < 2 || args[1] == "-") ?
stdin : args[1].File;
auto outputFile = new File("output.txt");
foreach (line; inputStream.byLine(KeepTerminator.yes))
outputFile.write(line);
```
std.exception.ErrnoException@C:\D\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(2877):
(No error)
According to the documentation, byLine can throw an UTFException
so relying on the fact that it doesn't in some cases doesn't seem
like a good idea.
Re: How to inform dub of generated source files?
On Thursday, 17 May 2018 at 17:42:21 UTC, Bastiaan Veelo wrote: Isn't preGenerateCommands meant to cover this case? Appears to be a bug. Filed https://github.com/dlang/dub/issues/1474
How to inform dub of generated source files?
Hi, Context: https://github.com/veelo/Pascal2D One of my source files is generated by executing `cd source && rdmd generate.d`, which creates the file `source/epparser.d`. (There is actually one step in between, calling `rdmd make.d`, which checks creation times, but that's not relevant here). I have added this step as preGenerateCommands in dub.json [1]. The problem is that the first time `dub build` is run, it does not seem to be aware of `source/epparser.d` and linking fails [2]. A second `dub build` succeeds. Isn't preGenerateCommands meant to cover this case? I can add `epparser.d` in an extra soureFile line in dub.json, which fixes the very first build, but that makes successive builds fail because it causes the file name to appear twice in the arguments to dmd once the file exists. Changing preGenerateCommands into preBuildCommands makes no difference. Thanks! Bastiaan. [1] https://github.com/veelo/Pascal2D/blob/master/dub.json#L22 [2] https://travis-ci.org/veelo/Pascal2D/builds/379096446#L511
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 15:25:37 UTC, Sjoerd Nijboer wrote:
Given the following code
`struct Foo(T)
if(isNumeric!T)
{
T t;
.. other code
}
struct Bar(T)
if(isNumeric!T)
{
T t;
.. other code
}
Foo!float foo_float;
Foo!double foo_double;
Bar!float bar_float;
`
I want to make a template mixin that is able to cast one of
these generic structs to the other explicitly. I have a bunch
of these structs and therefore I thought it would make sense to
do it by template mixin. I just don't know what language or
library features I need to use and how to apply them.
`
fooDouble = cast (Foo!double) foo_float; // case [1] if
possible, done by
// an implicit cast.
Else by
//explicit.
fooFloat = cast (Foo!float) foo_double; // case [2]
barFloat = cast (Bar!float) foo_float; // case [3]
`
How would I do this in D?
Before you write anything yourself, check whether
std.conv.castFrom does what you need:
https://dlang.org/phobos/std_conv.html#castFrom
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 15:25:37 UTC, Sjoerd Nijboer wrote: I want to make a template mixin that is able to cast one of these generic structs to the other explicitly. I have a bunch of these structs and therefore I thought it would make sense to do it by template mixin. I just don't know what language or library features I need to use and how to apply them. It sounds like you want to overload opCast: https://dlang.org/spec/operatoroverloading.html#cast
Creating a template mixin for explicit casts.
Given the following code
`struct Foo(T)
if(isNumeric!T)
{
T t;
.. other code
}
struct Bar(T)
if(isNumeric!T)
{
T t;
.. other code
}
Foo!float foo_float;
Foo!double foo_double;
Bar!float bar_float;
`
I want to make a template mixin that is able to cast one of these
generic structs to the other explicitly. I have a bunch of these
structs and therefore I thought it would make sense to do it by
template mixin. I just don't know what language or library
features I need to use and how to apply them.
`
fooDouble = cast (Foo!double) foo_float; // case [1] if
possible, done by
// an implicit cast.
Else by
//explicit.
fooFloat = cast (Foo!float) foo_double; // case [2]
barFloat = cast (Bar!float) foo_float; // case [3]
`
How would I do this in D?
Re: fft and isPowerOf2?
On Thursday, 17 May 2018 at 12:34:25 UTC, Andre Pany wrote: this applications throws an error in std.numeric (Line 2826). => assert(isPowerOf2(range.length)); Isn't it possible to give an arbitrary length of data to fft like in numpy? There is a mathematical background which is well explained here: https://math.stackexchange.com/questions/77118/non-power-of-2-ffts#77152
fft and isPowerOf2?
Hi,
this applications throws an error in std.numeric (Line 2826).
=> assert(isPowerOf2(range.length));
---
module std.numeric
void fftImplPureReal(Ret, R)(R range, Ret buf) const
in
{
assert(range.length >= 4);
assert(isPowerOf2(range.length));
}
---
The application is rewritten from python (numpy).
How can I workaround this issue? Isn't it possible to give an
arbitrary length
of data to fft like in numpy?
import std.numeric;
import std.datetime.stopwatch;
import std.stdio: writeln;
import std.random;
import std.array : array;
import std.range : generate, takeExactly;
void fourierTest(int samplingRate)
{
auto sw = StopWatch(AutoStart.yes);
foreach(n; 0..60 * 24)
{
sw.stop();
double[] x = generate!(() =>
uniform01).takeExactly(samplingRate * 60).array;
sw.start();
writeln(x);
auto y = fft(x);
}
sw.stop();
writeln("microseconds: ", sw.peek.total!"usecs");
}
void main()
{
fourierTest(50);
}
Kind regards
André
Re: Dependency injection pattern
On Sunday, 13 May 2018 at 07:42:10 UTC, Suliman wrote:
Could anybody give small example of Dependency injection
pattern? I googled about it, but found only C# examples and I
am not quite sure how to use them.
Also I would like get some explanation/comments for code.
Dependency injection is explicit dependencies are injected in to
an object for use by the creator of the object rather than hard
coding dependencies in to the object.
For example, if you are creating an object that will lightly
depend on an interface I then you can the interface and require
that the object for the interface be injected(specified) by the
user of the object.
class X
{
I i;
void foo() { /* uses i */ };
}
Then X.i = _i; is a dependency injection of _i so that X uses _i.
Usually one does not want the dependency to be freely changed as
to avoid changing a dependency in the middle of it being used.
Various schemes can be create to avoid these issues such as
allowing injection only during construction or some special
synchronization techniques to avoid usages. e.g.,
void foo() { I _i = i; /* uses _i */ };
Then reassignment won't change the dependency when a function is
using it.
Whatever means, the idea is simple: One does not want to hard
code dependencies directly so that refactoring code is ultimately
easier. Dependencies can be injected in to the working code by
the user. This makes the code depend on the type of the
dependency so any specific dependency can be injected.
Top answer here says pretty much exactly what I'm saying:
https://stackoverflow.com/questions/3058/what-is-inversion-of-control?utm_medium=organic_source=google_rich_qa_campaign=google_rich_qa
