Re: How come a count of a range becomes 0 before a foreach?
On Monday, 10 April 2023 at 01:01:59 UTC, Steven Schveighoffer wrote: On 4/9/23 9:16 AM, Ali Çehreli wrote: On 4/8/23 21:38, ikelaiah wrote: > I will modify the code to construct it twice. Multiple iterations of dirEntries can produce different results, which may or may not be what your program will be happy with. Sticking an .array at the end will iterate a single time and maintain the list forever because .array returns an array. :) auto entries = dirEntries(/* ... */).array; I'd be cautious of that. I don't know what the underlying code uses, it may reuse buffers for e.g. filenames to avoid allocation. If you are confident the directory contents won't change in that split-second, then I think iterating twice is fine. -Steve Steve, The Rmd files are not on a network drive, but saved locally. So, I'm confident, the files won't change in a split-second. -ikelaiah.
Re: How come a count of a range becomes 0 before a foreach?
On Sunday, 9 April 2023 at 13:16:51 UTC, Ali Çehreli wrote: On 4/8/23 21:38, ikelaiah wrote: > I will modify the code to construct it twice. Multiple iterations of dirEntries can produce different results, which may or may not be what your program will be happy with. Sticking an .array at the end will iterate a single time and maintain the list forever because .array returns an array. :) auto entries = dirEntries(/* ... */).array; Ali Ali, I didn't think about returning `dirEntries` as `array`. Thanks for the Gems (and your online book too). Regards, ikelaiah
Re: How come a count of a range becomes 0 before a foreach?
On Sunday, 9 April 2023 at 03:39:52 UTC, Steven Schveighoffer wrote: On 4/8/23 9:38 PM, ikelaiah wrote: // Get files in specified inputPath variable with a specific extension auto rmdFiles = file.dirEntries(inputPath, file.SpanMode.shallow) .filter!(f => f.isFile) .filter!(f => f.name.endsWith(fileEndsWith)); // LINE 72 -- WARNING -- If we count the range here, later it will become 0 in line 82 writeln(programName ~ ": number of files found " ~ to!string(rmdFiles.walkLength)); dirEntries returns an *input range*, not a *forward range*. This means that once it's iterated, it's done. If you want to iterate it twice, you'll have to construct it twice. -Steve Steve, You're absolutely right. I did not read the manual correctly. It is clearly written [here](https://dlang.org/library/std/file/dir_entries.html) that `dirEntry` is an `input range`. I will modify the code to construct it twice. Many thanks! -ikelaiah
How come a count of a range becomes 0 before a foreach?
Hi,
I've written a file that converts Rmd (R Markdown file), to a
MarkDown file.
All works well if and only if line 73 is commented out.
I marked line 72 in the code below.
If line 73 is not commented out, `foreach` does not execute as
the `rmdFiles.walklength` in line 82 becomes `0`.
How does `rmdFiles.walkLength` becomes `0` before the `foreach`?
I'm a but confused.
Can someone clarify? Thank you.
```d
module rmd2md;
import std.algorithm;
import std.stdio;
import file = std.file;
import std.conv;
import std.regex;
import std.getopt;
import std.path;
import std.datetime;
import std.parallelism;
import std.range;
void main(string[] args)
{
// Set variables for the main program
string programName = "Rmd2md";
// Setup Regex for capturing Rmd code snippet header
Regex!char re = regex(r"`{3}\{r[a-zA-Z0-9= ]*\}", "g");
// Set default values for the arguments
string inputPath = file.getcwd();
string fileEndsWith = ".Rmd";
string outputPath = file.getcwd();
// Set GetOpt variables
auto helpInformation = getopt(
args,
"path|p", "Path of Rmd files. Default: current working
directory.", ,
"fext|e", "Extension of Rmd files. Default: `.Rmd`",
,
"fout|o", "Output folder to save the MD files. Default:
current working directory.",
);
if (helpInformation.helpWanted)
{
defaultGetoptPrinter("Rmd to Markdown (md) file
converter.",
helpInformation.options);
return;
}
// is the path valid?
if (!std.path.isValidPath(inputPath))
{
writeln(programName ~ ": invalid input path");
return;
}
// is output path valid?
if (!std.path.isValidPath(outputPath))
{
writeln(programName ~ ": invalid output path");
return;
}
// is file extension valid?
if (!startsWith(fileEndsWith, "."))
{
writeln(programName ~ ": invalid extension given");
return;
}
writeln(programName ~ ": input directory is " ~ inputPath);
writeln(programName ~ ": output directory is " ~ outputPath);
writeln(programName ~ ": ...");
// Get files in specified inputPath variable with a specific
extension
auto rmdFiles = file.dirEntries(inputPath,
file.SpanMode.shallow)
.filter!(f => f.isFile)
.filter!(f => f.name.endsWith(fileEndsWith));
// LINE 72 -- WARNING -- If we count the range here, later it
will become 0 in line 82
writeln(programName ~ ": number of files found " ~
to!string(rmdFiles.walkLength));
// Get start time
auto stattime = Clock.currTime();
// Process each Rmd file
int fileWrittenCount = 0;
// LINE 81 -- WARNING -- if line 73 is not commented out, the
walkLength returns 0
writeln(programName ~ ": number of files found " ~
to!string(rmdFiles.walkLength));
foreach (file.DirEntry item; parallel(rmdFiles))
{
writeln(programName ~ ": processing " ~ item.name);
try
{
// Read content as string
string content = file.readText(item.name);
// Replace ```{r} or ```{r option1=value} with ```R
string modified = replaceAll(content, re, "```R");
// Set the Markdown output file
string outputFile = replaceAll(baseName(item.name),
regex(r".Rmd"), ".md");
// Build an output path, using output path and
baseName(item.name)
string outputFilenamePath = buildPath(outputPath,
outputFile);
// Save output Markdown file
file.write(outputFilenamePath, modified);
writeln(programName ~ ": written " ~
outputFilenamePath);
// Increase counter to indicate number of files
processed
fileWrittenCount++;
}
catch (file.FileException e)
{
writeln(programName ~ ": " ~ e.msg);
}
}
writeln(programName ~ ": ...");
// Gett end clock
auto endttime = Clock.currTime();
auto duration = endttime - stattime;
writeln("Duration: ", duration);
// Console output a summary
writeln(programName ~ ": written " ~
to!string(fileWrittenCount) ~ " files");
}
```
For testing, you can create a text file, save as `.Rmd` in the
same folder as the D file. Run the script as:
```bash
rdmd rmd2md.d
```
It will find the `.Rmd` file in current path and save it in the
current path.
Re: My new programming book ...
On Monday, 7 November 2022 at 00:55:33 UTC, zjh wrote: Is there a second edition? After all, it has been many years. Hi zjh, I'm aware of the publication date. However, I find the content still highly relevant to many day-to-day tasks (my use case). No doubt, there will be new features in the std library. For further/additional tips and tricks, I'd gladly ask in the D discord channel or this forum. -Ikel
My new programming book ...
Hi, I got a new programming book yesterday, authored by Adam D. Rupee. . I'm still in **Chapter 1**, on the matter of `Immutable` variables. Can't wait to read **Chapter 7** on **Corectness and Documentation**. So far, I really like the book. - Alexandrei's foreword on the importance of drawing experience from and for others. - The book convention (code, info, warning, new terms, etc) is easy to follow. - The explanation is also easy to digest. A bit of context of myself: I used to do C++, Java and C#. Currently work with: Python, R, PHP, SQL, common web-stack (html, css, js+libs). I'm hoping to integrate more D into my data transformation and data reporting workflow. @Adam D. Rupee, thanks for this book. Regards, Ikel
Re: Main foreach loop fails when another foreach is added
On Monday, 8 August 2022 at 02:45:54 UTC, Steven Schveighoffer wrote: And now, you tried to read it again! Which means you are trying to read more data from an empty stream. You need to either a) reopen the file, or b) do both in the same loop. -Steve Steve! You are Legend! **Thank you**. It is obvious now the second `foreach` is reading from an empty stream! Regards, ikel
Main foreach loop fails when another foreach is added
Hi,
I'm writing a program that reads a text file and launch my work
URLs in it.
It worked fine, and very happy.
Then I added another `foreach` loop to count total number of
lines.
After this, the main `foreach` won't work.
Does anyone know as to why this happens?
I might have missed something very obvious.
Thank you.
Regards,
ikel
```d
module openurls;
import std.stdio : writeln, File, KeepTerminator;
import std.file;
import std.range;
import std.ascii;
import std.getopt : getopt, GetoptResult, defaultGetoptPrinter;
import std.process : browse;
import core.thread.osthread;
import core.time;
void main(string[] args)
{
string input_filename = "";
// Get command line option
GetoptResult user_args = getopt(args,
"input|i", "A list of URLs separated by new lines.",
_filename);
// Does user need help?
if (user_args.helpWanted)
{
defaultGetoptPrinter("Command line options for Open
URLs.",
user_args.options);
}
// Is input file specified?
if (input_filename.length == 0)
{
writeln("Open URL: No file specified.\n");
defaultGetoptPrinter("Command line option for Open URLs.",
user_args.options);
return;
}
// Does the file exist?
if (!input_filename.exists)
{
writeln("Open URL: input file --" ~ input_filename ~ "--
does not exist.\n");
return;
}
else
{
writeln("Open URL: reading --" ~ input_filename ~ "--");
}
// Open input file for reading
size_t line_count = 0;
File input_file = File(input_filename);
auto file_range = input_file.byLine(KeepTerminator.no,
std.ascii.newline);
// if this foreach not commented out
// the last foreach won't run.
foreach (char[] line; file_range)
{
line_count += 1;
}
// the MAIN foreach loop
// -- won't run if above foreach is enabled
foreach (char[] line; file_range)
{
if (!line.empty)
{
writeln("Open URL: " ~ line);
Thread.sleep(dur!("seconds")(2));
browse(line);
}
}
writeln("Open URL: completed");
writeln("Closing " ~ input_filename);
input_file.close();
}
```
Re: Combining JSON arrays into a single JSON array -- better way than this?
On Monday, 1 August 2022 at 17:02:10 UTC, Salih Dincer wrote: When printing the steps `import std.stdio : writeln;` use it. Thus, it does not conflict with `std.file`. SDB@79 Will do and thank you for this.
Re: Combining JSON arrays into a single JSON array -- better way than this?
On Monday, 1 August 2022 at 16:11:52 UTC, frame wrote: ... because loading all in memory while not necessary is wasted memory if you have large files to process. Noted with thanks. Actually, some JSON input files are pretty large (>1Mb). I would just process each file and write it to the file directly but it always depends on the intended purpose. Noted as well. Thank you, this will come handy in the future. -ikel
Re: Combining JSON arrays into a single JSON array -- better way than this?
On Monday, 1 August 2022 at 05:52:36 UTC, frame wrote:
If the JSON files are already parsed, why you stringify and
reparse it?
Because of ...
1. mental block and
2. I didn't know `jsonResult.array = [];`
Many thanks for pointing this out. I tried the following, and
didn't work, and hence my earlier convoluted approach:
- `JSONValue[] jsonResult;`
- `JSONValue jsonResult = [];`
A plain array should be also mergeable via `jsonResult.array ~=
j.array` (if `j` really is an array, you need to check the type
first)
Based in your suggestion, the snippet is now more brief.
```d
module combinejsonv3;
import std.file;
import std.stdio;
import std.json;
import std.array;
import std.algorithm.searching;
void main()
{
// merged JSON to be stored here
JSONValue jsonResult;
jsonResult.array = [];
foreach (string filename; dirEntries(".", "*.json",
SpanMode.shallow))
{
// if filename contains 'output' in it, ignore
if(canFind(filename, "output")) {
std.stdio.writeln("ignoring: " ~ filename);
continue;
}
// read JSON file as string
string content = std.file.readText(filename);
// parse as JSON
JSONValue j = parseJSON(content);
// if JSONType is array, merge
if(j.type == JSONType.array) {
// show status to console
std.stdio.writeln("processing JSON array from: "
~ filename);
jsonResult.array ~= j.array;
}
}
// write to file
std.file.write("output-combined.json",
jsonResult.toPrettyString);
}
```
Thank you!
-ikel
Re: Combining JSON arrays into a single JSON array -- better way than this?
On Monday, 1 August 2022 at 07:35:34 UTC, Christian Köstlin wrote:
An arguably shorter solution (that drops some of your logging)
could be:
```d
import std;
void main() {
dirEntries(".", "*.json", SpanMode.shallow)
.filter!(f => !f.name.canFind("output"))
.map!(readText)
.map!(parseJSON)
.fold!((result, json) { result ~= json.array; return
result; })
.toPrettyString
.reverseArgs!(std.file.write)("output-combined.json");
}
```
not sure if you are looking for this style though.
kind regards,
Christian
Hi Christian,
So we can do that in D?!
Thanks for sharing this amazing approach.
-ikelaiah
Combining JSON arrays into a single JSON array -- better way than this?
Hi,
I've written a cli tool to merge JSON files (containing JSON
array) in the current folder as a single JSON file.
My algorithm:
1. Create a string to store the output JSON array as a string,
2. read each file
3. read each object in JSON array from input file
4. append the string representation of each JSON object in a
string
5. Parse the result string as JSON
6. Save string representation of point 6 above.
**Question**
While this works, is there a more concise way than this?
Thank you for your time.
```d
module combinejsonv2;
import std.file;
import std.stdio;
import std.json;
import std.array;
import std.algorithm.searching;
void main()
{
// can't I use this for JSON array?
JSONValue jj;
// create a string opening [ for JSON array
string stringResult = "[";
foreach (string filename; dirEntries(".", "*.json",
SpanMode.shallow))
{
// if filename contains 'output' in it, ignore
if(canFind(filename, "output")) {
std.stdio.writeln("ignoring: " ~ filename);
continue;
}
// show status to console
std.stdio.writeln("processing: " ~ filename);
// read JSON file as string
string content = std.file.readText(filename);
// parse as JSON
JSONValue j = parseJSON(content);
foreach (JSONValue jsonObject; j.array) {
// Combine objects from files into a single list of
JSON object
// std.stdio.writeln(jsonObject.toPrettyString);
stringResult ~= jsonObject.toString;
stringResult ~= ",";
}
}
// create closing ] for the JSON array
stringResult ~= "]";
// parse the string as a JSON object
JSONValue jsonResult = parseJSON(stringResult);
// write to file
std.file.write("output-combined.json",
jsonResult.toPrettyString);
}
```
