Re: [Scilab-users] using csvRead

[jean-philippe grivet]

Both vi and emacs can handle vary large files. You could use either to 
replace all

separatots by commas and then use csvRead.

Le 15/10/2016 15:45, Samuel Gougeon a écrit :

Hello

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:


HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

dd = day
mm = month
 = year

HH = hour
MM = minute
SS = second
Ms = milli second

ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?
Beside the solution provided by Serge, if you wish to stick to 
csvRead(), have you tried using its "substitute" option, with a [":" 
","] value?
I didn't, but would be interested by results. I am wondering whether 
the substitution is done before splitting the line with the separator 
(then "substitute" will work), or after. This detail would deserve 
being documented.


Read you soon
Samuel Gougeon



___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users




---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead vs mfscanf and fscanfMat

[Rafael Guerra]

Hi,

The problem is not with csvRead, but with the need in the example we are 
considering here to use the evstr function afterwards on large string data to 
convert it to useable numeric values.
See time breakdown here below for another 50,000 line input data test:

time1= 0.6552// mfscanf
time2= 0.4680   // fscanfMat
time3a= 0.2028 // csvRead
time3= 34.3514 // csvRead + evstr

Note that method#2 writes a temporary file to disk and so it will run much 
faster on PC's with SSD drives.

Regards,
Rafael

-Original Message-
From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Lester Anderson
Sent: Monday, October 17, 2016 10:26 AM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead vs mfscanf and fscanfMat

Hello, I ran the same code on my machine and actually got worse results:

-->exec('Q:\Scilab_code\csvread_write.sce', -1)

1.12.2015.1.15.0.12.1.1  - 2.2

1.12.2015.1.15.0.12.1.1  - 2.2

!01.12.2015  1  15  0.12  1.1  -2.2  !

time1= 1.21681
time2= 2.19961
time3= 51.7923

Windows 7 64-bit and 64 Gb Ram. (Scilab 5.5.2). Is this a bug if the
csvRead result is so different?

Lester

>>
>> The results for a 50,000-lines input ASCII file are:
>> time1= 0.686404   // mfscanf
>> time2= 0.499203   // fscanfMat
>> time3= 35.3966// csvRead
>

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead vs mfscanf and fscanfMat

[Lester Anderson]

Hello, I ran the same code on my machine and actually got worse results:

-->exec('Q:\Scilab_code\csvread_write.sce', -1)

1.12.2015.1.15.0.12.1.1  - 2.2

1.12.2015.1.15.0.12.1.1  - 2.2

!01.12.2015  1  15  0.12  1.1  -2.2  !

time1= 1.21681
time2= 2.19961
time3= 51.7923

Windows 7 64-bit and 64 Gb Ram. (Scilab 5.5.2). Is this a bug if the
csvRead result is so different?

Lester

>>
>> The results for a 50,000-lines input ASCII file are:
>> time1= 0.686404   // mfscanf
>> time2= 0.499203   // fscanfMat
>> time3= 35.3966// csvRead
>
> .
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Philipp Mühlmann]

Dear Samuel, Rafael,

thanks for taking my initial question further to a higher level.

I wouldn't have expected to create a bug out of it.

So, for me:

 I am fine with either creating a temporary file and using fscanfMat or
mfscanf().

In fact:

fscanfMat() is closer to my level of code understanding, since I get a
m-x-n matrix as a result

with mfscanf() I get an mlist.

Thanks a lot,
Philipp




2016-10-15 23:40 GMT+02:00 Samuel Gougeon :

> Le 15/10/2016 21:16, Samuel Gougeon a écrit :
>
> Hello Philipp,
>
> Le 14/10/2016 00:08, Philipp Mühlmann a écrit :
>
> Dear Scilab users,
>
> having a data file (*.cvs) containg following format:
>
>
> HEADER-Line
> dd.mm., HH:MM:SS.MS, value01, value02
>
> dd = day
> mm = month
>  = year
>
> HH = hour
> MM = minute
> SS = second
> Ms = milli second
>
> ValueXY = numerical value
>
> Is it possible to use cvsRead in such a way to define the separator beeing
> ',' and ':' at the same time?
>
> .
> Yes and no: Yes because each character of the "separator" option is
> (sometimes) considered as a
> separator. And no, because it sometimes fails.
> I was designing an example to add to the csvRead() page, when a found this
> bug:
>
> --> mputl("12.08.2016, 13:54:18.45, 3.145, 548.4", TMPDIR+"/test.csv")
>
> --> r = "''"+csvRead("test.csv","8.",[],"string")+"''"
>  r  =
> !'12'  '0'  '2016, 13:54:1'  '45, 3'  '145, 54'  '4'  !
> --> // it works: both "8" and "." are *separately* considered as
> separators
>
> --> // now trying with "," and ":":
> --> r = "''"+csvRead("test.csv",",:", [], "string")+"''"
> --> r = "''"+csvRead("test.csv",",:", [], "string")+"''"
>  r  =
>  '12.08.2016, 13:54:18.45, 3.145, 548.4'
> --> // The row is kept as is: no splitting!
> --> // We could expect: ['12.08.2016' ' 13' '54' '18.45' ' 3.145' '
> 548.4']
> --> //  (with leading spaces, since they are not set as separators)
> -->
>
> There are already 10 pending reported bugs about csvRead(). One more.
>
> Report : http://bugzilla.scilab.org/14809
>
> Each char of a separator with length > 1 is considered a specific
> separator only if the whole separator pattern is met in the read row.
> Otherwise, no splitting is done. Hence,
> --> r = csvRead(TMPDIR+"test.csv","8.",[],"string") // splits the
> "12.08.2016" row, while
> --> r = csvRead(TMPDIR+"test.csv",".8",[],"string") // does not!
>
> Samuel Gougeon
>
>
> ___
> users mailing list
> users@lists.scilab.org
> http://lists.scilab.org/mailman/listinfo/users
>
>


-- 
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Le 15/10/2016 21:16, Samuel Gougeon a écrit :

Hello Philipp,

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:


HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

dd = day
mm = month
 = year

HH = hour
MM = minute
SS = second
Ms = milli second

ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?

.
Yes and no: Yes because each character of the "separator" option is 
(sometimes) considered as a

separator. And no, because it sometimes fails.
I was designing an example to add to the csvRead() page, when a found 
this bug:


--> mputl("12.08.2016, 13:54:18.45, 3.145, 548.4", TMPDIR+"/test.csv")

--> r = "''"+csvRead("test.csv","8.",[],"string")+"''"
 r  =
!'12'  '0'  '2016, 13:54:1'  '45, 3'  '145, 54'  '4'  !
--> // it works: both "8" and "." are *separately* considered as 
separators


--> // now trying with "," and ":":
--> r = "''"+csvRead("test.csv",",:", [], "string")+"''"
--> r = "''"+csvRead("test.csv",",:", [], "string")+"''"
 r  =
 '12.08.2016, 13:54:18.45, 3.145, 548.4'
--> // The row is kept as is: no splitting!
--> // We could expect: ['12.08.2016' ' 13' '54' '18.45' ' 3.145' ' 
548.4']

--> //  (with leading spaces, since they are not set as separators)
-->

There are already 10 pending reported bugs about csvRead(). One more.

Report : http://bugzilla.scilab.org/14809

Each char of a separator with length > 1 is considered a specific 
separator only if the whole separator pattern is met in the read row. 
Otherwise, no splitting is done. Hence,
--> r = csvRead(TMPDIR+"test.csv","8.",[],"string") // splits the 
"12.08.2016" row, while

-->r = csvRead(TMPDIR+"test.csv",".8",[],"string") // does not!

Samuel Gougeon

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead vs mfscanf and fscanfMat

[Samuel Gougeon]

Hello Rafael,

Le 15/10/2016 20:49, Rafael Guerra a écrit :

.../...

The results for a 50,000-lines input ASCII file are:
time1= 0.686404   // mfscanf
time2= 0.499203   // fscanfMat
time3= 35.3966// csvRead

.
Thanks for these very convincing results.
I thought that evstr() is vectorized because it is so on my PC, after 
having worked on it.

But a trivial improvement was still unsubmitted.
Repaired: https://codereview.scilab.org/#/c/18586/

Despite this improvement fasten evstr() by a factor > 10, this is not 
enough to reach mfscanf()'s speed.


BR
Samuel

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Hello Philipp,

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:


HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

dd = day
mm = month
 = year

HH = hour
MM = minute
SS = second
Ms = milli second

ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?

.
Yes and no: Yes because each character of the "separator" option is 
(sometimes) considered as a

separator. And no, because it sometimes fails.
I was designing an example to add to the csvRead() page, when a found 
this bug:


--> mputl("12.08.2016, 13:54:18.45, 3.145, 548.4", TMPDIR+"/test.csv")

--> r = "''"+csvRead("test.csv","8.",[],"string")+"''"
 r  =
!'12'  '0'  '2016, 13:54:1'  '45, 3'  '145, 54'  '4'  !
--> // it works: both "8" and "." are *separately* considered as separators

--> // now trying with "," and ":":
--> r = "''"+csvRead("test.csv",",:", [], "string")+"''"
--> r = "''"+csvRead("test.csv",",:", [], "string")+"''"
 r  =
 '12.08.2016, 13:54:18.45, 3.145, 548.4'
--> // The row is kept as is: no splitting!
--> // We could expect: ['12.08.2016' ' 13' '54' '18.45' ' 3.145' ' 548.4']
--> //  (with leading spaces, since they are not set as separators)
-->

There are already 10 pending reported bugs about csvRead(). One more.

Samuel

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead vs mfscanf and fscanfMat

[Rafael Guerra]

Hi Samuel,

Please check test code here below, comparing csvRead vs mfscanf and fscanfMat 
for the asci format used by Philipp and a file with 50,000 lines of data.
On my laptop it takes about 35 s to run mainly because of evstr function, which 
is avoided in the mfscanf and fscanfMat methods as shown.


// Simple test of mfscanf, fscanfMat and csvRead methods
//START OF CODE
clear;
txt = [ "HEADER-Line",
"01.12.2015, 01:15:00.12, 1.1, -2.2"];

u = mopen("myfile.txt","w");
mfprintf(u,"%s\n",txt(1));
mfprintf(u,"%s\n",repmat(txt(2),5,1));  //output file with 50,000 lines
mclose(u)

timer();

// SOLUTION#1: mfscanf
u = mopen("myfile.txt","r");
h = mfscanf(1,u,"%s\n");
r = mfscanf(-1,u,"%d.%d.%d, %d:%d:%d.%d, %f, %f\n");
mclose(u)
r = r(:,:);  //to convert from mlist of ctype to matrix of constant type
t1 = timer();


// SOLUTION#2: fscanfMat
u = mopen("myfile.txt","r");
tx = mgetl(u,-1)
mclose(u);
tx = tx(2:$);  // get rid of header line
tx1 = part(tx,1:24);  // get date and time
tx2 = part(tx,25:$);  // get numeric data values
// Now get rid of separators:
tx1 = strsubst(tx1,'.',' ');
tx1 = strsubst(tx1,':',' ');
tx1 = strsubst(tx1,',',' ');
tx2 = strsubst(tx2,',',' ');
tx = tx1 + tx2; // regroups all data but now with numeric values only
fd = mopen("temp.bak","w");
mputl(tx,fd);
mclose(fd);
m = fscanfMat('temp.bak')
mdelete('temp.bak');
t2 = timer();

// SOLUTION#3: csvRead
q = csvRead("myfile.txt",",",[],"string",[":", ","],[],[],1);
tx1 = q(:,1);
tx2 = q(:,2:$);
q2 = evstr(tx2);  // Most time consuming step
// (plus, work will still be required to handle dates in txt1)
t3 = timer();

disp( [tx1(1,:) string(q2(1,:))], r(1,:), m(1,:) )
printf("\ntime1= %g\ntime2= %g\ntime3= %g",t1,t2,t3)
//END OF CODE

The results for a 50,000-lines input ASCII file are:
   time1= 0.686404
   time2= 0.499203
   time3= 35.3966

Regards,
Rafael


From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Samuel Gougeon
Sent: Saturday, October 15, 2016 7:36 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

Le 15/10/2016 19:16, Rafael Guerra a écrit :
Hi Samuel,
 
As the data is loaded by csvRead as strings in the example below (if loading as 
doubles then we get NaN's), it will require further processing to convert it to 
numeric (using evstr, tokens or other).
For very large data files, this seems to be rather slow compared to the mfscanf 
or fscanfMat solutions.
 
What do you think?
.
AFAIK, fscanfMat() is very stiff. It can parse files only for numbers, with no 
interstitial contents.
I know no benchmark comparing csvRead() + evstr() vs mfscanf(). Despite evstr() 
is vectorized, you may be right. Explicit results would be interesting.
mfscanf() requires the structure of a row been explicitly known. But then it 
looks certainly the most versatile and adaptable solution to read and split it.
csvRead() requires just to know the separator.

Samuel
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Le 15/10/2016 19:32, Rafael Guerra a écrit :


Thanks Samuel, but I could not see where does it say how can we 
convert a mlist of ctype to a regular matrix, so that we can use all 
usual matrix operations. Could you please advise.




You are right, the *cblock* type is nowhere documented. Simple tests 
show that it is similar to -- but more handy than -- a row of cells:


-->b = msscanf(-1,['012345 abc zoo';"457.1 tfr 845"],'%f %s %s')
 b  =
  12345  abc  zoo
  457.10001  tfr  845

-->b(:,1), typeof(b(:,1))
 ans  =
12345.
457.10001
 ans  =
 constant

-->b(:,2), typeof(b(:,2))
 ans  =
!abc  !
! !
!tfr  !
 ans  =
 string

-->b(1,:)
 ans =
  12345  abc  zoo  // Yeh!

Searching for "cblock" in the help shows that, apart for Xcos, this 
mtype is used only by m*scanf().
Its addressing indices are exactly what we could expect for addressing 
cells contents, with no specific useless and awkward {} insertors or 
extractors.
[ BTW, {} yield an error at Scilab 5 compilation time. So the same macro 
code can't be compiled with Scilab 5 and Scilab 6 as soon as {} 
indexation is used].


Samuel

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Le 15/10/2016 19:16, Rafael Guerra a écrit :


Hi Samuel,

As the data is loaded by csvRead as strings in the example below (if 
loading as doubles then we get NaN’s), it will require further 
processing to convert it to numeric (using evstr, tokens or other).


For very large data files, this seems to be rather slow compared to 
the mfscanf or fscanfMatsolutions.


What do you think?


.
AFAIK, fscanfMat() is very stiff. It can parse files only for numbers, 
with no interstitial contents.
I know no benchmark comparing csvRead() + evstr() vs mfscanf(). Despite 
evstr() is vectorized, you may be right. Explicit results would be 
interesting.
mfscanf() requires the structure of a row been explicitly known. But 
then it looks certainly the most versatile and adaptable solution to 
read and split it.

csvRead() requires just to know the separator.

Samuel

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

Thanks Samuel, but I could not see where does it say how can we convert a mlist 
of ctype to a regular matrix, so that we can use all usual matrix operations. 
Could you please advise.

PS: in the example under discussion, doing:   r=r( :,: );  seems to do the 
trick.


From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Samuel Gougeon
Sent: Saturday, October 15, 2016 7:26 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

Le 15/10/2016 18:58, Rafael Guerra a écrit :
Hi Serge,

One question:
For some strange reason, the variable 'r' is in this case a mlist of type 
cblock and not a matrix of type constant.
To convert it to constant, it seems that we need a further instruction, such 
as:   r=r( : , : );
Any comments on this?
This is fully documented: https://help.scilab.org/docs/6.0.0/en_US/mfscanf.html
It is possible to specify several output variables to collect field columns one 
by one.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Le 15/10/2016 18:58, Rafael Guerra a écrit :


Hi Serge,

One question:

For some strange reason, the variable ‘r’ is in this case a mlist of 
type cblock and not a matrix of type constant.


To convert it to constant, it seems that we need a further 
instruction, such as:r=r( : , : );


Any comments on this?

This is fully documented: 
https://help.scilab.org/docs/6.0.0/en_US/mfscanf.html
It is possible to specify several output variables to collect field 
columns one by one.


___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

Hi Samuel,

As the data is loaded by csvRead as strings in the example below (if loading as 
doubles then we get NaN's), it will require further processing to convert it to 
numeric (using evstr, tokens or other).
For very large data files, this seems to be rather slow compared to the mfscanf 
or fscanfMat solutions.

What do you think?

Regards,
Rafael

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Samuel Gougeon
Sent: Saturday, October 15, 2016 5:55 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

Le 15/10/2016 15:45, Samuel Gougeon a écrit :
Hello

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :
Dear Scilab users,
having a data file (*.cvs) containg following format:

HEADER-Line
dd.mm., HH:MM:SS.MS<http://SS.MS>, value01, value02

dd = day
mm = month
 = year
HH = hour
MM = minute
SS = second
Ms = milli second
ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator beeing ',' 
and ':' at the same time?
Beside the solution provided by Serge, if you wish to stick to csvRead(), have 
you tried using its "substitute" option, with a [":" ","] value?
I didn't, but would be interested by results.
.
It works: the substitution is done before the splitting.
Example:
The file.csv content:

Header line
12.08.2016, 13:54:18.45, 3.145, 548.4


-->csvRead("file.csv",",",[],"string",[":", ","],[],[],1)
 ans  =
!12.08.2016   13  54  18.45   3.145   548.4  !

Samuel
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

Hi Serge,

One question:
For some strange reason, the variable 'r' is in this case a mlist of type 
cblock and not a matrix of type constant.
To convert it to constant, it seems that we need a further instruction, such 
as:   r=r( : , : );
Any comments on this?

Regards,
Rafael


From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Serge Steer
Sent: Friday, October 14, 2016 8:52 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

you can use the mfscanf function:

u=mopen("myfile","r");
h=mfscanf(1,u,"%s\n");
r=mfscanf(-1,u,"%d.%d.%d, %d:%d:%f, %f, %f\n");
mclose(u)

Serge Steer

Le 14/10/2016 à 00:08, Philipp Mühlmann a écrit :
Dear Scilab users,
having a data file (*.cvs) containg following format:

HEADER-Line
dd.mm., HH:MM:SS.MS<http://SS.MS>, value01, value02

dd = day
mm = month
 = year
HH = hour
MM = minute
SS = second
Ms = milli second
ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator beeing ',' 
and ':' at the same time?
Background:

desired Matrix after reading the file is
M = [dd mm  HH MM SS MS value1 value2]

Thank you,
Philipp

--
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.




___

users mailing list

users@lists.scilab.org<mailto:users@lists.scilab.org>

http://lists.scilab.org/mailman/listinfo/users


___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Le 15/10/2016 15:45, Samuel Gougeon a écrit :

Hello

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:


HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

dd = day
mm = month
 = year

HH = hour
MM = minute
SS = second
Ms = milli second

ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?
Beside the solution provided by Serge, if you wish to stick to 
csvRead(), have you tried using its "substitute" option, with a [":" 
","] value?
I didn't, but would be interested by results. 

.
It works: the substitution is done *before* the splitting.
Example:
The file.csv content:

Header line
12.08.2016, 13:54:18.45, 3.145, 548.4


-->csvRead("file.csv",",",[],"string",[":", ","],[],[],1)
 ans  =
!12.08.2016   13  54  18.45   3.145   548.4  !

Samuel

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:

HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

.../...
Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?


desired Matrix after reading the file is

M = [dd mm  HH MM SS MS value1 value2]

.
This requires also that "." to be considered as a separator to split 
dd.mm., in addition to "," and ":".

Then, the SS.MS will yield 2 separate fields.

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Samuel Gougeon]

Hello

Le 14/10/2016 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:


HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

dd = day
mm = month
 = year

HH = hour
MM = minute
SS = second
Ms = milli second

ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?
Beside the solution provided by Serge, if you wish to stick to 
csvRead(), have you tried using its "substitute" option, with a [":" 
","] value?
I didn't, but would be interested by results. I am wondering whether the 
substitution is done before splitting the line with the separator (then 
"substitute" will work), or after. This detail would deserve being 
documented.


Read you soon
Samuel Gougeon

___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

Thanks for the perfect neat solution in C-style.
A nice example of Columbus' egg and also of why bother making it simple when we 
made it difficult.

Kind regards,
Rafael

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Serge Steer
Sent: Friday, October 14, 2016 8:52 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

you can use the mfscanf function:

u=mopen("myfile","r");
h=mfscanf(1,u,"%s\n");
r=mfscanf(-1,u,"%d.%d.%d, %d:%d:%f, %f, %f\n");
mclose(u)

Serge Steer

Le 14/10/2016 à 00:08, Philipp Mühlmann a écrit :
Dear Scilab users,
having a data file (*.cvs) containg following format:

HEADER-Line
dd.mm., HH:MM:SS.MS<http://SS.MS>, value01, value02

dd = day
mm = month
 = year
HH = hour
MM = minute
SS = second
Ms = milli second
ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator beeing ',' 
and ':' at the same time?
Background:

desired Matrix after reading the file is
M = [dd mm  HH MM SS MS value1 value2]

Thank you,
Philipp

--
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.




___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Serge Steer]

you can use the mfscanf function:

u=mopen("myfile","r");
h=mfscanf(1,u,"%s\n");
r=mfscanf(-1,u,"%d.%d.%d, %d:%d:%f, %f, %f\n");
mclose(u)

Serge Steer

Le 14/10/2016 à 00:08, Philipp Mühlmann a écrit :

Dear Scilab users,

having a data file (*.cvs) containg following format:


HEADER-Line
dd.mm., HH:MM:SS.MS , value01, value02

dd = day
mm = month
 = year

HH = hour
MM = minute
SS = second
Ms = milli second

ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator 
beeing ',' and ':' at the same time?


Background:

desired Matrix after reading the file is

M = [dd mm  HH MM SS MS value1 value2]


Thank you,
Philipp


--
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.


___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users



___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Philipp Mühlmann]

Dear Rafael,

thank you for the example.

One comment:

before mputl(tx,fd)

I had to use:

tx = [tx1+tx2]--> 3 x 1 matrix

instead of

tx = [tx1 tx2]   --> 3 x 2 matrix, does not work with mputl

However,
thanks for the solution.
Philipp



2016-10-14 13:28 GMT+02:00 Rafael Guerra <jrafaelbgue...@hotmail.com>:

> NOTE:  fyi, after testing over very large text files, it happens that
> doing *mputl* of  ‘tx’ to disk followed by *M=fscanfMat *is much faster
> than doing simply*  M=evstr(tx) *…
>
>
>
>
>
> *From:* users [mailto:users-boun...@lists.scilab.org] *On Behalf Of *Rafael
> Guerra
> *Sent:* Friday, October 14, 2016 12:58 PM
>
> *To:* Users mailing list for Scilab <users@lists.scilab.org>
> *Subject:* Re: [Scilab-users] using csvRead
>
>
>
> *Correction:*  after having obtained ‘tx’ as below, just do:M=evstr(tx
> );
>
>
>
> *From:* users [mailto:users-boun...@lists.scilab.org
> <users-boun...@lists.scilab.org>] *On Behalf Of *Rafael Guerra
> *Sent:* Friday, October 14, 2016 12:53 PM
> *To:* Users mailing list for Scilab <users@lists.scilab.org>
> *Subject:* Re: [Scilab-users] using csvRead
>
>
>
> Hello Philipp,
>
>
>
> Say that after *mopen* you got all your text input into array of strings
> ‘txt’:
>
>
>
> txt = [
>
> "HEADER-Line",
>
> "01.12.2015, 01:15:00.12, 1.1, -2.2";
>
> "03.12.2015, 11:15:00.12, -11.1, 2.5";
>
> "12.12.2015, 21:15:00.12, 5.1, 6.2"];
>
>
>
> Then do the following:
>
>
>
> tx=txt(2:$);  // get rid of header line
>
> tx1=part(tx,1:24);  // get date and time
>
> tx2=part(tx,25:$);  // get numeric data values
>
> // Now get rid of separators:
>
> tx1 = strsubst(tx1,'.',' ');
>
> tx1 = strsubst(tx1,':',' ');
>
> tx1 = strsubst(tx1,',',' ');
>
> tx2 = strsubst(tx2,',',' ');
>
> tx = [tx1 tx2]; // regroups all data but now with numeric values only
>
>
>
> Use  *mputl* to output ‘tx’ to temporary disk file and use *fscanfMat* to
> read like a breeze the large disk file (now fully numeric) into a Scilab
> numeric matrix ‘M’.
>
>
>
> Regards,
>
> Rafael
>
>
>
> *From:* users [mailto:users-boun...@lists.scilab.org
> <users-boun...@lists.scilab.org>] *On Behalf Of *Philipp Mühlmann
> *Sent:* Friday, October 14, 2016 11:50 AM
> *To:* Users mailing list for Scilab <users@lists.scilab.org>
> *Subject:* Re: [Scilab-users] using csvRead
>
>
>
> Dear Denis,
>
>
>
> yes thats the way I do it right now.
>
>
>
> use mopen --> open file for reading
>
> use mgetl --> read data, result = array of strings
>
> use strsplit --> split  string Array as desired
>
>
>
> use evestring() --> convert string to double
>
>
>
> Point of disadvantage
>
>
>
> So fa I know strsplit() can handle only one string.
>
> Hence I use a for-loop to split each line of the Initial string array into
> a group of strings and convert each part into a double.
>
>
>
> OK for few data...may take long for many data
>
>
>
> Idea:
>
> Read data from file and try to spare conversation string-to-double.
>
> fscanfMat() won't do it, because the data does not contain only numerical
> values and "."-sign as decimal.
>
> So I tried csvRead.
>
> using the help I find:  separator :  a 1-by-1 matrix of strings, the
> field separator used.
>
>
>
> Note:   STRINGS   = plural.
>
>
>
> So I wonder if it is possible to have more than one separator here.
>
>
>
> BR
>
> Philipp
>
>
>
> mclose( :-) )
>
>
>
>
>
> ___
> users mailing list
> users@lists.scilab.org
> http://lists.scilab.org/mailman/listinfo/users
>
>


-- 
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

NOTE:  fyi, after testing over very large text files, it happens that doing 
mputl of  ‘tx’ to disk followed by M=fscanfMat is much faster than doing simply 
 M=evstr(tx) …


From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Rafael Guerra
Sent: Friday, October 14, 2016 12:58 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

Correction:  after having obtained ‘tx’ as below, just do:M=evstr(tx);

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Rafael Guerra
Sent: Friday, October 14, 2016 12:53 PM
To: Users mailing list for Scilab 
<users@lists.scilab.org<mailto:users@lists.scilab.org>>
Subject: Re: [Scilab-users] using csvRead

Hello Philipp,

Say that after mopen you got all your text input into array of strings ‘txt’:

txt = [
"HEADER-Line",
"01.12.2015, 01:15:00.12, 1.1, -2.2";
"03.12.2015, 11:15:00.12, -11.1, 2.5";
"12.12.2015, 21:15:00.12, 5.1, 6.2"];

Then do the following:

tx=txt(2:$);  // get rid of header line
tx1=part(tx,1:24);  // get date and time
tx2=part(tx,25:$);  // get numeric data values
// Now get rid of separators:
tx1 = strsubst(tx1,'.',' ');
tx1 = strsubst(tx1,':',' ');
tx1 = strsubst(tx1,',',' ');
tx2 = strsubst(tx2,',',' ');
tx = [tx1 tx2]; // regroups all data but now with numeric values only

Use  mputl to output ‘tx’ to temporary disk file and use fscanfMat to read like 
a breeze the large disk file (now fully numeric) into a Scilab numeric matrix 
‘M’.

Regards,
Rafael

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Philipp 
Mühlmann
Sent: Friday, October 14, 2016 11:50 AM
To: Users mailing list for Scilab 
<users@lists.scilab.org<mailto:users@lists.scilab.org>>
Subject: Re: [Scilab-users] using csvRead

Dear Denis,

yes thats the way I do it right now.

use mopen --> open file for reading
use mgetl --> read data, result = array of strings
use strsplit --> split  string Array as desired

use evestring() --> convert string to double

Point of disadvantage

So fa I know strsplit() can handle only one string.
Hence I use a for-loop to split each line of the Initial string array into a 
group of strings and convert each part into a double.

OK for few data...may take long for many data

Idea:
Read data from file and try to spare conversation string-to-double.
fscanfMat() won't do it, because the data does not contain only numerical 
values and "."-sign as decimal.
So I tried csvRead.
using the help I find:  separator :  a 1-by-1 matrix of strings, the field 
separator used.

Note:   STRINGS   = plural.

So I wonder if it is possible to have more than one separator here.

BR
Philipp

mclose( :-) )


___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

Correction:  after having obtained ‘tx’ as below, just do:M=evstr(tx);

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Rafael Guerra
Sent: Friday, October 14, 2016 12:53 PM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

Hello Philipp,

Say that after mopen you got all your text input into array of strings ‘txt’:

txt = [
"HEADER-Line",
"01.12.2015, 01:15:00.12, 1.1, -2.2";
"03.12.2015, 11:15:00.12, -11.1, 2.5";
"12.12.2015, 21:15:00.12, 5.1, 6.2"];

Then do the following:

tx=txt(2:$);  // get rid of header line
tx1=part(tx,1:24);  // get date and time
tx2=part(tx,25:$);  // get numeric data values
// Now get rid of separators:
tx1 = strsubst(tx1,'.',' ');
tx1 = strsubst(tx1,':',' ');
tx1 = strsubst(tx1,',',' ');
tx2 = strsubst(tx2,',',' ');
tx = [tx1 tx2]; // regroups all data but now with numeric values only

Use  mputl to output ‘tx’ to temporary disk file and use fscanfMat to read like 
a breeze the large disk file (now fully numeric) into a Scilab numeric matrix 
‘M’.

Regards,
Rafael

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Philipp 
Mühlmann
Sent: Friday, October 14, 2016 11:50 AM
To: Users mailing list for Scilab 
<users@lists.scilab.org<mailto:users@lists.scilab.org>>
Subject: Re: [Scilab-users] using csvRead

Dear Denis,

yes thats the way I do it right now.

use mopen --> open file for reading
use mgetl --> read data, result = array of strings
use strsplit --> split  string Array as desired

use evestring() --> convert string to double

Point of disadvantage

So fa I know strsplit() can handle only one string.
Hence I use a for-loop to split each line of the Initial string array into a 
group of strings and convert each part into a double.

OK for few data...may take long for many data

Idea:
Read data from file and try to spare conversation string-to-double.
fscanfMat() won't do it, because the data does not contain only numerical 
values and "."-sign as decimal.
So I tried csvRead.
using the help I find:  separator :  a 1-by-1 matrix of strings, the field 
separator used.

Note:   STRINGS   = plural.

So I wonder if it is possible to have more than one separator here.

BR
Philipp

mclose( :-) )


___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Rafael Guerra]

Hello Philipp,

Say that after mopen you got all your text input into array of strings ‘txt’:

txt = [
"HEADER-Line",
"01.12.2015, 01:15:00.12, 1.1, -2.2";
"03.12.2015, 11:15:00.12, -11.1, 2.5";
"12.12.2015, 21:15:00.12, 5.1, 6.2"];

Then do the following:

tx=txt(2:$);  // get rid of header line
tx1=part(tx,1:24);  // get date and time
tx2=part(tx,25:$);  // get numeric data values
// Now get rid of separators:
tx1 = strsubst(tx1,'.',' ');
tx1 = strsubst(tx1,':',' ');
tx1 = strsubst(tx1,',',' ');
tx2 = strsubst(tx2,',',' ');
tx = [tx1 tx2]; // regroups all data but now with numeric values only

Use  mputl to output ‘tx’ to temporary disk file and use fscanfMat to read like 
a breeze the large disk file (now fully numeric) into a Scilab numeric matrix 
‘M’.

Regards,
Rafael

From: users [mailto:users-boun...@lists.scilab.org] On Behalf Of Philipp 
Mühlmann
Sent: Friday, October 14, 2016 11:50 AM
To: Users mailing list for Scilab <users@lists.scilab.org>
Subject: Re: [Scilab-users] using csvRead

Dear Denis,

yes thats the way I do it right now.

use mopen --> open file for reading
use mgetl --> read data, result = array of strings
use strsplit --> split  string Array as desired

use evestring() --> convert string to double

Point of disadvantage

So fa I know strsplit() can handle only one string.
Hence I use a for-loop to split each line of the Initial string array into a 
group of strings and convert each part into a double.

OK for few data...may take long for many data

Idea:
Read data from file and try to spare conversation string-to-double.
fscanfMat() won't do it, because the data does not contain only numerical 
values and "."-sign as decimal.
So I tried csvRead.
using the help I find:  separator :  a 1-by-1 matrix of strings, the field 
separator used.

Note:   STRINGS   = plural.

So I wonder if it is possible to have more than one separator here.

BR
Philipp

mclose( :-) )


___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[Philipp Mühlmann]

Dear Denis,

yes thats the way I do it right now.

use mopen --> open file for reading
use mgetl --> read data, result = array of strings
use strsplit --> split  string Array as desired

use evestring() --> convert string to double

Point of disadvantage

So fa I know strsplit() can handle only one string.
Hence I use a for-loop to split each line of the Initial string array into
a group of strings and convert each part into a double.

OK for few data...may take long for many data

Idea:

Read data from file and try to spare conversation string-to-double.

fscanfMat() won't do it, because the data does not contain only numerical
values and "."-sign as decimal.

So I tried csvRead.

using the help I find:  separator :  a 1-by-1 matrix of strings, the
field separator used.


Note:   STRINGS   = plural.

So I wonder if it is possible to have more than one separator here.




BR
Philipp

mclose( :-) )


2016-10-14 0:51 GMT+02:00 CRETE Denis <denis.cr...@thalesgroup.com>:

> Hello Philipp,
>
>
>
> In this case I first attempt to read first (e.g. with “mopen” and “getl”)
> and then split the strings with “tokens” where the feature of multiple
> separators exists.
>
> Of course you can also use “cvsRead” with one separator and then use
> “tokens” to finish with the other separators.
>
> HTH
>
> Denis
>
>
>
> [@@ THALES GROUP INTERNAL @@]
>
>
>
> Unité Mixte de Physique CNRS / THALES
>
> 1 Avenue Augustin Fresnel
>
> 91767 Palaiseau CEDEx - France
>
> Tel : +33 (0)1 69 41 58 52 Fax : +33 (0)1 69 41 58 78
>
> e-mail :
>
>  denis.cr...@thalesgroup.com <mailto:%20denis.cr...@thalesgroup.com
> <%20denis.cr...@thalesgroup.com>>
>
> http://www.trt.thalesgroup.com/ump-cnrs-thales
>
> http://www.research.thalesgroup.com
>
>
>
> *De :* users [mailto:users-boun...@lists.scilab.org] *De la part de*
> Philipp Mühlmann
> *Envoyé :* vendredi 14 octobre 2016 00:08
> *À :* International users mailing list for Scilab.
> *Objet :* [Scilab-users] using csvRead
>
>
>
> Dear Scilab users,
>
> having a data file (*.cvs) containg following format:
>
> HEADER-Line
>
> dd.mm., HH:MM:SS.MS, value01, value02
>
> dd = day
>
> mm = month
>
>  = year
>
> HH = hour
>
> MM = minute
>
> SS = second
>
> Ms = milli second
>
> ValueXY = numerical value
>
>
>
> Is it possible to use cvsRead in such a way to define the separator beeing
> ',' and ':' at the same time?
>
> Background:
>
> desired Matrix after reading the file is
>
> M = [dd mm  HH MM SS MS value1 value2]
>
>
>
> Thank you,
>
> Philipp
>
>
>
> --
>
> In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.
>
>
>
> There we have the salad.
>
> ___
> users mailing list
> users@lists.scilab.org
> http://lists.scilab.org/mailman/listinfo/users
>
>


-- 
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users

Re: [Scilab-users] using csvRead

[CRETE Denis]

Hello Philipp,

In this case I first attempt to read first (e.g. with “mopen” and “getl”) and 
then split the strings with “tokens” where the feature of multiple separators 
exists.
Of course you can also use “cvsRead” with one separator and then use “tokens” 
to finish with the other separators.
HTH
Denis

[@@ THALES GROUP INTERNAL @@]

Unité Mixte de Physique CNRS / THALES
1 Avenue Augustin Fresnel
91767 Palaiseau CEDEx - France
Tel : +33 (0)1 69 41 58 52 Fax : +33 (0)1 69 41 58 78
e-mail :
 denis.cr...@thalesgroup.com<mailto:denis.cr...@thalesgroup.com> 
<mailto:%20denis.cr...@thalesgroup.com>
http://www.trt.thalesgroup.com/ump-cnrs-thales
http://www.research.thalesgroup.com

De : users [mailto:users-boun...@lists.scilab.org] De la part de Philipp 
Mühlmann
Envoyé : vendredi 14 octobre 2016 00:08
À : International users mailing list for Scilab.
Objet : [Scilab-users] using csvRead

Dear Scilab users,
having a data file (*.cvs) containg following format:

HEADER-Line
dd.mm., HH:MM:SS.MS<http://SS.MS>, value01, value02

dd = day
mm = month
 = year
HH = hour
MM = minute
SS = second
Ms = milli second
ValueXY = numerical value

Is it possible to use cvsRead in such a way to define the separator beeing ',' 
and ':' at the same time?
Background:

desired Matrix after reading the file is
M = [dd mm  HH MM SS MS value1 value2]

Thank you,
Philipp

--
In Kanada is' ka' na' da. Sonst wär' Kanada Jemanda.

There we have the salad.
___
users mailing list
users@lists.scilab.org
http://lists.scilab.org/mailman/listinfo/users