Re: How to detect EOL character in text file (James Crate)

2017-05-17 Thread Jim Medlen via 4D_Tech 

<>vCR:=Char(13)

<>vLF:=Char(10)

<>vCRLF:=Char(13)+Char(10)


vText:=Replace string(vText;<>vCRLF;<>vCR)
vText:=Replace string(vText;<>vLF;<>vCR)

When parsing text I replace Carriage Return + Line Feed with Carriage Return
Then replace Line Feed with carriage return.

Then the Line endings are all Carriage Returns.

Thanks,

Jim Medlen
Computer & Information Systems
Functional Devices, Inc.
j.med...@functionaldevices.com
phone (765) 883-5538 x 428
fax (765) 883-4262
http://www.functionaldevices.com

This email was transmitted on 100 percent recycled electrons




**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-15 Thread Alan Chan via 4D_Tech 
Replace String($text;lf;cr) -> all lf becomes cr including crlf becoming crcr
Replace string($text;crlf;cr) -> none would be found

That's why you need to run Replace String($text;crlf;cr) first.

Alan Chan

4D iNug Technical <4d_tech@lists.4d.com> writes:
>That should have read replace string of course
>if you change all the possible EOL characters to CR, then position will
>find all the CRs
>or have I missed something?
>pm
>
>On Sat, May 13, 2017 at 8:30 PM, James Crate via 4D_Tech <
>4d_tech@lists.4d.com> wrote:
>
>> On May 13, 2017, at 8:53 AM, Peter Mew via 4D_Tech <4d_tech@lists.4d.com>
>> wrote:
>> >
>> > Replace text($text;lf;cr)
>> > Replace text($text;crlf;cr)
>> > Position($text;cr)
>> > -pm
>>
>> This won’t detect the line endings, which is what was asked, it will
>> change them to CR. However, to properly normalize line endings you need to
>> replace CRLF first.
>>
>> Jim Crate
>

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-14 Thread Peter Mew via 4D_Tech 
That should have read replace string of course
if you change all the possible EOL characters to CR, then position will
find all the CRs
or have I missed something?
pm

On Sat, May 13, 2017 at 8:30 PM, James Crate via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> On May 13, 2017, at 8:53 AM, Peter Mew via 4D_Tech <4d_tech@lists.4d.com>
> wrote:
> >
> > Replace text($text;lf;cr)
> > Replace text($text;crlf;cr)
> > Position($text;cr)
> > -pm
>
> This won’t detect the line endings, which is what was asked, it will
> change them to CR. However, to properly normalize line endings you need to
> replace CRLF first.
>
> Jim Crate
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread James Crate via 4D_Tech 
On May 13, 2017, at 8:53 AM, Peter Mew via 4D_Tech <4d_tech@lists.4d.com> wrote:
> 
> Replace text($text;lf;cr)
> Replace text($text;crlf;cr)
> Position($text;cr)
> -pm

This won’t detect the line endings, which is what was asked, it will change 
them to CR. However, to properly normalize line endings you need to replace 
CRLF first.

Jim Crate

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Arnaud de Montard via 4D_Tech 

> Le 13 mai 2017 à 14:04, David Adams via 4D_Tech <4d_tech@lists.4d.com> a 
> écrit :
> 
> On Sat, May 13, 2017 at 9:47 PM, Keisuke Miyako via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> 
>> if you are on v14 or later,
>> "Document to text" will normalise the end-of-line to whichever one you
>> prefer.
>> 
> 
> What a great feature! Thanks for pointing it out...I'd never noticed it.

I don't like this, for my own. Sounds like: "Hey guy, we're on macOS? Let's bet 
this file is CR!". Once it took me a while to understand a bug was due to Text 
to document without $3 explicit. BTW I seem to remember this is how excel works 
on drag'n drop. 

Text document is not xml, the inner structure should be identified before 
processing it. If it comes from an unknown source, better to have a preview 
were the user can adjust these parameters. 

-- 
Arnaud de Montard 



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Arnaud de Montard via 4D_Tech 

> Le 13 mai 2017 à 16:01, Arnaud de Montard via 4D_Tech <4d_tech@lists.4d.com> 
> a écrit :
> 
[...]

Previous code modified to evaluate if time taken to add more and more memory is 
linear. 
It seems not, each step "costs" a little more than the previous one (macOS). 

+++
C_BLOB($_x)
$Mbyte_l:=1*10^6
SET BLOB SIZE($_x;$Mbyte_l)
ARRAY BLOB($_ax;0)
ARRAY LONGINT($_al;0)
$i_l:=0
APPEND TO ARRAY($_al;Tickcount)
Repeat 
$i_l:=$i_l+1
If ($i_l%1000=0)
APPEND TO ARRAY($_al;Tickcount)  //time to add previous 1000 
elements
End if 
APPEND TO ARRAY($_ax;$_x)
Until ($i_l>MAXINT)
APPEND TO ARRAY($_al;Tickcount)  //total time
+++

(subtracting each $_al value to previous)
114
129
126
141
139
144
147
156
188
200
204
209
218
222
225
234
232
236
240
243
246
249
255
256
260
294
269
299
289
288
299
298
229

-- 
Arnaud de Montard 




**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

​Re: How to detect EOL character in text file

2017-05-13 Thread steve simpson via 4D_Tech 
On Sat, May 13, 2017 at 9:04 AM,
​
​
Peter Mew  wrote:
>
>
> Replace text($text;lf;cr)
> Replace text($text;crlf;cr)
> Position($text;cr)
>

​+1. I find this the fastest and simplest method of handling high volumes
of incoming "unknown" or "known to be dirty" or "suspect" text. (And not
just to find an EOL.) Plus you can't always use something like ​"Document
to text" as the initial spigot point. And by the way, some incoming text
contains actual text like "/r" that does not immediately equate to char(13)
assumed above, which is just as easily handled with "replace text".


Steve Simpson
​Cimarron Software
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Arnaud de Montard via 4D_Tech 

> Le 13 mai 2017 à 15:04, Jörg Knebel via 4D_Tech <4d_tech@lists.4d.com> a 
> écrit :
> 
>> On 13 May 2017, at 22:58 AEST, Jörg Knebel via 4D_Tech 
>> <4d_tech@lists.4d.com> wrote:
>> if the content bigger than 1GB? I don’t think TEXT could handle it.
> 
> 4D v16 64bit can do big things with texts too, can it not?
> 
> What’s the size limit for TEXT (var/filed) in v16-64???

a/ in 32 bits, Document to text with a 500K document seems to be enough. 

b/ in 64 bits

using this code (one block):
+++
C_BLOB($_x)
$Mbyte_l:=1*10^6
$i_l:=0
Repeat 
$i_l:=$i_l+1
$size_l:=$i_l*$Mbyte_l
SET BLOB SIZE($_x;$size_l)
ASSERT(BLOB size($_x)=$size_l)
Until ($size_l>MAXLONG)
+++
I get a "not enough memory" error when $i_l=1,074 - very close to MAXLONG/2… 
With a text document the limit seems to be similar. 

using this code (chunk in array):
+++
C_BLOB($_x)
$Mbyte_l:=1*10^6
SET BLOB SIZE($_x;$Mbyte_l)
ARRAY BLOB($_ax;0)
$i_l:=0
Repeat 
$i_l:=$i_l+1
APPEND TO ARRAY($_ax;$_x)
Until ($i_l>MAXINT)
+++
I reach the end of the loop with >7Gbyte of physical memory used - besides my 
good old macPro fans are complaining  ;-)

So, 
- the limit 4D can accept seems to be about half a Gbyte
- if I had to manage huge texts, I'd split them in a text array

-- 
Arnaud de Montard 



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Jörg Knebel via 4D_Tech 

> On 13 May 2017, at 22:58 AEST, Jörg Knebel via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> if the content bigger than 1GB? I don’t think TEXT could handle it.



4D v16 64bit can do big things with texts too, can it not?

What’s the size limit for TEXT (var/filed) in v16-64???
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Jörg Knebel via 4D_Tech 

> On 13 May 2017, at 21:47 AEST, Keisuke Miyako via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> if you are on v14 or later,

I’m on v16


> "Document to text" will normalise the end-of-line to whichever one you prefer.
> 
> http://doc.4d.com/4Dv15/4D/15.4/Document-to-text.301-3274243.en.html 
> 
> 
> it's better than using regex not knowing what the end-of-line is in the 
> loaded text.

Cool, worth to consider but what if the content bigger than 1GB? I don’t think 
TEXT could handle it.


Oh, just in case I forgot, thanks for the PDF-Manuals.
Even if the bookmark column is missing it’s still better than that html-crap

Cheers
Jörg

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Peter Mew via 4D_Tech 
Replace text($text;lf;cr)
Replace text($text;crlf;cr)
Position($text;cr)
-pm


Sent from my iPad

> On 13 May 2017, at 09:16, Jörg Knebel via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> G’day Guys,
> 
> I’m not sure if this was already a topic in the past, at least I couldn’t 
> find anything in my 174-month-Nug-Postings-archive.
> 
> BBEdit knows 3 EOL-“Markers”
> 
>Mac(legacy)CR 
>UnixLF
>WindowsCRLF
> 
> Having a bit of a singularity in my human CPU after a very long day here is 
> the question.
> 
> Given an unknown text-file, how can one detect the correct EOL 
> programmatically in 4D?
> 
> 
> Thanks
> 
> Cheers
> Jörg 
> 
> 
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread David Adams via 4D_Tech 
On Sat, May 13, 2017 at 9:47 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> if you are on v14 or later,
> "Document to text" will normalise the end-of-line to whichever one you
> prefer.
> 
>

What a great feature! Thanks for pointing it out...I'd never noticed it.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Keisuke Miyako via 4D_Tech 
if you are on v14 or later,
"Document to text" will normalise the end-of-line to whichever one you prefer.

http://doc.4d.com/4Dv15/4D/15.4/Document-to-text.301-3274243.en.html

it's better than using regex not knowing what the end-of-line is in the loaded 
text.

2017/05/13 17:16、Jörg Knebel via 4D_Tech 
<4d_tech@lists.4d.com> のメール：
Given an unknown text-file, how can one detect the correct EOL programmatically 
in 4D?



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How to detect EOL character in text file

2017-05-13 Thread Wayne Stewart via 4D_Tech 
Jörg,

I'd just look and see.

Grab a bunch of text say less than 1k

Then use position (with the *) and see if you can find crlf, not there Look
for cr or LF then.

Wayne

PS. Are you coming to Sydney world tour?

On Sat, 13 May 2017 at 18:16, Jörg Knebel via 4D_Tech <4d_tech@lists.4d.com>
wrote:

> G’day Guys,
>
> I’m not sure if this was already a topic in the past, at least I couldn’t
> find anything in my 174-month-Nug-Postings-archive.
>
> BBEdit knows 3 EOL-“Markers”
>
> Mac(legacy) CR
> UnixLF
> Windows CRLF
>
> Having a bit of a singularity in my human CPU after a very long day here
> is the question.
>
> Given an unknown text-file, how can one detect the correct EOL
> programmatically in 4D?
>
>
> Thanks
>
> Cheers
> Jörg
>
>
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

-- 

Regards,

Wayne


[image: --]
Wayne Stewart
[image: http://]about.me/waynestewart

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**