Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Ronie Salgado]

Hi Holger,

Sorry for not noticing this thread before. Some months ago, I did some
experiments myself using inotify and epoll, with the UFFI. I was creating a
file system monitoring API similar in design to OSWindow. I just did some
minor changes for making this usable along with making a configure script.
I left all of the code and some scripts for loading the FileSystemMonitor
API at: http://smalltalkhub.com/#!/~ronsaldo/OSCalls .

For testing the monitoring API, try the following in a playground:

monitor := FileSystemMonitor for: '.' do: [ :event |
Transcript show: event; cr.
].

If you do this, open a Transcript and you start browsing some methods in
Nautilus, you will start receiving some events where Pharo is accessing the
.changes file. For the destroying the monitor and stop receiving the
events, you can do the following:

monitor destroy.

Currently the events given by this API are only holding a copy of the raw
inotify event in rawEvent and a copy of the raw path in rawPath. Further
work is required for translating the events into a more platform
independent interface, and for supporting OS X and Windows with this API.
However, this is already something that could be useful to you.

Best regards,
Ronie

2016-09-20 15:26 GMT-03:00 Holger Freyther :

> Hi,
>
> I explored to use the Linux inotify API using UFFI. The
> inotify_init/inotify_init1 routine will me a Unix filedescriptor and I
> would like to
>
> a.) monitor it for being readable
> b.) read from it with a stream
>
>
> For a.) I have found the AioEventHandler and think I will be able to call
> >>#descriptor: directly and then can use it (still figuring out the API,
> probably just wait for the >>#changed: call).
>
> For b.) I thought I could use AttachableFileStream but that required a
> "fileId" but not a filedescriptor. Is there a way I can read from my fd
> using the standard stream API (otherwise I can try to use UFFI for read)
>
>
> regards
> holger
>

Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Holger Freyther]

> On 22 Sep 2016, at 23:33, Mariano Martinez Peck  wrote:
> 
> Hi Holger,

Hey!

> 
> I just run all OSSubprocess tests and they all worked! (tested in Pharo 5.0). 
> 
> I guess I will commit this on the dev branch and hopefully when this is 
> integrated into the VM I can merge that for my next OSSubprocess release.

cool and thank you for trying it so quickly. I thought it make sense to mark 
non files the same as isStdioStream but that triggers a funny case:

do {
clearerr(file);
if (fread(dst, 1, 1, file) == 1) {
bytesRead += 1;
if (dst[bytesRead-1] == '\n'
 || dst[bytesRead-1] == '\r')
break;
}
} while (bytesRead <= 0 && ferror(file) && errno == EINTR);

which means 0 or 1 char is read with >>#primRead:into:startingAt:count: and in 
case of inotify the event is lost (partially read and the rest discarded). I 
will have to start a discussion why primRead should be line buffered at all.

Anyway. My good news is that:

"an INotify"
| arr fileID |
self init: 8r4000.
self prim_add_watch: '/tmp' flags: 16r0100.
arr := ByteArray new: 4096.
fileID := StandardFileStream new primFdOpen: fd writable: false.
StandardFileStream new
primRead: fileID into: arr startingAt: 1 count: 4096.
arr

has read a file notification event.. :)

Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Mariano Martinez Peck]

Hi Holger,

OK, I made a new VM with your PR after I reviewed it. Good news? It
works!!!  If you want to check OSSubprocess, I am using now the
primCreatePipe from OSProcess because that would answer me directly the
SQFiles of the pipe. Originally (before ending up doing that), I was trying
to make the pipes myself via FFI ( pipe() ) but I came to the problem we
discussed earlier (remember the #name:attachTo:writable:).

So now I can go back to my original solution. The code is in #
*makePipeWithReadBlocking*: and now it looks like this:

| pipePointer returnValue fileDescriptors pipe fileIDsArray fileDescriptor1
fileDescriptor2 |
pipePointer := ExternalAddress allocate: 8.
[
returnValue := self primitivePipe: pipePointer.
(returnValue = -1) ifTrue: [ self perror: 'pipe()' ].
fileIDsArray := Array new: 2.
fileDescriptor1 := pipePointer nbUInt32AtOffset: 0.
fileDescriptor2 := pipePointer nbUInt32AtOffset: 4.
fileIDsArray at: 1 put: *(self primitiveFileOpenUseFileDescriptor:
fileDescriptor1 writeFlag: false)*.
fileIDsArray at: 2 put: *(self primitiveFileOpenUseFileDescriptor:
fileDescriptor2 writeFlag: true).*
pipe := OSSPipe newWith: fileIDsArray readBlocking: aBoolean.
] ensure:[
pipePointer free.
].
^ pipe


I just run all OSSubprocess tests and they all worked! (tested in Pharo
5.0).

I guess I will commit this on the dev branch and hopefully when this is
integrated into the VM I can merge that for my next OSSubprocess release.

Thank you very much Holger




On Thu, Sep 22, 2016 at 12:35 PM, Mariano Martinez Peck <
marianop...@gmail.com> wrote:

>
>
> On Thu, Sep 22, 2016 at 12:12 PM, Holger Freyther 
> wrote:
>
>>
>> > On 21 Sep 2016, at 15:09, Mariano Martinez Peck 
>> wrote:
>> >
>> >
>> >
>> > Exactly. I have been wanting this a couple of times while doing
>> OSSubprocess.
>>
>>
>> https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you
>> could review it and give it a try. It adds two primitive (one to work on fd
>> one to work on FILE).
>>
>> I probably also want to do:
>>
>> sqFile->isStdioStream = isatty(fileno(file))
>>
>>
>> > Yes, exactly. I remember now. And as I said, I also wanted to be able
>> to work at fd or FILE* level and I failed.
>>
>> Could you give the above a try and then I try to get it into the
>> Opensmalltalk VM.
>>
>
> Hi Holger,
>
> I just reviewed the PR and it looks really good. Please, allow me some
> time to get updated to the VM compiling instructions and I will give it a
> try. Probably, I will be testing it by doing my original (unused now)
> #name:attachToCFile:writable:  use the new primitive and running the
> OSSubprocess tests. Does this make sense to you?
>
> Thanks a LOT for going deep and fix it!
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>



-- 
Mariano
http://marianopeck.wordpress.com

On Thu, Sep 22, 2016 at 12:35 PM, Mariano Martinez Peck <
marianop...@gmail.com> wrote:

>
>
> On Thu, Sep 22, 2016 at 12:12 PM, Holger Freyther 
> wrote:
>
>>
>> > On 21 Sep 2016, at 15:09, Mariano Martinez Peck 
>> wrote:
>> >
>> >
>> >
>> > Exactly. I have been wanting this a couple of times while doing
>> OSSubprocess.
>>
>>
>> https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you
>> could review it and give it a try. It adds two primitive (one to work on fd
>> one to work on FILE).
>>
>> I probably also want to do:
>>
>> sqFile->isStdioStream = isatty(fileno(file))
>>
>>
>> > Yes, exactly. I remember now. And as I said, I also wanted to be able
>> to work at fd or FILE* level and I failed.
>>
>> Could you give the above a try and then I try to get it into the
>> Opensmalltalk VM.
>>
>
> Hi Holger,
>
> I just reviewed the PR and it looks really good. Please, allow me some
> time to get updated to the VM compiling instructions and I will give it a
> try. Probably, I will be testing it by doing my original (unused now)
> #name:attachToCFile:writable:  use the new primitive and running the
> OSSubprocess tests. Does this make sense to you?
>
> Thanks a LOT for going deep and fix it!
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>



-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Mariano Martinez Peck]

On Thu, Sep 22, 2016 at 12:12 PM, Holger Freyther 
wrote:

>
> > On 21 Sep 2016, at 15:09, Mariano Martinez Peck 
> wrote:
> >
> >
> >
> > Exactly. I have been wanting this a couple of times while doing
> OSSubprocess.
>
>
> https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you
> could review it and give it a try. It adds two primitive (one to work on fd
> one to work on FILE).
>
> I probably also want to do:
>
> sqFile->isStdioStream = isatty(fileno(file))
>
>
> > Yes, exactly. I remember now. And as I said, I also wanted to be able to
> work at fd or FILE* level and I failed.
>
> Could you give the above a try and then I try to get it into the
> Opensmalltalk VM.
>

Hi Holger,

I just reviewed the PR and it looks really good. Please, allow me some time
to get updated to the VM compiling instructions and I will give it a try.
Probably, I will be testing it by doing my original (unused now)
#name:attachToCFile:writable:  use the new primitive and running the
OSSubprocess tests. Does this make sense to you?

Thanks a LOT for going deep and fix it!




-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Holger Freyther]

> On 21 Sep 2016, at 15:09, Mariano Martinez Peck  wrote:
> 
> 
> 
> Exactly. I have been wanting this a couple of times while doing OSSubprocess. 


https://github.com/pharo-project/pharo-vm/pull/108. Would be nice if you could 
review it and give it a try. It adds two primitive (one to work on fd one to 
work on FILE).

I probably also want to do:

sqFile->isStdioStream = isatty(fileno(file))


> Yes, exactly. I remember now. And as I said, I also wanted to be able to work 
> at fd or FILE* level and I failed. 

Could you give the above a try and then I try to get it into the Opensmalltalk 
VM.

holger

Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Mariano Martinez Peck]

On Wed, Sep 21, 2016 at 3:55 AM, Holger Freyther  wrote:

>
> > On 20 Sep 2016, at 20:38, Mariano Martinez Peck 
> wrote:
> >
> > Hi Holger,
>
>
> Good Morning,
>
> thank you for your reply.
>
>
Good morning.


> > | reader |
> > reader := OSSAttachableFileStream name:'myStream' attachTo: aFileID
> writable: false.
> > reader setNonBlocking "optional"
>
> the only issue is that i have a "int fd" and not a SQFile. In
> >>#name:attachToCFile:writable: you create a SQFile* out of a FILE*
> (OSSCFile) but judging the comment it doesn't work.



Thanks! That was the method I was trying to remember and I was failing!
hahahaha.



> The 32/64 bit issue can be solved by using FFIExternalStructure to model
> SQFile but the question if the VM was built with large file support on
> GNU/Linux is a tricky one.
>
>
Yes exactly. The basic scenario was working, but having it working in all
the different flavors of OS and VMs was tricky as the comment says. My
conclusion in this case was that it's not worth trying to do this from
image side via UFFI and that instead we should have a VM primitive.



> So maybe we create another primitive to convert a FILE* to a SQFile* (and
> have it manage the lifetime of that memory?)? And maybe another primitive
> to do the same for a Socket?
>
>
Exactly. I have been wanting this a couple of times while doing
OSSubprocess.


>
> > I still didn't understand why do you mean with a). What do you mean by
> "monitor it from being readable" ?
>
> Imagine you want to exit the image in case the file /exit changes. You
> will charge inotify to watch this filepath and if the fd becomes readable
> you already know the answer, you don't have to read the event. But true if
> I have a Socket or FileStream I can do blocking read on it as well.
>

OK, now I understand. But if you do blocking read, wouldn't be locking the
rest of the image by that read?  Of course, you can do busy waiting with
none blocking reads but for that, inotify may be more performant?


>
>
> > I think you could dig a bit in OSSPipe, OSSAttachableFileStream and
> their usage. All classes have class comments, all methods are also
> documented, and there is quite some documentation in [1].
>
> Will look again but I didn't see anything obvious. E.g. primCreatePipe
> seems to already return two SQFile* ("fileId")?
>
>
Yes, exactly. I remember now. And as I said, I also wanted to be able to
work at fd or FILE* level and I failed.

Cheers,


-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-users] Using a Unix filedescriptor in a FileStream?

[Holger Freyther]

> On 20 Sep 2016, at 20:38, Mariano Martinez Peck  wrote:
> 
> Hi Holger,


Good Morning,

thank you for your reply.

> | reader |
> reader := OSSAttachableFileStream name:'myStream' attachTo: aFileID writable: 
> false.
> reader setNonBlocking "optional"

the only issue is that i have a "int fd" and not a SQFile. In 
>>#name:attachToCFile:writable: you create a SQFile* out of a FILE* (OSSCFile) 
but judging the comment it doesn't work. The 32/64 bit issue can be solved by 
using FFIExternalStructure to model SQFile but the question if the VM was built 
with large file support on GNU/Linux is a tricky one.

So maybe we create another primitive to convert a FILE* to a SQFile* (and have 
it manage the lifetime of that memory?)? And maybe another primitive to do the 
same for a Socket?


> I still didn't understand why do you mean with a). What do you mean by 
> "monitor it from being readable" ?

Imagine you want to exit the image in case the file /exit changes. You will 
charge inotify to watch this filepath and if the fd becomes readable you 
already know the answer, you don't have to read the event. But true if I have a 
Socket or FileStream I can do blocking read on it as well.


> I think you could dig a bit in OSSPipe, OSSAttachableFileStream and their 
> usage. All classes have class comments, all methods are also documented, and 
> there is quite some documentation in [1]. 

Will look again but I didn't see anything obvious. E.g. primCreatePipe seems to 
already return two SQFile* ("fileId")?

cheers
holger