Re: Getting file extensions [linux fs]
Às 01:35 de 06/04/19, Pablo Lucena escreveu: > Have you looked into eBPF? I'll take a look at that. Thanks Pablo. -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting file extensions [linux fs]
Have you looked into eBPF? They have mature Python bindings. It makes interacting with the kernel as efficient as possible - you can run it in production at high resolutions without putting things at risk. One of the benefits - any averaging / aggregation / histograms / etc can be done by the kernel within the eBPF runtime, then passed back to python using eBPF "maps" - the link between your userspace program and the eBPF kernel code. From python this "map" is just a dict. You'll need to be running at least kernel version 4.4 to get the basic functionality, but ideally 4.9 or higher for all the newer and stable features. No dependencies, its baked into the kernel. You will need clang support to compile stuff, if you want to build modules on your own. *Pablo Lucena* On Sat, Mar 30, 2019 at 8:30 PM Paulo da Silva < p_s_d_a_s_i_l_v_a...@netcabo.pt> wrote: > Às 22:18 de 28/03/19, Cameron Simpson escreveu: > > On 28Mar2019 01:12, Paulo da Silva > wrote: > >> Às 23:09 de 27/03/19, Cameron Simpson escreveu: > ... > > > > > Oh, just tangential to this. > > > > If you were doing this ad hoc, yes calling the filefrag executable is > > very expensive. But if you are always doing a large batch of filenames > > invoking: > > > > filefrag lots of filenames here ...> > > and reading from its output can be very effective, because the expense > > of the executable is amortized over all the files - the per file cost is > > much reduced. And it saves you working out how to use the ioctls from > > Python :-) > That's not the case. > I need to do it on some files basis which I don't know in advance. > Using IOCTL, I don't need to parse or unpack the output. Only compare > the output arrays. Besides I need to store many of the outputs. Doing > that from filefrag text output would be unpractical. I needed, at least, > to compress the data. Although may be I might have to compress the ioctl > arrays ... Let's see how big in average is the storage needed. > > I have to go with ioctl. I have to open the files anyway, so there is no > overhead for that when calling the ioctl. > > Anyway, thank you for the suggestion. > > Regards. > Paulo > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting file extensions [linux fs]
Às 22:18 de 28/03/19, Cameron Simpson escreveu: > On 28Mar2019 01:12, Paulo da Silva wrote: >> Às 23:09 de 27/03/19, Cameron Simpson escreveu: ... > > Oh, just tangential to this. > > If you were doing this ad hoc, yes calling the filefrag executable is > very expensive. But if you are always doing a large batch of filenames > invoking: > > filefrag lots of filenames here ...> > and reading from its output can be very effective, because the expense > of the executable is amortized over all the files - the per file cost is > much reduced. And it saves you working out how to use the ioctls from > Python :-) That's not the case. I need to do it on some files basis which I don't know in advance. Using IOCTL, I don't need to parse or unpack the output. Only compare the output arrays. Besides I need to store many of the outputs. Doing that from filefrag text output would be unpractical. I needed, at least, to compress the data. Although may be I might have to compress the ioctl arrays ... Let's see how big in average is the storage needed. I have to go with ioctl. I have to open the files anyway, so there is no overhead for that when calling the ioctl. Anyway, thank you for the suggestion. Regards. Paulo -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting file extensions [linux fs]
On 28Mar2019 01:12, Paulo da Silva wrote: Às 23:09 de 27/03/19, Cameron Simpson escreveu: On 27Mar2019 21:49, Paulo da Silva wrote: ... The filefrag manual entry says it works by calling one of 2 ioctls. You can do that from Python with the ioctl() function in the standard fcntl module. I haven't tried to do this, but the results should be basicly as fast as filefrag itself. You'll need to decode the result the ioctl returns of course. Thanks Cameron, I'll take a look at that. Oh, just tangential to this. If you were doing this ad hoc, yes calling the filefrag executable is very expensive. But if you are always doing a large batch of filenames invoking: filefrag lots of filenames here ... and reading from its output can be very effective, because the expense of the executable is amortized over all the files - the per file cost is much reduced. And it saves you working out how to use the ioctls from Python :-) You'll learn more from going the ioctl route though, and it gives you complete control if you need it. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting file extensions [linux fs]
Às 23:09 de 27/03/19, Cameron Simpson escreveu: > On 27Mar2019 21:49, Paulo da Silva wrote: ... > The filefrag manual entry says it works by calling one of 2 ioctls. You > can do that from Python with the ioctl() function in the standard fcntl > module. I haven't tried to do this, but the results should be basicly as > fast as filefrag itself. > > You'll need to decode the result the ioctl returns of course. > Thanks Cameron, I'll take a look at that. -- https://mail.python.org/mailman/listinfo/python-list
Re: Getting file extensions [linux fs]
On 27Mar2019 21:49, Paulo da Silva wrote: I don't know if this is the right group to ask ... sorry if it isn't. Is there a way to get the file extensions of a file in linux, the same way as "filefrag -e " does? The purpose is to see if two files are the same file, namely those copied with the --reflink option in btrfs. A solution for C is also welcome - I can write a small python extension to handle that. BTW, executing filefrag from python is not a solution, because I have lots of files and that would be too slow. The filefrag manual entry says it works by calling one of 2 ioctls. You can do that from Python with the ioctl() function in the standard fcntl module. I haven't tried to do this, but the results should be basicly as fast as filefrag itself. You'll need to decode the result the ioctl returns of course. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Getting file extensions [linux fs]
Hi! I don't know if this is the right group to ask ... sorry if it isn't. Is there a way to get the file extensions of a file in linux, the same way as "filefrag -e " does? The purpose is to see if two files are the same file, namely those copied with the --reflink option in btrfs. A solution for C is also welcome - I can write a small python extension to handle that. BTW, executing filefrag from python is not a solution, because I have lots of files and that would be too slow. Thanks for any help. Anyway, as last resource, I can always look at filefrag source. I am just trying to avoid that. -- https://mail.python.org/mailman/listinfo/python-list
