[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 Ettore Atalan changed: What|Removed |Added CC||atalantt...@googlemail.com -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 i.Dark_Templar changed: What|Removed |Added CC||idarktemp...@mail.ru -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #138 from bluescreenaven...@gmail.com --- I opened https://bugs.kde.org/show_bug.cgi?id=402298 If I set LANG differently, I DO get error prompts though. and I get more files copied. I'm not running kioclient5 headless, but in a case where it's running under a display, and I CAN get other dialogs... such as, when I forgot to set LANG back, I got actual messages telling me that it can't access those files... -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211
--- Comment #137 from David Edmundson ---
LANG definitely can make a difference to the way things run.
Filesystems don't have a concept of UTF-8 or anything else just a concept of a
raw bytestream. We then put that into a QUrl which obviously follows the locale
it runs in. Though the worst this should do is (correctly) fail on a file
conflict where one didn't previously exist.
I think your script is creating a byte array that isn't valid text in any
format. I can understand KIO failing to copy that because we don't treat things
as a byte stream all the way through.
However in your logs KIO isn't silent.
Your kioclient log ends with:
"Could not make folder
/tmp/kiocopy/1545096048/folder_dest/testdir/0l>╚>/1H{n.4N5>2ä<╚#uDJjà57vÉ7BFI*;9jÄ>w@Vdäâ2@Op*8w>i5-by4P{èåhÄÄDw9î1RÆ4x3©h23uÉè)
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #136 from David Edmundson --- Please make a new bug for a specific issue that we're trying to track. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 JonnyRobbie changed: What|Removed |Added CC||marc...@seznam.cz -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 Oleg changed: What|Removed |Added CC|s...@0leg.net | -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #135 from bluescreenaven...@gmail.com --- I have the full logs, and listings in the mentioned attachment, however the 3 bytes seem to be EF BF BD between the valid lower case 'g' and the '.' -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #134 from Christoph Feck --- You copied the replacement character, not the invalid raw byte that caused it. You might want to write the list of filenames from your script to a file, and inspect that with Okteta. It might be easier if you use one file per filename, e.g. FILENAME-01.txt contains 17 bytes, if the first filename is 16 bytes long etc. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #133 from bluescreenaven...@gmail.com --- (In reply to Christoph Feck from comment #132) > EDIT: ... need to see the raw bytes ... That should be in my "Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script" attachment: and then /tmp/kiocopy/1545096048/logs/kioclient.log, second to the last line... I tried to make sure I had the right byte copied and pasted, when piping into hexdump -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #132 from Christoph Feck --- EDIT: ... need to see the raw bytes ... -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #131 from Christoph Feck --- That character is just the replacement character for a non-UTF-8 sequence when converting to UTF-16 (which is what QString uses internally). You would need to raw bytes before that conversion. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #130 from bluescreenaven...@gmail.com --- (In reply to Christoph Feck from comment #128) > The filename limit is 255 bytes, not 255 characters. In UTF-8, any non-ASCII > character needs more than 1 byte. Additionally, the last character looks > cropped, causing an illegal UTF-8 name, which Qt does not handle. Yeah, that's the issue I was trying to fix in the script. Trying to find a way to count the BYTES in a variable with bash... as far as that unreadable char, this is what appears in Hexdump nerdopolis@nerdopolis:/tmp/kiocopy/1545096048$ echo -n �| hexdump 000 bfef 00bd 003 It could be the fact that the char is lopped, since that is not one of the random ones the script creates... ...but it apparently might have triggered the silent failure ...maybe? -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #129 from bluescreenaven...@gmail.com --- ALSO, for what it's worth, if I leave the two LC_ALL=C LANG=C variables set when running kioclient5, instead of setting them back with the script I get... -17 directories, 413 files +8 directories, 221 files Setting it back to LANG=en_US.UTF-8 LC_ALL= and I get the apparent silent failure (I set those variables in hopes of working around trying to fit Unicode chars in 255 bytes in the bash/generation portion...) I REALLY hope this info is helpful -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #128 from Christoph Feck --- The filename limit is 255 bytes, not 255 characters. In UTF-8, any non-ASCII character needs more than 1 byte. Additionally, the last character looks cropped, causing an illegal UTF-8 name, which Qt does not handle. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #127 from bluescreenaven...@gmail.com --- Created attachment 116981 --> https://bugs.kde.org/attachment.cgi?id=116981=edit Bash script of which to use to recreate the file tree that failed to copy. Pass the path to the Seed.txt to it. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #126 from bluescreenaven...@gmail.com --- Created attachment 116980 --> https://bugs.kde.org/attachment.cgi?id=116980=edit Archive containing the logs, and the tree, and the Seed.txt file to pass to the kiocopy script -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211
--- Comment #125 from bluescreenaven...@gmail.com ---
I THINK I JUST REPLICATED IT!
I should have been MUCH more aggressive with the File Sizes, but I was was
worried about off-by-1 errors causing my Script to attempt to create invalid
files, and not create enough.
So the script is somewhat incomplete trying to make 255 char files, it doesn't
generate as many files as I was before. (as Unicode chars were causing too
large names)
Copying with kioclient, with all the KIO logs on, there is a big line
"Could not make folder
/tmp/kiocopy/1545096048/folder_dest/testdir/0l>╚>/1H{n.4N5>2ä<╚#uDJjà57vÉ7BFI*;9jÄ>w@Vdäâ2@Op*8w>i5-by4P{èåhÄÄDw9î1RÆ4x3©h23uÉè)
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 David Edmundson changed: What|Removed |Added Status|RESOLVED|CLOSED -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 David Edmundson changed: What|Removed |Added Status|CONFIRMED |RESOLVED Resolution|--- |FIXED --- Comment #124 from David Edmundson --- As per #119 I am closing this report. There was an issue in kdelibs that was fixed many many years ago. Most comments here relate to that. It was reopened with unrelated reasons. There was also some mishandling in kuiserver which is fixed quite recently which could give misleading messages which confuses this bug. No attempts to reproduce a bug with up-to-date code has proved successful despite multiple efforts from multiple people. That isn't to claim kio is bug free, but it will be much more productive to treat that as a new bug with new information than to add data here which is full of fixed outdated noise. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 wg9rffujwz8y2...@airmail.cc changed: What|Removed |Added CC||wg9rffujwz8y2...@airmail.cc --- Comment #123 from wg9rffujwz8y2...@airmail.cc --- (In reply to David Edmundson from comment #119) > > >I wonder if any of the reports are because of Dolphin going down? > > Note that was fixed in 2beb1a0ad23177f7dc2e5ee622bed3a70f671278 in > plasma-workspace. I CC'd a different bug. https://bugs.kde.org/show_bug.cgi?id=401055 -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #122 from bluescreenaven...@gmail.com --- (In reply to David Edmundson from comment #121) > I like how you've ended up going down the same paths and patterns as I did ! > > Running stat on a directory is pretty FS independent, it's 0 bytes on NTFS. > ls sometimes puts different results too > > I did propose something https://phabricator.kde.org/D17071 but I'm not very > convinced. I used the tree command and the -s flag to show the sizes instead of the Properties dialog however. I saw all kinds of lines that were different on the BTRFS side. Instead of being fixed at 4096, there were lines that where coming up when I diffed the files, at say 0 bytes, 72 bytes, 54 bytes; etc. I got really concerned that I might have replicated it, but then I realized when I added the -F flag to append a / to directories, that every single line that was different was a directory, and all the files where the same So as it turns out, it looks like BTRFS also varies from EXT4 on how it measures the size of directories. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 David Edmundson changed: What|Removed |Added CC||k...@davidedmundson.co.uk --- Comment #121 from David Edmundson --- I like how you've ended up going down the same paths and patterns as I did ! Running stat on a directory is pretty FS independent, it's 0 bytes on NTFS. ls sometimes puts different results too I did propose something https://phabricator.kde.org/D17071 but I'm not very convinced. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #120 from bluescreenaven...@gmail.com --- Cool. I will also add that ran a second test with my large tree generating script script to check all the files have all the correct sizes. At first I saw some differences, but then I realized that I was copying from a test ext4 image to a test btrfs image. The lines with the different sizes in my diff were all *directories* and no files were clipped... I guess not all dirs in BTRFS are 4096 bytes like in EXT4 -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #119 from David Edmundson --- Your correct in your analysis of how KIO works. I did KIO speedups last week that should somewhat reduce the CPU and memory on the file.so slave for many files. >I wonder if any of the reports are because of Dolphin going down? Note that was fixed in 2beb1a0ad23177f7dc2e5ee622bed3a70f671278 in plasma-workspace. I CC'd a different bug. The history of this bug is quite messy, most reports date back to before the potentially important e60e8b96b211290e330c7ff6a3b84b20ab85850d on kdelibs handling of inaccessible folders. This bug was reopened primarily from someone citing copying from a zip which is an entirely different story. There's not a lot to suggest we still have the same issue as we might have had when that bug was opened. Then there's also noise from people not unmounting drives and potentially some reports from kuiserver handling of the host app crashing that I just mentioned which I fixed only very recently. I've tested copying a million files repeatedly, bluescreenavenger has done the same. -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #118 from David Edmundson --- *** Bug 352761 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #117 from bluescreenaven...@gmail.com --- One odd thing I notice during the test, is that the kioslaves have the high CPU usage, kioclient5 has the high memory usage. I assume that means kioclient5 stores the list of files to copy, and then it must make the kioslave do the copying. My theory was maybe a process (possibly a helper is crashing on the reporters computers, such as kio_file.so) So I tried a smaller, manual test with Dolphin. Copying a smaller directory of ~24 files, Dolphin's memory usage grows a bit (not a lot). So that confirms my theory that Dolphin must stay alive to tell the kioslaves what file to copy. Killing the kioslave that the CPU usage spiked with did NOT result in a silent failure. I got a dialog warning that the file protocol died, and it gave me some options to recover which it recovered from fully. So it's not silently failing because of that. HOWEVER trying again, I went after Dolphin this time. I closed the Dolphin window. The Dolphin process smartly stayed running to continue the copy (so the issue is not the Dolphin window being clsed). As a test, to simulate a crash or abrupt exit of Dolphin, I killed the Dolphin process. Sure enough the copy stopped. ...and I got a "The copy has completed successfully" notification. I wonder if any of the reports are because of Dolphin going down? -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 --- Comment #116 from bluescreenaven...@gmail.com --- I made a script that makes 1,000,000 randomly placed files, with random names, with various sizes from 1kb to 1gb, into 1,000,000 randomly placed folders, with random names. It selects 3 random other directories to fill them with 100,000 additional files each as well. Then it uses kioclient5 to copy the items. https://raw.githubusercontent.com/n3rdopolis/otherstuff/master/kiocopy The names have kinds of random numbers, symbols, some are 20 chars long, some have weird symbols that try to break parsing. It then lets me diff the trees. I I tried it on my KDE frameworks 5.44 box... ...but after all that, I was NOT able to replicate it. the trees match. I have a log, and a seed file for the script, but the .tar.xz is 83MB, and it's quite worthless because it worked perfectly anyway. I am at a loss at how to replicate it. maybe I should change the outlier generated maximum file name size to 220 chars? -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 taocris...@hotmail.com changed: What|Removed |Added CC||taocris...@hotmail.com -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 alex...@gmx.net changed: What|Removed |Added CC||alex...@gmx.net -- You are receiving this mail because: You are watching all bug changes.
[frameworks-kio] [Bug 162211] Copying to an external causes lots of missing files
https://bugs.kde.org/show_bug.cgi?id=162211 Julian Schraner changed: What|Removed |Added Assignee|kdelibs-b...@kde.org|fa...@kde.org Component|general |general CC||kdelibs-b...@kde.org Product|kdelibs |frameworks-kio -- You are receiving this mail because: You are watching all bug changes.
