Re: [crossfire] Handling massive loot
On Tue, 6 Apr 2021, nicolas.wee...@laposte.net wrote: The problem, as I see it, is that there is a need for scripting on the client in the first place... CF is, for me at least, a RPG game, not a script-making game... So let's figure WHY we need scripting, to fix that, instead of fixing the lack of documentation. Hi Nicholas. I'd like to highlight the difference between *writing* scripts and *using* scripts. I agree that no one should have to write scripts in order to play. That you can write scripts is a feature but it should never be required. Using scripts though could be transparent to the user. Perhaps the client could maintain a standard location for scripts and automatically install a series of scripts there the first time it starts. These scripts would essentially be transparent to the user but power users would still have the ability to extend the scripts or produce new ones. I was thinking of extending to "knowledge attempt* " to attempt as long as the player got ingredients in stock. This way, no need for scripting... To the extent that you can predict how scripting will be used. Even if you make the extensions you're talking about I'd urge the devs to retain client side scripting. Some of us find it very useful. Rob ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
On Sat, 27 Mar 2021, Nathaniel Kipps wrote: Case in point, I've played the game off and on for years, and still haven't figured out how to script properly. This is the problem. Even experienced players have issues with Crossfire scripting. My concern is that scripting is under-documented. Scouring the Internet years ago I found that some people had developed some quite sophisticated Crossfire scripts but almost none of them had been distributed beyond being mentioned in blog posts and the like. Everyone is welcome to any scripts I've developed. None of them are particularly complex. A few more complex examples can be found at the link below. http://murga-linux.com/puppy/viewtopic.php?t=80525 Rob ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
I totally agree that my proposed fix is a pretty big hammer to fix the problem, but it is a simple way to do it. It might not be that hard to add some logic (even on the server) of something like like 'repeat last command until successful/complete'. So you could do a dropall, and it drops 10 items/tick, and keeps doing so until there is nothing more to drop. Same for pickup (though in that case, there would be a change in behavior, as you would have to sit on the space until nothing is left to be picked up, and this might be non obvious if you have different pickup modes set (value density), because there might still be a bunch of items on the space, but nothing that matches the pickup criteria. Now this actually works reasonably well - at each players tick turn, the socket can check if the player has issued any new command - if so, it executes that, thus breaking the sequence of repeating the command. To me, it would be really annoying to do something like 'drop all', and be frozen with no way of interrupting it until everything is dropped, which could be many seconds if the limit/tick is low. As far as trees/linked lists - I was thinking more on how things are organized on the map, not queuing things up to dropped. If map spaces were trees instead of lists, it is possible that the finding matching object to merge would be fast enough that dropping lots of objects would not be an issue. In fact, I don't actually think that the time go through the inventory to find matching items to drop/pickup is the issue - I think the actual issue is the time to insert them into their destination. So in case of a pickup/dropall that happens over several ticks, just having the exact same logic now (looking at source for object that matches criteria) is probably fine, and there isn't any need to build a temporary list of objects that match the criteria. Building a temporary list has potential problems - mostly make sure it stays in sync, making sure it is properly cleared/initialized each time (if there is some bug where it isn't cleared, such that the player does a dropall, does something else that changes inventory, and does the dropall again, if the server thinks this is a continuation of the existing dropall because something wasn't cleared, good chance you are going to get some odd behavior) ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
On Wed, 24 Mar 2021, Nathaniel Kipps wrote: This is definitely a very blunt fix, and thus has blunt consequences. For example, let's say a player *did* need to pick up 1000 items. If they're limited to 10 items per tick, and the operation returns after those ten items, they'll have to issue "get all" 100 times before all the items are retrieved. I think it's definitely better to only inflict a time penalty, not a time penalty + UI penalty. This is probably a good time to remind everyone that CF supports scripting client and server side. I feel like this isn't mentioned enough. I've written scripts that could be adapted to deal with that. Eg, here is a simple shell script to pray a number of times: $less praylots #!/bin/bash case $1 in "") echo "Please specify a number of times to pray" exit 10 ;; *) NUM=$1 ;; esac for I in $(seq $NUM) do echo "issue 1 1 use_skill pray" done I'm not really commenting on Mark's proposal but faced with this situation I would script it. Rob ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
MWedel, thanks for the feedback. :) > The game could then display something like 'you have picked up/dropped all > you can right now', and player then has to repeat that action on the next > tick. This is definitely a very blunt fix, and thus has blunt consequences. For example, let's say a player *did* need to pick up 1000 items. If they're limited to 10 items per tick, and the operation returns after those ten items, they'll have to issue "get all" 100 times before all the items are retrieved. I think it's definitely better to only inflict a time penalty, not a time penalty + UI penalty. > Linked lists are also not particularly efficient when dealing with having to > search them I understand that trees are much better for random sorts, insertions, and lookups, but I'll point out that I proposed a linked list simply because it will only ever have items appended to the end, or processed in a linear fashion. Although, maybe there's a penalty to deleting items from the front of the list or something. --Nathan ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
I should just add, after the above: TLDR: simplest short term solution is limit pickup/drop to X items/action (tick). X could be a setting, so one can play around to what seems like a good value. This would require fairly minimal change to the code (just the pickup and drop logic to track how many have been dropped that tick, and then return once it hits the limit) The game could then display something like 'you have picked up/dropped all you can right now', and player than has to repeat that action on the next tick. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
So I had thought that in an ideal rewrite, there is 1 thread per map, and 1
thread per player (and maybe a few others that do stuff in the background).
That keeps things relative simple - you just have per map and per player thread
locks - if the thread has the lock, it can do whatever it wants.
This also helps crossfire use multiple cores of the cpu.
For maps, it makes the most sense, because right now, one slow map slows the
entire game down (be that someone dropping a pile of items in a store, or some
map with lots of spell effects). Each being contained to its own thread (map)
means that people on that map see the lag, but other players don't. And
certain map operations (load/save) are also known to take a while.
To me, it makes more sense to let the system library deal with thread
scheduling vs implementing some scheduler within crossfire itself. Threads are
a fairly cheap resource, and crossfire still would not need a lot of them.
Nathan is correct in that the problem is that until the drop (or pickup)
command completes, the server is not doing anything else. The approach he
presents would also work (and not require threading) - building a list of items
to drop, and dropping X of those/tick (X could be more than none). But he is
also correct that depending on how much stuff is on the square affects
performance. Dropping 1 item on a space that has 1000 items already will be
slow, and dropping 20 items on a space that is completely empty may be pretty
fast.
One way this could be handled is tracking the number of objects on a space, and if that
object count is >X, disallow dropping of more items there ("this area is overflowing
with stuff, and you just can't find an area to drop that %s")
There is still the problem with pickup - and when going through cleaning out a
dungeon, the players inventory can get quite long, especially since there tends
to be many variations of an item. Limiting pickup to X items/tick helps out
here, but could also be annoying, as you move over a space, not everything is
picked up, move back, move off again, notice not everything is picked up again,
repeat as needed.
Some of the problem as noted above is just the sheer number of items - while
having lots of variation is interesting, having 5 different types of swords
with minor variations, and then those 5 swords could have slightly different
artifact (of Lythander, etc) values, and add that they could then also have
different +magic values, and those 5 swords can turn into 100 variations of an
item, which increases the item count when dealing with merges.
Linked lists are also not particularly efficient when dealing with having to
search them - using something like a tree (organized on merging criteria) could
greatly speed up the insertions if it is well balanced, though can be a bit
less efficient when walking it.
Another thought would be rather than dumping items onto the linked list of the
space and marking that it needs to be sorted, there is in fact a different
linked list there contains the 'incoming' items - by the fact it exists means
those items on that incoming list are not sorted, and need to get added/merged
to the normal list. But this type of deferral gets complicated - you now have
to deal with the case where the player wants to pick up an item they just
(mistakenly) dropped - there are now 2 lists to look for.
In some ways, not merging items on the ground actually makes sense - things
don't automatically group together in real life, so having a space with '10
arrows, 3 swords, 4 arrows, ..' actually makes some sense.
IIRC, another performance issue (hack) was for the shop menus, where it
basically would use the merging logic to generate a more concise list, but when
people go unload 1000 items into the shop, it means that the logic it uses
(making a new list, copying objects onto it, etc) is not very efficient.
___
crossfire mailing list
crossfire@metalforge.org
http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
> slowing down the command processing from the client doesn't help out. If I understand correctly, the problem is less that commands are not processed quickly enough, and more that the drop command must be fully processed before the server can tick again. Here's a different (and probably flawed) approach: If a drop command is received, build a linked list of items that match the command, and drop the first item. Defer the rest of the list, and process the rest of the tick. During the next tick, pull the next item in the list, and drop it, then defer the rest for later. If the character moves, the list is discarded, cancelling the rest of the "transaction." This has several effects: Drop/pickup of lots of items now is capped at 1 item per tick, or 8 per second. Drop/Pickup of 1 item is still "free", it happens in the same tick. Drop/pickup speed could improve in the future if server tickrate is improved. Players are (quietly) encouraged to not pile items so deep, or manipulate them in bulk. > This becomes an O(n^2) operation, so when dropping a few items, it is pretty > fast, but when dropping a stack of 100 different items, that takes a lot of > time. Strictly speaking, isn't it n^2 only for dropping on an empty tile? Dropping items on a tile that already has lots of items is even worse, with a moderate perf penalty even for dropping a single item onto a pile of 100+ items. Not much way around that though, unless you defer the restacking to another thread or tick. But, that sounds quite complex to implement... --Nathan ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
On 2021-03-23 00:27, Mark Wedel wrote: On 3/22/21 11:58 AM, Nicolas Weeger wrote: Hello. One issue that happens is server delays when dealing with big loot. For instance a player selling some thousands rods can stuck the server for some seconds... Or even dropping many items on a big pile can be quite slow. So to handle that correctly, I was thinking of changing the server code like that: - split data/commands reception and processing - have one loop that reads from the socket, put commands in a waiting queue, then another function later processes those commands - for commands like pick/drop/examine, enable them to work on multiple processing loops instead of a single. So for instance the command would drop 100 items, then store its status, put itself back on top of the command waiting queue, and exit - this way the server can process something else. Being multi threaded would help some things out. It has been a while, by my recollection is the drop (and pickup) problem is that these are in fact handled by the server, eg, 'drop all' is done on the server, dropping everything of matching criteria. And likewise, pickup all is done on the server, so slowing down the command processing from the client doesn't help out. It is also my recollection in that the reason this is slow that each time an object is moved (ground to player or vice versa), it has to check the destination for duplicates to merge them (otherwise, you could have 20 different entries for silver coins). This becomes an O(n^2) operation, so when dropping a few items, it is pretty fast, but when dropping a stack of 100 different items, that takes a lot of time. There are a couple ways to deal with this: - A thought I had a while back is once a space gets too many items on it, items fall over to neighboring spaces - this would reduce the upper threshhold over that operation (never too many items to examine on a space), but this could basically result in an entire shop being filled to a level of 25 items deep. - For items going to the floor, marking the space as needing 'merging' later on could be done (one could even imagine something like a shop keeper standing on the space as this merging is done, preventing players from interacting with it) - For items going into the players inventory (they step onto a pile of junk), deferred merging could be done, but would result in new items getting send to the client, and then and update down the road that the new item does not exist and in fact item X has a different quantity. I'm not sure how visually that works out. - Maybe limit the number of items picked up at once, even with pickup all, to something like 25 or other number where performance is still reasonable. This makes it harder to clean up everything in the dungeon, but is somewhat more realistic (you can only pick up stuff so quickly). - I can't remember if the gold insertion when selling a bunch of items is intelligent. That is to say, if you sell 100 items, does is calculate the net proceeds and inserts it once (intelligent) or inserts the coinage 100 items (1/each item, keeping in mind that there could actually be several different coins being inserted for each item). So changing that (deferred gold updating) could also have some benefit. Well, first look over the drop code to be sure there aren't any optimizations that have been missed, but then I would suggest looking for the simplest solution. For example, only merge-match on the top 50 items, but then flag the square as needing merging. Move the map save code to a separate thread, and have that thread do the merging. Pickup is more complicated, as that can't be deferred. And did the issue with loading an apartment map with too many items get fixed? I really liked the general solution idea that someone mentioned years ago where when something is too slow, the server drops ticks on that map until it's caught up, but other maps continue to get processed. The easiest way of doing something like that would be to have one thread per map, but that's probably excessive, and having tiled maps makes per-map processing more complicated. A more general system would be to have multiple threads that pick up processing for maps, and maps would be put on a queue, so anything that makes things slow would only impact that map. It gets a little complicated when things move from one map to another (especially with tiled maps), so perhaps active tiled maps would be put on the queue as a unit. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
Re: [crossfire] Handling massive loot
On 3/22/21 11:58 AM, Nicolas Weeger wrote: Hello. One issue that happens is server delays when dealing with big loot. For instance a player selling some thousands rods can stuck the server for some seconds... Or even dropping many items on a big pile can be quite slow. So to handle that correctly, I was thinking of changing the server code like that: - split data/commands reception and processing - have one loop that reads from the socket, put commands in a waiting queue, then another function later processes those commands - for commands like pick/drop/examine, enable them to work on multiple processing loops instead of a single. So for instance the command would drop 100 items, then store its status, put itself back on top of the command waiting queue, and exit - this way the server can process something else. Being multi threaded would help some things out. It has been a while, by my recollection is the drop (and pickup) problem is that these are in fact handled by the server, eg, 'drop all' is done on the server, dropping everything of matching criteria. And likewise, pickup all is done on the server, so slowing down the command processing from the client doesn't help out. It is also my recollection in that the reason this is slow that each time an object is moved (ground to player or vice versa), it has to check the destination for duplicates to merge them (otherwise, you could have 20 different entries for silver coins). This becomes an O(n^2) operation, so when dropping a few items, it is pretty fast, but when dropping a stack of 100 different items, that takes a lot of time. There are a couple ways to deal with this: - A thought I had a while back is once a space gets too many items on it, items fall over to neighboring spaces - this would reduce the upper threshhold over that operation (never too many items to examine on a space), but this could basically result in an entire shop being filled to a level of 25 items deep. - For items going to the floor, marking the space as needing 'merging' later on could be done (one could even imagine something like a shop keeper standing on the space as this merging is done, preventing players from interacting with it) - For items going into the players inventory (they step onto a pile of junk), deferred merging could be done, but would result in new items getting send to the client, and then and update down the road that the new item does not exist and in fact item X has a different quantity. I'm not sure how visually that works out. - Maybe limit the number of items picked up at once, even with pickup all, to something like 25 or other number where performance is still reasonable. This makes it harder to clean up everything in the dungeon, but is somewhat more realistic (you can only pick up stuff so quickly). - I can't remember if the gold insertion when selling a bunch of items is intelligent. That is to say, if you sell 100 items, does is calculate the net proceeds and inserts it once (intelligent) or inserts the coinage 100 items (1/each item, keeping in mind that there could actually be several different coins being inserted for each item). So changing that (deferred gold updating) could also have some benefit. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
[crossfire] Handling massive loot
Hello. One issue that happens is server delays when dealing with big loot. For instance a player selling some thousands rods can stuck the server for some seconds... Or even dropping many items on a big pile can be quite slow. So to handle that correctly, I was thinking of changing the server code like that: - split data/commands reception and processing - have one loop that reads from the socket, put commands in a waiting queue, then another function later processes those commands - for commands like pick/drop/examine, enable them to work on multiple processing loops instead of a single. So for instance the command would drop 100 items, then store its status, put itself back on top of the command waiting queue, and exit - this way the server can process something else. For the player, it means picking/dropping a big pile of items may take some time, but the server will go on for other players during that time. Does that sound ok? Best regards Nicolas signature.asc Description: This is a digitally signed message part. ___ crossfire mailing list crossfire@metalforge.org http://mailman.metalforge.org/mailman/listinfo/crossfire
