On Thu, Dec 24, 2020 at 1:08 AM Artur Vianna <lordhowen...@gmail.com> wrote:
> Before using gob was using encoding.BinaryMarshaler, but that would mean
> the user of the api would need to implement a MarshalBinary for every type,
> which is kind of cumbersome.
>
> An option might be to let the user choose gob, BinaryMarshaler or Json etc
> to best fit the use case, but that takes the simplicity of only gobs away.
>
I am all in favor of API simplicity, but gob just isn't super useful for
this. Not to repeat myself, but you should really at least try JSON - it
provides exactly the same convenience as gob, but doesn't suffer these
problems. It might have a bit more overhead and might even be costlier to
encode - but the savings from being able to eliminate duplicate effort
should offset that (and I'm not even super convinced - I don't think gob is
the most well-optimized encoding in the stdlib).
But IMO, if you provide an API, the best solution is to a) just use
`[]byte` at the base-layer and then b) provide convenience-wrappers around
that for other formats. This lets the users decide what they want (they
might want something completely different anyway) while still providing a
decently convenient API for simple uses.
>
> I did try your solution to reset the client too but i'm getting
> inconsistent behaviour, in one server it works and in another it doesn't
> ("corrupted data or unknown type"). I think synching the server and client
> will be error prone, while also increasing the use of network.
>
> The easiest solution now is to label the package for ≤32 players and test
> alternative encodings that keep the API as clean as with gob. I took a look
> at flatbuffers but it will be cumbersome for the user to create the
> builders, and i really wanted the simplest possible API.
>
> Maybe i should try UDP Broadcast too and see what happens, probably chaos
> :D
>
>
> On Wed, 23 Dec 2020, 20:36 Axel Wagner, <axel.wagner...@googlemail.com>
> wrote:
>
>> No, it wouldn't. Because the encoder keeps state about which
>> type-information it already sent and wouldn't sent it again - causing the
>> client to be unable to decode. So you'd also need a new encoder on the
>> server. And at that point, you're back to the status quo, with one encoder
>> per client and the duplication of encoding effort.
>>
>> A solution would, perhaps, be if the gob API would give you a way to send
>> *only* the type-info (so you could, if the connection breaks, create a new
>> encoder, send all the type info, and *then* multicast the encoded values).
>> But it doesn't.
>>
>> Really, I think it's far less effort to just use a different format (and
>> I would maintain that even json would probably be fine) than trying to make
>> this work with gob :)
>>
>> On Thu, Dec 24, 2020 at 12:20 AM Matthew Zimmerman <mzimmer...@gmail.com>
>> wrote:
>>
>>> If you would "reset" each client with a new decoder each time you make a
>>> new encoder, everything should work fine. Just would take some
>>> coordination.
>>>
>>> On Wed, Dec 23, 2020, 6:08 PM Artur Vianna <lordhowen...@gmail.com>
>>> wrote:
>>>
>>>> I will look into other protocols, although for now the performance is
>>>> not an issue in servers with less than 100 players.
>>>>
>>>> The problem with io.MultiWriter is that a player inside the group may
>>>> disconnect or a new player may come in. This means a new io.MultiWriter
>>>> must be created each time you dispatch, since the group may have changed in
>>>> the meantime. This would also need a new encoder and then the "duplicate
>>>> type received" happens.
>>>>
>>>> On Wed, 23 Dec 2020, 19:58 'Axel Wagner' via golang-nuts, <
>>>> golang-nuts@googlegroups.com> wrote:
>>>>
>>>>> The issue with that approach is that gob keeps state about which
>>>>> type-information it still has to send. So if you encode to, say, a
>>>>> bytes.Buffer, it would encode all type-info on every message sent, which
>>>>> is
>>>>> a significant overhead.
>>>>> TBH, I don't understand why `io.MultiWriter` wouldn't work. It would
>>>>> be helpful to see the code that causes the error message OP is seeing.
>>>>>
>>>>> However, really, gob just doesn't provide a good API for this sorta
>>>>> thing, as mentioned. The format itself is fine, but the stateful
>>>>> connection
>>>>> means that if you don't want to write *exactly* the same data in exactly
>>>>> the same order to all connections (which can perform poorly and lead to
>>>>> operational problems with timeouts and intermittently lost connections and
>>>>> the like), you are going to have a bad time.
>>>>> You honestly would fare better with a full-fledged RPC framework such
>>>>> as gRPC. Or, if you don't want the overhead of its IDL, even json. Because
>>>>> at least the "encode once, send to each client" is trivial to solve with
>>>>> that.
>>>>>
>>>>> But, that's just my 2¢ :)
>>>>>
>>>>> On Wed, Dec 23, 2020 at 11:43 PM Robert Engels <reng...@ix.netcom.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, that is why you need to create your own protocol. Use the gob to
>>>>>> encode to a buffer then send the buffer on each of the connections using
>>>>>> your protocol.
>>>>>>
>>>>>> On Dec 23, 2020, at 4:19 PM, Matthew Zimmerman <mzimmer...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> My understanding is that gob streams are unique.
>>>>>>
>>>>>> From https://golang.org/pkg/encoding/gob/
>>>>>> "A stream of gobs is self-describing. Each data item in the stream is
>>>>>> preceded by a specification of its type, expressed in terms of a small
>>>>>> set
>>>>>> of predefined types."
>>>>>>
>>>>>> In my own rudimentary understanding/terms, it sends the struct
>>>>>> definition once, then uses shorthand for it afterwards. E.g, how many
>>>>>> bytes and what order. If you mix and match streams that send definitions
>>>>>> in different orders, then chaos ensues.
>>>>>>
>>>>>> I think this is why people use other encoders in the scenario you're
>>>>>> taking about. For a one to one stream gob works great, but in this multi
>>>>>> scenario I don't think it does.
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> On Wed, Dec 23, 2020, 5:07 PM Artur Vianna <lordhowen...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> If i create a bytes.Buffer and a gob.Encoder, each time i write to a
>>>>>>> group of connections i get "duplicate type received" and if i try and
>>>>>>> reuse
>>>>>>> the encoder, i get "corrupted data" and "unknown type".
>>>>>>> It seems i can't use both net.Conn.Write and gob.Encoder.Encode in
>>>>>>> the same connection, i will try always encoding to a buffer in both
>>>>>>> unicast
>>>>>>> and multicast like you said and report if it works.
>>>>>>>
>>>>>>> On Wed, 23 Dec 2020, 18:49 Robert Engels, <reng...@ix.netcom.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> You need to encode once to a byte array then send the byte array on
>>>>>>>> each connection.
>>>>>>>>
>>>>>>>> On Dec 23, 2020, at 3:45 PM, meera <lordhowen...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I am trying to create a package for game servers using gob. The
>>>>>>>> current approach is an application level multicasting over TCP, having
>>>>>>>> a
>>>>>>>> gob encoder and decoder for each player connection, and set up a
>>>>>>>> goroutine
>>>>>>>> to receive and another to dispatch for each one. The code for the
>>>>>>>> dispatcher is here. But summarized, it simply receives data from a
>>>>>>>> channel
>>>>>>>> and encodes it.
>>>>>>>>
>>>>>>>> The problem is that if i want to transmit a single piece of data to
>>>>>>>> all players, this piece of data is encoded again and again for each
>>>>>>>> connection, doing duplicate work. With less than 100 players this is
>>>>>>>> not a
>>>>>>>> problem, but with 300+ my machine is at almost 100% usage and the
>>>>>>>> profiler
>>>>>>>> shows that most of it is spent on encoding. Here's the issue on github.
>>>>>>>>
>>>>>>>> I tryied using a io.MultiWriter but gob complains of duplicate type
>>>>>>>> received, and if i try to write the raw bytes from the gob.Encoder i
>>>>>>>> get
>>>>>>>> corrupted data. An option is using UDP Broadcasting but since gob
>>>>>>>> expects a
>>>>>>>> stream, i'm affraid i will run into unexpected behavior when packets
>>>>>>>> start
>>>>>>>> to be lost and fragmented.
>>>>>>>>
>>>>>>>> Does gob expect a single encoder and decoder to own the stream? Not
>>>>>>>> allowing two encoders on the server for one decoder on the client?
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com
>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "golang-nuts" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com
>>>>>> <https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFcE9fJvLaEB1oSRFaWOWWAOy%2BuZgO6NQs56O_1SxovzQ%40mail.gmail.com.
