Our StringPiece type has been made obsolete by the std::string_view type introduced in C++17, so we will eventually get rid of StringPiece and replace it with std::string_view (or absl::string_view, which has the same API but is available in C++11). So if you are making a local modification to support allocating strings on arenas, it would probably be best to go straight to std::string_view and avoid our StringPiece type. To handle both the arena and non-arena case, the simplest solution would be to store both a std::string_view and a std::string. When arenas are used, you can have the string_view point into arena-allocated memory, and when arenas are not used, it can just point to the std::string's data.
On Thu, Sep 3, 2020 at 3:19 AM X Ah <[email protected]> wrote: > Hi Feng, > I have an API design problem, Since StringPiece doesn't own the data and > it's impossible to get memory from it, So how should the behavior be if I > do ParseFromString and the message is not in arena? I think the StringPiece > field should be empty if current message doesn't own an arena, but it is > strange for user. Could you introduce how Google internal use StringPiece > in protobuf? > Thanks! > On Saturday, January 16, 2016 at 3:32:39 AM UTC+8 [email protected] > wrote: > >> On Thu, Jan 14, 2016 at 6:06 PM, Austin Schuh <[email protected]> >> wrote: >> >>> Hi, >>> >>> I've got an application where I can't allocate memory while using >>> protobufs. Arenas have been awesome for doing that. I'm able to allocate >>> a big block of memory at startup time or stack allocate memory for the >>> arena, and then use that for allocating protobufs. Thanks! >>> >>> I'd like to be able to allocate strings in the arena. I'm willing to do >>> the implementation, and wouldn't mind up-streaming if my implementation is >>> complete enough and there is interest. It looks like I should start by >>> implementing ctype=STRING_PIECE and then allocate memory in the arena to >>> back it. The class in //src/google/protobuf:arenastring.h looks like the >>> place to do all the operations. It looks like I need to modify the >>> interface to provide setters and getters to support STRING_PIECE there. >>> >>> Is that the right place to start? Is there any more guidance that you >>> can give me? >>> >> Hi Austin, >> >> Thanks for contacting us and offering help! >> >> You are looking at the right direction. We actually already opensourced >> the StringPiece implementation not very long ago: >> >> https://github.com/google/protobuf/blob/master/src/google/protobuf/stubs/stringpiece.h >> >> It's intended to be used to implement "ctype = STRING_PIECE" for string >> fields and since it's merely a <const char*, size_t> pair, it can be >> directed at the buffer in the arena. Such features are implemented inside >> Google but unfortunately it's not opensourced due to dependency issues. We >> plan to get them out eventually but hasn't have enough time to work on it. >> Since we already have an internal version of it, we probably won't be able >> to accept your contributions. I can't give a concrete timeline about when >> we will get our implementation opensourced also. Sorry for that... >> >> If you need this soon, I suggest you try to implement it as simple as >> possible. Better to only support lite runtime with arena enabled. Some >> changes you want to make: >> 1. Make ArenaStringPtr work with StringPiece, or introduce an >> ArenaStringPiecePtr which might be easier to implement. >> 2. Update protocol compiler to use ArenaStringPtr/ArenaStringPiecePtr to >> store ctype=STRING_PIECE fields and expose a StringPiece API: >> // proto >> message Foo { >> string bar = 1 [ctype = STRING_PIECE]; >> } >> // generated C++ code >> message Foo { >> public: >> StringPiece bar() const; >> void set_bar(StringPiece value); // Note that we need to do a deep >> copy here because StringPiece doesn't own the underlying data. >> void set_alias_bar(StringPiece value); // Make the field point to the >> StringPiece data directly. Caller must make sure the underlying data >> outlives the Foo message. >> >> private: >> ArenaStringPiecePtr bar_; >> }; >> >> Look at the string_field.cc implementation in the compiler directory >> <https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/cpp/cpp_string_field.cc> >> and you can create a string_piece_field.cc implementation based on that. >> Most of the work will be done here, including not only the generated API >> but also all the parsing/serialization/copy/constructor/destructor support. >> >> That's pretty all that needed to support StringPiece in lite-runtime + >> arena. A lot more work will be needed to support other combinations >> (lite-runtime + no arena, full-runtime + arena, full-runtime + non-arena), >> but since you have a specific targeted platform and we will opensource the >> StringPiece support eventually, it's probably not worthwhile to invest time >> to support anything you don't actually need right now. >> >> Hope this helps. >> >> Regards, >> Feng >> >> >>> Thanks, >>> Austin >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/protobuf/ccc70ecc-873a-4297-8102-8e2319ffd760n%40googlegroups.com > <https://groups.google.com/d/msgid/protobuf/ccc70ecc-873a-4297-8102-8e2319ffd760n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CADqAXr4AUyuqy_uPAXhau7OjPrfYtH8kwSgzhpWDxOM%3DCQjE5g%40mail.gmail.com.
