Re: [protobuf] modifying lists within a message

2010-03-18 Thread Kenton Varda
Anything with internal in the name may change in future releases, so you
should not rely on it.  I will definitely be changing this because it's
supposed to be impossible to obtain a reference to a message that can still
be modified, which calling internalGetResult() would do.  Also please assume
that anything marked protected is only intended to be accessed by
subclasses/superclasses, not code that happens to be in the same package, as
this is what I intended when I wrote it.  :/

I am open to ideas for how to make this stuff better, but I have certain
requirements that must be met:
1) If code outside the protobuf implementation holds a reference to a
message, it must be guaranteed that that message cannot change after that
point.
2) The interface should not tend to make it more convenient to write
inefficient code than efficient code.
3) The implementation should perform well.

On Wed, Mar 17, 2010 at 1:57 PM, Ron Reynolds tequila...@ymail.com wrote:

 but this message isn't built yet - i'm trying to modify a copy of a
 previous message wrapping it in a newBuilder so i don't have to copy all the
 fields over from the previous (immutable) to the new (mutable until i call
 Builder.build()).  the only thing (so far) that i can't do to that new
 message is remove a single element from its list variable.

 you mean it's O(n) because of the shift of all elements after the index
 since ArrayList is a continuous block?  i wouldn't have called that O(n).
 certainly it's O(n) to find the element in the unsorted list, but that's
 unavoidable (unless i add code to put the elements in sorted...)  once i
 find the index of the single element in the list i want i have to remove it
 and then add a new (copied and then modified from the original) element to
 replace it.  (order is unimportant, at least in this case).

 and yes, it is true - classes in the same package have access to each
 others protected members.  package and protected might MEAN something
 different but the way the JVM (the Sun JVM anyway) acts the access level is
 an int:  0 = public, 1 = protected, 2 = package, 3 = private - you can
 access everything = your access-level with respect to the other class.
 t'would be a pity for you to fix it - i was rather happy to figure that
 one out. :)  in which case wiping the entire list and inserting N-1 elements
 back in will be my only (non-reflection-based) option (which i'm pretty sure
 is more costly than removing a single element from an ArrayList by
 index...).

 --
 *From:* Kenton Varda ken...@google.com
 *To:* Ron Reynolds tequila...@ymail.com
 *Cc:* protobuf@googlegroups.com
 *Sent:* Wed, March 17, 2010 1:32:50 PM

 *Subject:* Re: [protobuf] modifying lists within a message

 Also note that we cannot return a modifiable list even from the Builder's
 getter because the caller could then hold on to that list and modify it
 after the message is built.  We really need to prevent any kind of
 modifications from happening after the message is built.

 On Wed, Mar 17, 2010 at 1:31 PM, Kenton Varda ken...@google.com wrote:



 On Wed, Mar 17, 2010 at 12:59 PM, Ron Reynolds tequila...@ymail.comwrote:

 seems odd that you can delete all via clearXXX() or add via addAllXXX()
 and addXXX() but not remove a single element by index...


 Our experience is that if we provide a way to remove a single element by
 index, people will mistakenly think that the method is more efficient than
 clearing the whole list and rebuilding it, when in fact both operations are
 O(n).  The result of this is that people write loops like:

   for (all elements) {
 if (shouldFilter(element)) {
   remove(element);
 }
   }

 This loop is O(n^2).


 well, here's a work-around - if you put a utility class into the same
 package as the ProtobufMessages generated class it can access the protected
 Builder.internalGetResult() - since package-level access is higher than
 protected-level access...


 Yikes, is that true?  I don't think it is, but if so we need to fix it.
  I'm pretty sure protected is neither higher nor lower than package-level
 -- it means something entirely different.


  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed

Re: [protobuf] modifying lists within a message

2010-03-17 Thread Jason Hsueh
Messages are indeed immutable once built. getXXXList() for both the builder
and the message should return unmodifiable lists: if you try to modify them
you should get an exception. If you haven't already, take a look at the Java
tutorial here:
http://code.google.com/apis/protocolbuffers/docs/javatutorial.html

For your particular problem I think you'll need do a clearXXX() on the
entire list, and add back all the elements except the entry you want to
discard.

On Wed, Mar 17, 2010 at 7:57 AM, Ron tequila...@ymail.com wrote:

 i didn't see anything on this in the archives but i admit i didn't do
 an exhaustive search.  i'm using protobuf (2.3.0) to store objects in
 a Voldemort cluster and am working on the code to modify these
 objects.  so i load the message fine, but it is composed of a list of
 other messages and i'd like to modify the contents of that list and
 then stream it back into the cluster.  i assumed this is what
 Type.newBuilder(Type) was for, but have run into an interesting
 wrinkle trying to modify the list (once i've located the entry i want
 to remove by index).  Builder.getXXXList() returns an
 unmodifiableList, but the actual message's getXXXList() returns a ref
 to the actual list (i.e., i could modify the list in the message, but
 not the list within the message within the builder).  i haven't gone
 through all the code yet but this seemed rather odd since i had
 thought messages within builders were mutable until build() but once
 built were considered immutable (tho perhaps that's just in my head).

 is there a doc showing best practices for modifying Protobuf messages
 i should be reading?

 thanks,
 ...ron.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] modifying lists within a message

2010-03-17 Thread Kenton Varda
When you call Builder.getXXXList(), the underlying list is still modifiable
because it's still being built, so it has to convert the list to
unmodifiable before returning.  But when you call Builder.build(), the
underlying list is made unmodifiable.  Thus the message's getXXXList() does
not need to wrap the list on every call.

On Wed, Mar 17, 2010 at 8:57 AM, Ron tequila...@ymail.com wrote:

 i didn't see anything on this in the archives but i admit i didn't do
 an exhaustive search.  i'm using protobuf (2.3.0) to store objects in
 a Voldemort cluster and am working on the code to modify these
 objects.  so i load the message fine, but it is composed of a list of
 other messages and i'd like to modify the contents of that list and
 then stream it back into the cluster.  i assumed this is what
 Type.newBuilder(Type) was for, but have run into an interesting
 wrinkle trying to modify the list (once i've located the entry i want
 to remove by index).  Builder.getXXXList() returns an
 unmodifiableList, but the actual message's getXXXList() returns a ref
 to the actual list (i.e., i could modify the list in the message, but
 not the list within the message within the builder).  i haven't gone
 through all the code yet but this seemed rather odd since i had
 thought messages within builders were mutable until build() but once
 built were considered immutable (tho perhaps that's just in my head).

 is there a doc showing best practices for modifying Protobuf messages
 i should be reading?

 thanks,
 ...ron.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] modifying lists within a message

2010-03-17 Thread Ron Reynolds
seems odd that you can delete all via clearXXX() or add via addAllXXX() and 
addXXX() but not remove a single element by index...

well, here's a work-around - if you put a utility class into the same package 
as the ProtobufMessages generated class it can access the protected 
Builder.internalGetResult() - since package-level access is higher than 
protected-level access...





From: Jason Hsueh jas...@google.com
To: Ron tequila...@ymail.com
Cc: Protocol Buffers protobuf@googlegroups.com
Sent: Wed, March 17, 2010 9:31:58 AM
Subject: Re: [protobuf] modifying lists within a message

Messages are indeed immutable once built. getXXXList() for both the builder and 
the message should return unmodifiable lists: if you try to modify them you 
should get an exception. If you haven't already, take a look at the Java 
tutorial here: 
http://code.google.com/apis/protocolbuffers/docs/javatutorial.html

For your particular problem I think you'll need do a clearXXX() on the entire 
list, and add back all the elements except the entry you want to discard.


On Wed, Mar 17, 2010 at 7:57 AM, Ron tequila...@ymail.com wrote:

i didn't see anything on this in the archives but i admit i didn't do
an exhaustive search.  i'm using protobuf (2.3.0) to store objects in
a Voldemort cluster and am working on the code to modify these
objects.  so i load the message fine, but it is composed of a list of
other messages and i'd like to modify the contents of that list and
then stream it back into the cluster.  i assumed this is what
Type.newBuilder(Type) was for, but have run into an interesting
wrinkle trying to modify the list (once i've located the entry i want
to remove by index).  Builder.getXXXList() returns an
unmodifiableList, but the actual message's getXXXList() returns a ref
to the actual list (i.e., i could modify the list in the message, but
not the list within the message within the builder).  i haven't gone
through all the code yet but this seemed rather odd since i had
thought messages within builders were mutable until build() but once
built were considered immutable (tho perhaps that's just in my head).

is there a doc showing best practices for modifying Protobuf messages
i should be reading?

thanks,
...ron.

--
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



  

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] modifying lists within a message

2010-03-17 Thread Kenton Varda
On Wed, Mar 17, 2010 at 12:59 PM, Ron Reynolds tequila...@ymail.com wrote:

 seems odd that you can delete all via clearXXX() or add via addAllXXX() and
 addXXX() but not remove a single element by index...


Our experience is that if we provide a way to remove a single element by
index, people will mistakenly think that the method is more efficient than
clearing the whole list and rebuilding it, when in fact both operations are
O(n).  The result of this is that people write loops like:

  for (all elements) {
if (shouldFilter(element)) {
  remove(element);
}
  }

This loop is O(n^2).


 well, here's a work-around - if you put a utility class into the same
 package as the ProtobufMessages generated class it can access the protected
 Builder.internalGetResult() - since package-level access is higher than
 protected-level access...


Yikes, is that true?  I don't think it is, but if so we need to fix it.  I'm
pretty sure protected is neither higher nor lower than package-level -- it
means something entirely different.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] modifying lists within a message

2010-03-17 Thread Kenton Varda
Also note that we cannot return a modifiable list even from the Builder's
getter because the caller could then hold on to that list and modify it
after the message is built.  We really need to prevent any kind of
modifications from happening after the message is built.

On Wed, Mar 17, 2010 at 1:31 PM, Kenton Varda ken...@google.com wrote:



 On Wed, Mar 17, 2010 at 12:59 PM, Ron Reynolds tequila...@ymail.comwrote:

 seems odd that you can delete all via clearXXX() or add via addAllXXX()
 and addXXX() but not remove a single element by index...


 Our experience is that if we provide a way to remove a single element by
 index, people will mistakenly think that the method is more efficient than
 clearing the whole list and rebuilding it, when in fact both operations are
 O(n).  The result of this is that people write loops like:

   for (all elements) {
 if (shouldFilter(element)) {
   remove(element);
 }
   }

 This loop is O(n^2).


 well, here's a work-around - if you put a utility class into the same
 package as the ProtobufMessages generated class it can access the protected
 Builder.internalGetResult() - since package-level access is higher than
 protected-level access...


 Yikes, is that true?  I don't think it is, but if so we need to fix it.
  I'm pretty sure protected is neither higher nor lower than package-level
 -- it means something entirely different.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] modifying lists within a message

2010-03-17 Thread Ron Reynolds
but this message isn't built yet - i'm trying to modify a copy of a previous 
message wrapping it in a newBuilder so i don't have to copy all the fields over 
from the previous (immutable) to the new (mutable until i call 
Builder.build()).  the only thing (so far) that i can't do to that new message 
is remove a single element from its list variable.  

you mean it's O(n) because of the shift of all elements after the index since 
ArrayList is a continuous block?  i wouldn't have called that O(n).  certainly 
it's O(n) to find the element in the unsorted list, but that's unavoidable 
(unless i add code to put the elements in sorted...)  once i find the index of 
the single element in the list i want i have to remove it and then add a new 
(copied and then modified from the original) element to replace it.  (order is 
unimportant, at least in this case).

and yes, it is true - classes in the same package have access to each others 
protected members.  package and protected might MEAN something different 
but the way the JVM (the Sun JVM anyway) acts the access level is an int:  0 = 
public, 1 = protected, 2 = package, 3 = private - you can access everything = 
your access-level with respect to the other class.  t'would be a pity for you 
to fix it - i was rather happy to figure that one out. :)  in which case 
wiping the entire list and inserting N-1 elements back in will be my only 
(non-reflection-based) option (which i'm pretty sure is more costly than 
removing a single element from an ArrayList by index...).





From: Kenton Varda ken...@google.com
To: Ron Reynolds tequila...@ymail.com
Cc: protobuf@googlegroups.com
Sent: Wed, March 17, 2010 1:32:50 PM
Subject: Re: [protobuf] modifying lists within a message

Also note that we cannot return a modifiable list even from the Builder's 
getter because the caller could then hold on to that list and modify it after 
the message is built.  We really need to prevent any kind of modifications from 
happening after the message is built.


On Wed, Mar 17, 2010 at 1:31 PM, Kenton Varda ken...@google.com wrote:




On Wed, Mar 17, 2010 at 12:59 PM, Ron Reynolds tequila...@ymail.com wrote:



seems odd that you can delete all via clearXXX() or add via addAllXXX() and 
addXXX() but not remove a single element by index...



Our experience is that if we provide a way to remove a single element by 
index, people will mistakenly think that the method is more efficient than 
clearing the whole list and rebuilding it, when in fact both operations are 
O(n).  The result of this is that people write loops like:


  for (all elements) {
if (shouldFilter(element)) {
  remove(element);
}
  }


This loop is O(n^2).

 
well, here's a work-around - if you put a utility class into the same package 
as the ProtobufMessages generated class it can access the protected 
Builder.internalGetResult() - since package-level access is higher than 
protected-level access...



Yikes, is that true?  I don't think it is, but if so we need to fix it.  I'm 
pretty sure protected is neither higher nor lower than package-level -- it 
means something entirely different.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



  

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] modifying lists within a message

2010-03-17 Thread Oliver Jowett
Kenton Varda wrote:

 well, here's a work-around - if you put a utility class into the
 same package as the ProtobufMessages generated class it can access
 the protected Builder.internalGetResult() - since package-level
 access is higher than protected-level access...
 
 
 Yikes, is that true?  I don't think it is, but if so we need to fix it.
  I'm pretty sure protected is neither higher nor lower than
 package-level -- it means something entirely different.

Both default (package) access and protected access members can be
accessed by classes in the same package. Yeah, it's a bit
counterintuitive. See the JVM spec section 2.7.4:

 A field, method, or constructor of a class may be declared using at most one 
 of the public, private, or protected keywords. A public member may be 
 accessed by any class or interface. A private member may be accessed only 
 from within the class that contains its declaration. A member that is not 
 declared public, protected, or private is said to have default access and may 
 be accessed from, and only from, anywhere in the package in which it is 
 declared.
 
 A protected member of an object may be accessed only by code responsible for 
 the implementation of that object. To be precise, a protected member may be 
 accessed from anywhere in the package in which it is declared and, in 
 addition, it may be accessed from within any declaration of a subclass of the 
 class type that contains its declaration, provided that certain restrictions 
 are obeyed. 

If you really, really want to prevent anything from inserting additional
code into your package, you can make a sealed package, and
classloaders will refuse to later load classes declared in that package
from other sources. See
http://java.sun.com/docs/books/tutorial/deployment/jar/sealman.html

(This is a packaging step, though, so you'd probably need to have more
of a standard Java packaging step, rather than the current use maven or
build it by hand)

-O

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.