why won't slicing lists raise IndexError?
I was extending a `list` and am wondering why slicing lists will never raise an IndexError, even if the `slice.stop` value if greater than the list length. Quick example: my_list = [1, 2, 3] my_list[:100] # does not raise an IndexError, but instead returns the full list Is there any background on why that doesn't raise an IndexError? Knowing that might help me design my extended list class better. For my specific use case, it would simplify my code (and prevent `if isinstance(item, slice)` checks) if the slicing raised an IndexError in the example I gave. -- https://mail.python.org/mailman/listinfo/python-list
Re: why won't slicing lists raise IndexError?
I'll try to summarize what I've learned with a few responses in hodge-podge order and to no one in particular: >That's a feature dude, not a bug. Absolutely. I _do not_ think that how slicing works in python should be changed, but I _do_ want to understand its design decisions because it will make me a better programmer. >one thing Python consistently emphasises is making things easy for >the user, even if that convenience comes at a cost to the implementer. I completely agree, and I'm always happy to do the work as a developer to make the user experience better. That's something I really appreciate about python as well. >I've got a buffer class with that behaviour which is very useful for parsing data streams. [...] >In summary, I think you should give your LazyList a .extend method to establish your code's precondition (enough elements) and then you can proceed with your slicing in surety that it will behave correctly. Thanks for this detailed response, I'll have to reread it a few times to understand it but I think I'll understand how I might improve my implementation from your example. In particular, that last "In summary" sentence hit on something I'd been thinking about. A `.extend` method might clean things up nicely. -- It seems to me that the consensus on the reason for why slices don't throw IndexErrors is essentially "It's very convenient for them not to" and I'm just fine with that response (not super thrilled, but happy enough). There were two comments in particular that sounded pretty good: >But as a user, I can say that I find the ability to use slices without checking for >out of bounds cases or handling exceptions to be really convenient. That's fair. The alternative would be something that someone else mentioned: `my_string[ : min(n, len(my_string))]` and that's not very pretty, although I don't think it's absolutely horrible. And: the use case where you are trying to split a list or string at index `n` is: `first_half, second_half = my_string[:n], my_string[n:]` and if n > len(my_string) then this use case becomes a bit annoying to write (although it could use the `min` example above). So I guess the conclusion (as far as I can tell) is that: `my_string[:n]` is more convenient than `my_string[:min(n, len(my_string))]`, and that sounds okay enough to me. If I'm being completely honest I kinda like the logical explicitness of the latter, but it's way less convenient. And I'm 100% sure the people who made this decision are incredibly smart, and I certainly trust their decision. Thanks everyone for the responses. On Mon, Dec 4, 2017 at 6:13 PM, Chris Angelicowrote: > On Tue, Dec 5, 2017 at 10:50 AM, Rick Johnson > wrote: > > Chris Angelico wrote: > >> wrote: > >> > Terry Reedy wrote: > >> > > >> > [...] > >> > > >> >> try: > >> >> item = seq[n] > >> >> except IndexError > >> >> do_without_item() > >> >> else: > >> >> process(item) > >> >> > >> >> item = seq[n:n+1] > >> >> if item: > >> >> process(item) > >> >> else: > >> >> do_without_item() > >> >> > >> >> Many prefer the second. > >> > > >> > And they'll prefer it even more when they realize the entire ELSE > >> > clause of your latter example is superfluous. > >> > >> ... how is it superfluous? > > > > If the only purpose of an if/else logic structure is to > > process an arbitrary value _only_ when that value is > > "truthy", and futhermore, the else clause does nothing of > > any significance (and in the example provided, appears to > > be nothing but a placeholder for dead code) then why > > bother writing the else clause in the first place? > > Ahhh, I see how it is. You didn't run the code, ergo you don't > understand it. Makes perfect sense. :) > > Hint: Truthiness is fairly clearly defined here, regardless of the > value of item. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: why won't slicing lists raise IndexError?
> > >> This is explained in the Python tutorial for strings > >> https://docs.python.org/3/tutorial/introduction.html#strings, as a list > >> is a sequence just like a string it will act in exactly the same way. > >> > > > > The only relevant bit I found in that link is: "However, out of range > > slice indexes are handled gracefully when used for slicing". I do > > understand _how_ slices work, but I would really like to know a bit more > > about why slices will never throw out-of-bounds IndexErrors. > > That's what "handled gracefully" means. Instead of throwing, they get > clamped. > > ChrisA Cool! Why? I know I'm being a bit pedantic here, but I truly would like to understand _why_ slices get clamped. I've been writing python code every day for almost 7 years, so usually I can figure stuff like this out, but I'm struggling here and would really appreciate some insight. -- https://mail.python.org/mailman/listinfo/python-list
Re: why won't slicing lists raise IndexError?
> > This is explained in the Python tutorial for strings > https://docs.python.org/3/tutorial/introduction.html#strings, as a list > is a sequence just like a string it will act in exactly the same way. > The only relevant bit I found in that link is: "However, out of range slice indexes are handled gracefully when used for slicing". I do understand _how_ slices work, but I would really like to know a bit more about why slices will never throw out-of-bounds IndexErrors. -- https://mail.python.org/mailman/listinfo/python-list
Re: why won't slicing lists raise IndexError?
>Why would this simplify your code? What are you doing that would benefit >from an IndexError here? Without simplifying too much, I'm writing a wrapper around a REST API. I want lazy-loading functionality for lists-of-things, so I am creating a LazyList class. This LazyList class will load items as needed, and store them on the list. When a user triggers `__getitem__`, the LazyList may not yet have data for the index the user requests, in which case it needs to load more data. I'll write out the basic functionality of the LazyList class so you can explicitly see how slicing is impacting the implementation: class LazyList(list): def __getitem__(item): try: return super().__getitem__(item) except IndexError: self._generate_more_data(item) # If, for example, item == 10, this function will keep generating more data until there are 10 items in the list return super().__getitem__(item) This works great when `item` is an integer. However, when `item` is a slice, the IndexError is never triggered, and so new data will never be loaded. Consider this snippet: lazy_list = LazyList() sublist = lazy_list[:10] In this case, `sublist` will always be empty, because the first `return super().__getitem__(item)` "succeeds" by returning an empty list. If slicing raised an IndexError, the above code would work great. Instead, I need to add a check before the try/except to check if the `item` is a `slice`, and if it is, I need to do bounds checking on the list. It seems a bit unpythonic to have to do an isinstance check here, so I am wondering if I am missing something important about why slices don't raise IndexErrors. And I'll be honest -- I like the implementation of the LazyList I wrote above. I think it's pretty logical, because it allows you to think about the lazy list like this: "Treat the list like a norma list. If you run out of bounds, get more data, then treat the list like a normal list again." And I really like that clean logic. -- https://mail.python.org/mailman/listinfo/python-list
Re: why won't slicing lists raise IndexError?
> > Have you ever used a language that does that? I have. > The String class in the C# language does that, and it's /really/ annoying. > I have to add extra code to prevent such exceptions. > In practice, I find that the way that Python does it is much nicer. (And > Python isn't unique in this respect, either.) > I'm not here to criticize one way or the other, just understand. As far as I'm aware, I don't rely on python's slicing functionality allowing indexes that are over the length of a list. But I'm just one person and I'm sure there are many other use cases I'm not thinking about. If you think what you wrote is a good explanation for why slicing doesn't raise IndexErrors in python, I'd be interesting in seeing some of your use cases about when you had to explicitly check bounds of list slices in C#. -- https://mail.python.org/mailman/listinfo/python-list
why won't slicing lists raise IndexError?
I was extending a `list` and am wondering why slicing lists will never raise an IndexError, even if the `slice.stop` value if greater than the list length. Quick example: my_list = [1, 2, 3] my_list[:100] # does not raise an IndexError, but instead returns the full list Is there any background on why that doesn't raise an IndexError? Knowing that might help me design my extended list class better. For my specific use case, it would simplify my code (and prevent `if isinstance(item, slice)` checks) if the slicing raised an IndexError in the example I gave. -- https://mail.python.org/mailman/listinfo/python-list
Re: what exactly does type.__call__ do?
Thanks for your response. I'm confident that my use case for this metaclass is correct for what I'm trying to do. I've been using them for a few years now, but sparingly because as you said they are almost always unnecessary. I simplified my example significantly for discussion -- I'm not actually using a Singleton, but it's an easy situation to think about. I'd like to ask a couple follow up questions here: By using... @classmethod def normal_constructor(cls, *args, **kwargs): return type.__call__(cls, *args, **kwargs) # Aside: Yes, the `cls` does need to be in here ... I'm assuming that there are no inherited metaclasses. I'm explicitly saying, "No matter what the super() functionality for creating the class is, always use `type`.__call__'s functionality". That seems wrong/unpythonic to me, or at least inconsistent with how non-metaclass inheritance works in python (ie you are encouraged to use `super()` when using single inheritance). Because of that, my guess is... @classmethod def normal_constructor(cls, *args, **kwargs): return super(cls).__call__(cls, *args, **kwargs) # or maybe it should be: return super(my_metaclass, cls).__call__(cls, *args, **kwargs) ... is closer to the correct approach because it will properly hit the `__call__` method of the metaclass's parent class (which will often ey `type`'s `__call__` method if the metaclass doesn't inherit from any other metaclass). However, I've never seen anything like that after years of working with / reading about metaclasses. Does that help redefine my question a little bit? You said all three look "weird and scary", but I don't know why. Can you explain that a bit please? The 3rd example I gave (using `cls.__new__` and `self.__init__`) I agree is a bit weird and scary, and in my opinion that's because I am assuming what the correct sequence of calls is for how classes are normally instantiated. So I'm the least thrilled with that one. I feel like using super is the correct choice -- for the reasons I listed above -- but I would like a more expert opinion (and I'd like to learn why :)). Thanks! Jason On Thu, Nov 2, 2017 at 12:28 AM, Steve D'Aprano <steve+pyt...@pearwood.info> wrote: > On Thu, 2 Nov 2017 10:13 am, Jason Maldonis wrote: > > > Hi everyone, > > > > I want to use a metaclass to override how class instantiation works. I've > > done something analogous to using the Singleton metaclass from the > Python3 > > Cookbook example. > > In my opinion, nine times out of ten, using a metaclass for something like > that is overkill. > > (And that's putting aside the fact that 999 out of a thousand, using a > Singleton is the wrong solution, no matter what the question is.) > > > > However, I want to provide a classmethod that allows for "normal" class > > instantiation that prevents this metaclass from being used. > > To me, that strongly suggests that a metaclass is the wrong solution. > > > > To do that, I think I just make a @classmethod constructor function. > > However, I can imagine a few different ways of writing this: > > > > @classmethod > > def normal_constructor(cls, *args, **kwargs): > > return type.__call__(*args, **kwargs) > > Untested, but I think that should be: > > return type.__call__(cls, *args, **kwargs) > > > > @classmethod > > def normal_constructor(cls, *args, **kwargs): > > return super(???).__call__(*args, **kwargs) # I'm not sure what > should > > go in the super here (I'm using python3) > > > > @classmethod > > def normal_constructor(cls, *args, **kwargs): > > self = cls.__new__(cls) > > self.__init__(*args, **kwargs) > > return self > > > > Is one of these correct? Or do they all do the same thing? > > None of them look "correct", they all look "weird and scary" :-) > > If I had to pick one of these three -- and I hope that I would not -- I'd > pick > the first one. > > > > I was looking for documentation for what exactly `type.__call__` does so > > that I can emulate it, > > And then if type.__call__ changes, your emulation will be wrong. > > > > but I wasn't able to find any docs explicitly > > detailing what that method does. If someone knows where this info is that > > would be great too. > > > > -- > Steve > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure > enough, things got worse. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: what exactly does type.__call__ do?
Ok no worries then! Thanks for the tips. I might wait until tomorrow then until someone comes along who deals with metaclasses and alternate class constructors. In case you're curious, I'm doing two things that are relevant here, and I'll link the python3 cookbook examples that are super useful (I love that book): http://chimera.labs.oreilly.com/books/123000393/ch09. html#_discussion_155 and http://chimera.labs.oreilly.com/books/123000393/ch08.html#_solution_134 They explain things very well. Metaclasses are so useful when you hit a use case that they fit into. I don't often deal with multiple class constructors, so I'm a bit new to that territory and I'm trying to delve into the details of how python classes are constructed (namely, whether `type.__call__` does more than just call `cls.__new__` and `self.__init__`). Thanks for the help too. On Wed, Nov 1, 2017 at 8:49 PM, Stefan Ramwrote: > r...@zedat.fu-berlin.de (Stefan Ram) writes: > >|PyObject *meth = lookup_method(self, ___call__); > ... > >|Call self as a function. > > Oh, I guess this might be what makes > > int(x) > > call > > int.__call__(x) > > . And all other instances of this metaclass. > > Let's try to overwrite it to call 'charles' instead. > > main.py > > class mymetaclass( type ): > def __call__( this, *args ): > this.charles( *args ) > > class myclass( metaclass=mymetaclass ): > def charles(): > print( "Charles" ) > > myclass() > > transcript > > Charles > > Ok, while I managed to make »myclass« print »Charles« > using a »__call__« function of it's metaclass, I still > don't know what I am doing. This is the first time I > actually deal with the topic of metaclasses. I have not > yet read a text book chapter about them, so I'm still > in the dark like the Chinese in the Chinese room. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: what exactly does type.__call__ do?
Thanks for the reply. And I think I wasn't clear enough. I was wondering what the metaclass `type`'s `type.__call__` does explicitly. I'm reasonably comfortable writing metaclasses when I need them, and I understand how `.__call__` works for non-metaclass objects. In my first email I gave three possible ways to write a @classmethod that returns an instance of that class. I'm not sure which one is the "most pythonic" or "most correct", but I'm pretty sure I can get them all to work. I'll paste them below again, and I'm wondering which one I should use and why. Thanks! @classmethod def normal_constructor(cls, *args, **kwargs): return type.__call__(*args, **kwargs) @classmethod def normal_constructor(cls, *args, **kwargs): return super(???).__call__(*args, **kwargs) # I'm not sure what should go in the super here (I'm using python3) @classmethod def normal_constructor(cls, *args, **kwargs): self = cls.__new__(cls) self.__init__(*args, **kwargs) return self On Wed, Nov 1, 2017 at 6:51 PM, Stefan Ram <r...@zedat.fu-berlin.de> wrote: > Jason Maldonis <jjmaldo...@gmail.com> writes: > >I was looking for documentation for what exactly `type.__call__` does so > >that I can emulate it, but I wasn't able to find any docs explicitly > >detailing what that method does. If someone knows where this info is that > >would be great too. > > Instances are callable if their class has a __call__ method. > > When an instance is called, this __call__ method will be called. > > main.py > > class example: > def __call__( this ): > print( 'called' ) > instance = example() > instance() > > transcript > > called > > The general contract of this method is that it can do > whatever the author of the class deems appropriate. > > What exectly this is has to be documented in the > documentation of each specific class. > > In fact, the attribute »instance.__call__« has a »__call__« > attribute itself, thus: > > main.py > > class example: > def __call__( this ): > print( 'called' ) > instance = example() > instance() > instance.__call__() > instance.__call__.__call__() > > transcript > > called > called > called > > A type also is an instance, so > > int.__call__(whatever) > > should do the same as > > int( whatever ) > > . In Python 2 they had > > operator.isCallable( obj ) > > , in Python 3 we can emulate this using > > hasattr( obj, '__call__' ) > > . > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
what exactly does type.__call__ do?
Hi everyone, I want to use a metaclass to override how class instantiation works. I've done something analogous to using the Singleton metaclass from the Python3 Cookbook example. However, I want to provide a classmethod that allows for "normal" class instantiation that prevents this metaclass from being used. To do that, I think I just make a @classmethod constructor function. However, I can imagine a few different ways of writing this: @classmethod def normal_constructor(cls, *args, **kwargs): return type.__call__(*args, **kwargs) @classmethod def normal_constructor(cls, *args, **kwargs): return super(???).__call__(*args, **kwargs) # I'm not sure what should go in the super here (I'm using python3) @classmethod def normal_constructor(cls, *args, **kwargs): self = cls.__new__(cls) self.__init__(*args, **kwargs) return self Is one of these correct? Or do they all do the same thing? I was looking for documentation for what exactly `type.__call__` does so that I can emulate it, but I wasn't able to find any docs explicitly detailing what that method does. If someone knows where this info is that would be great too. -- https://mail.python.org/mailman/listinfo/python-list
Re: __getattribute__'s error is not available in __getattr__
Thank you Ethan and Chris for the tips. I may be able to adapt that decorator for my use cases -- I hadn't thought of using something like that. Ethan, I'll drop a note over at Python Ideas too with some details about this. Thanks for your help, Jason On Tue, May 2, 2017 at 9:47 PM, Chris Angelicowrote: > On Wed, May 3, 2017 at 12:37 PM, Ethan Furman wrote: > > Until then Chris' decorator is probably the easiest work around -- but > you > > should only catch AttributeError, not everything. > > It does. (Or rather, it catches only those exception(s) listed as > parameters.) Any other exceptions pass through unchanged. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: __getattribute__'s error is not available in __getattr__
@Steve they asked me to move it here because it was more fitting. I hope that's okay? After some testing, it looks like I'm okay with how things work if the problem-object isn't a descriptor (although I do still things it's a bit odd that an error gets squashed, but it's no big deal). However, I use @property all the time, so that's likely why I'm running into this so often. If you can help explain to me what's going on in that case, that would be great! I have two examples below for the sake of discussion. Here's an example where the underlying error is completely hidden: class A(object): def _some_complex_code_hidden_from_the_user(self): # Run a bunch of complex stuff that raises an attribute error internally # This could go layers deep into different modules return self.this_doesnt_exist @property def x(self): return self._some_complex_code_hidden_from_the_user() def __getattr__(self, attr): raise AttributeError("raised from A.__getattr__ to stop execution") a = A() print(a.x) This results in the following output: Traceback (most recent call last): File "test3.py", line 17, in print(a.x) File "test3.py", line 14, in __getattr__ raise AttributeError("raised from A.__getattr__ to stop execution") AttributeError: raised from A.__getattr__ to stop execution Here the real reason the code errors (`sys.this_doesnt_exist` throwing an AttributeError) is not in the traceback's stack. My current opinion on this is that the error that triggered __getattr__ should be passed to __getattr__. This would allow us to use the "raise from" syntax. Here's an example where I manually fix this for a specific class: class A(object): def _some_complex_code_hidden_from_the_user(self): # a bunch of complex stuff that raises an attribute error internally # This could go layers deep into different modules return self.this_doesnt_exist @property def x(self): return self._some_complex_code_hidden_from_the_user() def __getattribute__(self, attr): try: return super().__getattribute__(attr) except AttributeError as error: return self.__custom_getattr__(attr, error) def __custom_getattr__(self, attr, error): raise AttributeError("raised from A.__getattr__ to stop execution") from error #def __getattr__(self, attr): #raise AttributeError("raised from A.__getattr__ to stop execution") from error a = A() print(a.x) This code correctly prints `AttributeError: module 'sys' has no attribute 'this_doesnt_exist'` because I `raise from error`. Note that __getattr__ can't be defined; if it is, it will squash the error message from __custom_getattr__ analogous to what's going on in the first example. More importantly, note that __getattribute__ can't just `return self.__getattr__(attr)` in its except block because __getattribute__ isn't supposed to directly call __getattr__ (because __getattr__ gets triggered later in the python attribute lookup order). Unfortunately, this behavior is not generalizable without overriding __getattribute__ and defining __custom_getattr__ on every single class in the inheritance structure if I wanted it work for each class. The 2nd example is the behavior I desire, and I don't really understand how the descriptor figures into this problem to change that behavior. -- I.e. if you just remove @property from the first example, it returns the full error stack exactly like we'd expect. That means the @property is changing the call order (?) in some way that I don't understand. Thanks! Jason On Tue, May 2, 2017 at 7:11 PM, Ethan Furman <et...@stoneleaf.us> wrote: > On 05/02/2017 11:16 AM, Jason Maldonis wrote: > > Here is the simplest example showing what I mean (there are many more >> complicating variations this, and unfortunately I'm running into some of >> them): >> >> class A(object): >> def __getattr__(self, attr): >> raise AttributeError("raised from A.__getattr__ to stop >> execution") >> >> a = A() >> print(a.x) >> >> results in: >> >> Traceback (most recent call last): >>File "test.py", line 6, in >> print(a.x) >>File "test.py", line 3, in __getattr__ >> raise AttributeError("raised from A.__getattr__ to stop execution") >> AttributeError: raised from A.__getattr__ to stop execution >> >> The thing to note here is that the AttributeError on the normal >> __getattribute__'s lookup isn't in the stack trace. That error is: >> Traceback (most recent call last): >>File "test2.py", line 35, in >> print(a.x) >> AttributeError: 'A' object has no attribute 'x' >>
__getattribute__'s error is not available in __getattr__
Moving this conversation from python-dev to here because I shouldn't have sent it to python-dev. My original message: I'm working on a large class architecture and I find myself often overloading __getattr__. I am continuously running into the issue where I want __getattr__ to have access to the error that was raised in __getattribute__, but it seems completely unavailable. Is that true? What I've learned since then: There is some weird interplay between __getattribute__, __getattr__, and python's attribute lookup order which I don't entirely understand. I've learned some things from Nick Coghlan and Joao S. O. Bueno, which I'll try to summarize: * __getattribute__ does not call __getattr__ within itself; instead, __getattr__ seems to be called during python's natural attribute lookup order, but only if an AttributeError is raised during this lookup. - This has the side effect that calling __getattr__ from __getattribute__ doesn't work properly. * If __getattr__ is triggered (when an AttributeError is raised during attribute lookup), it will not have any information about the AttributeError that triggered it. * Things may get more complicated if we are doing an attribute lookup on a property, and an AttributeError is raised within the property's method. (This is because properties are data descriptors, which complicates python's natural lookup order.) Here is the simplest example showing what I mean (there are many more complicating variations this, and unfortunately I'm running into some of them): class A(object): def __getattr__(self, attr): raise AttributeError("raised from A.__getattr__ to stop execution") a = A() print(a.x) results in: Traceback (most recent call last): File "test.py", line 6, in print(a.x) File "test.py", line 3, in __getattr__ raise AttributeError("raised from A.__getattr__ to stop execution") AttributeError: raised from A.__getattr__ to stop execution The thing to note here is that the AttributeError on the normal __getattribute__'s lookup isn't in the stack trace. That error is: Traceback (most recent call last): File "test2.py", line 35, in print(a.x) AttributeError: 'A' object has no attribute 'x' -- that last line, "AttributeError: 'A' object has no attribute 'x'" does not appear in the stack trace for my above example (because __getattr__ is implemented). This is because in python's attribute lookup order, __getattr__ is called if an AttributeError is raised, and that raised AttributeError gets completely discarded. So basically I want access to the intermediate AttributeError that caused __getattr__ to be raised in the first place. This is complicated, and I may have explained something poorly. If so, please don't hesitate to ask for more explanation or examples. This is already long, so I'll stop typing now. Thanks, Jason -- https://mail.python.org/mailman/listinfo/python-list