New issue 3014: JIT Issue inlining `struct.unpack('<HH', bytearray(...))` produces incorrect results https://bitbucket.org/pypy/pypy/issues/3014/jit-issue-inlining-structunpack-hh
Jason Madden: Versions: * Python 2.7.13 \(8cdda8b8cdb8, Apr 14 2019, 14:06:58\) \[PyPy 7.1.1 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 \(clang-1000.11.45.5\)\] \(macOS 10.14\) * Python 3.6.1 \(784b254d6699, Apr 14 2019, 10:22:55\) \[PyPy 7.1.1-beta0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 \(clang-1000.11.45.5\)\] * Python 2.7.13 \(8cdda8b8cdb8, Apr 14 2019, 14:06:58\) \[PyPy 7.1.1 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 \(clang-1000.11.45.5\)\] I’ve been working on getting RelStorage to run on PyPy with \(pure-Python\) mysql-connector-python at [https://github.com/zodb/relstorage/issues/228](https://github.com/zodb/relstorage/issues/228#issuecomment-492423284), but was running into a very strange, non-sensical error only on PyPy partway through the test sequence. In a nutshell, the value of the `status_flag` in the OK result packet from the MySQL server is coming back with the `ServerFlag.MORE_RESULTS_EXIST` bit set, leading to the error. But that’s not actually the packet that the server is sending. `protocol.py` has `parse_ok` that parses the `status_flag` from a packet \(a `bytearray`\) like this: ```python def parse_ok(self, packet): """Parse a MySQL OK-packet""" if not packet[4] == 0: raise errors.InterfaceError("Failed parsing OK packet (invalid).") ok_packet = {} ok_packet['orig_data'] = bytes(packet) try: ok_packet['field_count'] = struct_unpack('<xxxxB', packet[0:5])[0] (packet, ok_packet['affected_rows']) = utils.read_lc_int(packet[5:]) (packet, ok_packet['insert_id']) = utils.read_lc_int(packet) (ok_packet['status_flag'], # This is the bad value ok_packet['warning_count']) = struct_unpack('<HH', packet[0:4]) packet = packet[4:] if packet: (packet, ok_packet['info_msg']) = utils.read_lc_string(packet) ok_packet['info_msg'] = ok_packet['info_msg'].decode('utf-8') except ValueError: raise errors.InterfaceError("Failed parsing OK packet.") return ok_packet ``` \( `struct_unpack` is a small helper function for Python 2 that wraps a `bytearray` into a `buffer` and uses `struct.unpack_from`; I get the same results if I modify the code to use `struct.unpack` directly. Python 3 uses struct.unpack directly already.\) I tweaked the error message to report the packet value, and to also unpack the data at that time. \(This is a cold branch of code that’s never been taken before\) ```python if self._have_next_result: # The status_flag was not what we wanted. from .catch23 import PY2, struct_unpack raise errors.InterfaceError( 'Use cmd_query_iter for statements with multiple queries.' '\n%s\n%r\n%r\nUnpacks to:%r' % ( query, result, raw_result, struct_unpack('<HH', raw_result[0:4]))) ``` ``` InterfaceError: Use cmd_query_iter for statements with multiple queries. INSERT INTO temp_pack_visit (zoid, keep_tid) SELECT zoid, 0 FROM current_object WHERE tid > 274639514178723874 {'field_count': 0, 'affected_rows': 0, 'insert_id': 0, 'status_flag': 3480, 'warning_count': 0, 'info_msg': u'Records: 0 Duplicates: 0 Warnings: 0'} bytearray(b'.\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00&Records: 0 Duplicates: 0 Warnings: 0') Unpacks to: (46, 256) ``` You can see that protocol.py's `parse_ok` method gets a `status_flag` of `3480`, while performing the same unpacking on the same data in this function gets a value of `46` \(which is what CPython gets too\). \*\*Running with `--jit off` solves the problem and lets the tests complete successfully. Also, just running with `--jit inlining=0` solves the problem too.\*\* How can I help with this? I imagine you’ll want a JIT log of some sort? _______________________________________________ pypy-issue mailing list pypy-issue@python.org https://mail.python.org/mailman/listinfo/pypy-issue