Re: [Numpy-discussion] Multidimension array access in C via Python API
That's a very clever approach. I also found a way using the pandas library with the groupby function. points_df = pandas.DataFrame.from_records(buffer) new_buffer = points_df.groupby(qcut(points_df.index, resolution**3)).mean() I did the original approach with all of those loops because I need a way to measure and report on progress, and although these advanced functions are great, they are still asynchronous and blocking and provides no means of indicating progress. Still cool though, thanks! -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42736.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimension array access in C via Python API
This wasn't intended to be a histogram, but you're right in that it would be much better if I can just go through each point once and bin the results, that makes more sense, thanks! -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42733.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimension array access in C via Python API
The points are indeed arbitrarily spaced, and yes I have heard tale of using spatial indices for this sort of problem, and it looks like that would be the best bet for me. Thanks for the other suggestions as well! -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42732.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimension array access in C via Python API
The idea is that I want to thin a large 2D buffer of x,y,z points to a given resolution by dividing the data into equal sized "cubes" (i.e. resolution is number of cubes along each axis) and averaging the points inside each cube (if any). *# Fill up buffer data for demonstration purposes with initial buffer of size 10,000,000 to reduce to 1,000,000 size = 1000 buffer = np.ndarray(shape=(size,3), dtype=np.float) # fill it up buffer[:, 0] = np.random.ranf(size) buffer[:, 1] = np.random.ranf(size) buffer[:, 2] = np.random.ranf(size) # Create result buffer to size of cubed resolution (i.e. 100 ^ 3 = 1,000,000) resolution = 100 thinned_buffer = np.ndarray(shape=(resolution ** 3,3), dtype=np.float) # Trying to convert the following into C to speed it up x_buffer = buffer[:, 0] y_buffer = buffer[:, 1] z_buffer = buffer[:, 2] min_x = x_buffer.min() max_x = x_buffer.max() min_y = y_buffer.min() max_y = y_buffer.max() min_z = z_buffer.min() max_z = z_buffer.max() z_block = (max_z - min_z) / resolution x_block = (max_x - min_x) / resolution y_block = (max_y - min_y) / resolution current_idx = 0 x_idx = min_x while x_idx < max_x: y_idx = min_y while y_idx < max_y: z_idx = min_z while z_idx < max_z: inside_block_points = np.where((x_buffer >= x_idx) & (x_buffer <= x_idx + x_block) & (y_buffer >= y_idx) & (y_buffer <= y_idx + y_block) & (z_buffer >= z_idx) & (z_buffer <= z_idx + z_block)) if inside_block_points[0].size > 0: mean_point = buffer[inside_block_points[0]].mean(axis=0) thinned_buffer[current_idx] = mean_point current_idx += 1 z_idx += z_block y_idx += y_block x_idx += x_block return thin_buffer * -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42726.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimension array access in C via Python API
This is the reason I'm doing this in the first place, because I made a pure python version but it runs really slow for larger data sets, so I'm basically rewriting the same function but using the Python and Numpy C API, but if you're saying it won't run any faster then maybe I'm going at it the wrong way. (Why use the C function version if it's the same speed anyway?) You're suggesting perhaps a cython approach, or perhaps a strictly C/C++ approach given the raw data? -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42719.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimension array access in C via Python API
I think that I do, since I intend to do array specific operations on the resulting column of data. e.g: *PyArray_Min* *PyArray_Max* which require a PyArrayObject argument I also plan to use *PyArray_Where* to find individual point locations in data columns x,y,z within a 3D range, but it doesn't look like it needs PyArrayObject. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42717.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimension array access in C via Python API
Thanks for responding. It looks you made/found these yourself since I can't find anything like this in the API. I can't believe it isn't, so convenient! By the way, from what I understand, the ':' is represented as *PySlice_New(NULL, NULL, NULL) *in the C API when accessing by index, correct? Therefore the final result will be something like: *PyObject* first_column_tuple = PyTuple_New(2); PyTuple_SET_ITEM(first_column_tuple, 0, PySlice_New(NULL, NULL, NULL)); PyTuple_SET_ITEM(first_column_tuple, 1, PyInt_FromLong(0)); PyObject* first_column_buffer = PyObject_GetItem(src_buffer, first_column_tuple); * -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710p42715.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Multidimension array access in C via Python API
Hello, is there a C-API function for numpy that can implement Python's multidimensional indexing? For example, if I had a 2d array: PyArrayObject * M; and an index int i; how do I extract the i-th row M[i,:] or i-th column M[:,i]? Ideally it would be great if it returned another PyArrayObject* object (not a newly allocated one, but whose data will point to the correct memory locations of M). I've searched everywhere in the API documentation, Google, and SO, but no luck. Any help is greatly appreciated. Thank you. -Matthew -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-C-via-Python-API-tp42710.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API: multidimensional array indexing?
Cool! But I'm having trouble implementing this, could you provide an example on how exactly to do this? I'm not sure how to create the appropriate tuple and how to use it with PyObject_GetItem given an PyArrayObject, unless I'm misunderstood. Much appreciated, Matthew -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/C-API-multidimensional-array-indexing-tp7413p42693.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
