MaolmeoX opened a new issue, #525: URL: https://github.com/apache/arrow-js/issues/525
### Describe the bug, including details regarding any error messages, version, and platform. `Vector`, `Table` and `RecordBatch` set `Symbol.isConcatSpreadable` on their prototypes when the module loads ([vector.ts](https://github.com/apache/arrow-js/blob/e053f86f70bb0ae6efb490ef5053f703d3baa097/src/vector.ts#L372), [table.ts](https://github.com/apache/arrow-js/blob/e053f86f70bb0ae6efb490ef5053f703d3baa097/src/table.ts#L403), [recordbatch.ts](https://github.com/apache/arrow-js/blob/e053f86f70bb0ae6efb490ef5053f703d3baa097/src/recordbatch.ts#L309)). V8 keeps a single process-wide protector for that symbol: as soon as any object defines it, whatever its value, every `Array.prototype.concat` in the process takes the slow path, for the life of the process. So a plain `require('apache-arrow')` slows `concat` down everywhere, including in code that never touches an Arrow object. This reaches applications that do not use Arrow at all. `@elastic/elasticsearch` 8.x requires `apache-arrow` when it is imported (made lazy in 9.4 by elastic/elasticsearch-js#3246). In our LoopBack 4 API, which builds SQL `IN (…)` lists with one `concat` per value, an inclusion over 35,000 rows went from 6 s to 5 min 25 s, blocking the event loop, after we added that client. grafana/grafana#134525 measured `reduce` transformations 8× slower once `apache-arrow` was loaded, and strips the assignments at build time. The symbol does not make `concat` spread Arrow objects either. `concat` reads `length` and indexed properties, which `Vector` does not have and `Table` and `RecordBatch` lack entirely: ```js const { vectorFromArray, tableFromArrays } = require('apache-arrow'); const vector = vectorFromArray([1, 2, 3]); const table = tableFromArrays({ x: Int32Array.from([1, 2, 3]) }); [0].concat(vector); // [ 0, <3 empty items> ] [0].concat(table); // [ 0 ] [0].concat(table.batches[0]); // [ 0 ] ``` Without the symbol, each call returns `[0, <the object>]`, as for any other object. #### Reproduction apache-arrow 21.2.0, Linux: ```js // repro.js function bench() { let acc = []; const start = process.hrtime.bigint(); for (let i = 0; i < 20000; i++) acc = acc.concat([i]); return Number(process.hrtime.bigint() - start) / 1e6; } console.log(`before require: ${bench().toFixed(0)} ms`); require('apache-arrow'); console.log(`after require: ${bench().toFixed(0)} ms`); ``` | Node.js | before `require` | after `require` | | --- | --- | --- | | 20.20.2 | 468 ms | 48,390 ms | | 24.19.0 | 438 ms | 47,320 ms | With `node --allow-natives-syntax`, `%IsConcatSpreadableProtector()` returns `true` before `require('apache-arrow')` and `false` after it. With the three assignments deleted from the published files, the second run takes 458 ms on Node.js 24.19.0 and the protector stays `true`. Setting the property to `undefined` instead of `true` is not enough: defining it at all invalidates the protector. #### Suggested fix Remove `Symbol.isConcatSpreadable` from the three prototypes, together with the matching declarations in their interfaces (for example [vector.ts](https://github.com/apache/arrow-js/blob/e053f86f70bb0ae6efb490ef5053f703d3baa097/src/vector.ts#L55-L58)): ```diff (proto as any)._offsets = new Uint32Array([0]); - (proto as any)[Symbol.isConcatSpreadable] = true; (proto as any)[kVectorSymbol] = true; ``` No `concat` in `src/` passes an Arrow object to `Array.prototype.concat`: the calls concatenate arrays of `Data`, or are the classes' own `concat` methods. With the three assignments and declarations removed, `tsc`, `eslint` and the unit tests (`jestconfigs/jest.src.config.js`, 49 suites) pass. Code that passes a `Vector` to `Array.prototype.concat` gets the `Vector` as one element instead of empty slots; `[...vector]` and `vector.toArray()` already give its values. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
