Hi Ivan, Thank you for reply. I made a mistake on the first test when I used the REST API. The host from which I ran the tests and the network bandwidth were not the same. I repeated the test and the results I got using Ignite REST API and JDBC are almost the same.
REST API - 1.7 seconds JDBC - 2 seconds Regarding ODBC and pyodbc, I made a new test and SELECT * FROM TABLE LIMIT 20000 *is also around two seconds. * Thank you very much for your reply, it helped me a lot. Best regards Dren On Tue, Mar 14, 2023 at 12:47 PM Ivan Daschinsky <[email protected]> wrote: > Hi, Dren! > > Unfortunatelly, pyignite doesn't have an efficient native serialization > library, whereas psycopg2 has (it is a thin wrapper around libpq). > > I would suggest two options: > 1. Reduce a default batch size like this : `client.sql("SELECT * FROM > TABLE", page_size=10)`. Default 1024 seems too big and parsing of such a > big response seems to be really slow. > 2. Use ignite odbc driver and pyodbc over it. Both of them work pretty > well. > > вт, 14 мар. 2023 г. в 14:10, Dren Butković <[email protected]>: > >> >> Ignite and py client versions: >> >> - Apache Ignite 2.13.0 >> - pyignite 0.5.2 >> >> On Tue, Mar 14, 2023 at 11:46 AM Zhenya Stanilovsky via user < >> [email protected]> wrote: >> >>> Hi, plz append ignite and py client versions. >>> >>> >>> Hi, >>> >>> I made a speed comparison of retrieving data from Apache Ignite using >>> several methods. All records are in one table, I did not use any WHERE >>> condition, only a SELECT * FROM TABLE XYZ LIMIT 20000. >>> >>> Test results are: >>> Apache Ignite >>> >>> - Apache Ignite REST API - 0.52 seconds >>> - JDBC - 4 seconds >>> - Python pyignite - 40 seconds !!! >>> >>> pseudocode in Python using pyignite: >>> >>> client = Client(username="ignite", password="pass", use_ssl=False) >>> client.connect('localhost', 10800) >>> >>> cursor=client.sql('SELECT * FROM TABLE_XYZ LIMIT 20000')for row in cursor: >>> pass >>> >>> After that I made a speed comparison of retrieving data from PostgreSQL >>> using JDBC and psycopg2 Python package. SQL select is same, SELECT * FROM >>> TABLE XYZ LIMIT 20000 >>> PostgreSQL >>> >>> - JDBC - 3 seconds >>> - Python psycopg2 using fetchall - 3 seconds >>> - Python psycopg2 using fetchone - 4 seconds >>> >>> pseudocode in Python using psycopg2: >>> >>> import psycopg2 >>> >>> conn = psycopg2.connect(database=DB_NAME, >>> user=DB_USER, >>> password=DB_PASS, >>> host=DB_HOST, >>> port=DB_PORT) >>> >>> cur = conn.cursor() >>> cur.execute("SELECT * FROM TABLE_XYZ LIMIT 20000") >>> rows = cur.fetchall()for data in rows: >>> pass >>> >>> I can conclude that the pyignite implementation has much worse >>> performance compared to psycopg2 tests. The performance difference on >>> PostgreSQL between Java JDBC and Python psycopg2 is negligible. >>> >>> The performance difference on Apache Ignite between Java JDBC and Python >>> pyignite is very big. >>> >>> Please if someone can comment on the tests, did I do something wrong or >>> are these results expected? How can such large differences in execution >>> times be explained? Do you have any suggestions to get better results using >>> pyignite? >>> >>> Thank you >>> >>> >>> >>> >>> >>> >> > > -- > Sincerely yours, Ivan Daschinskiy >
