Re: [ANNOUNCE] Apache IGNITE python thin client (pyignite) 0.6.0 have been released
Great work Best Regards, Igor On Wed, Nov 16, 2022 at 1:50 PM Ivan Daschinsky wrote: > The Apache Ignite Community is pleased to announce the release of > Apache IGNITE python thin client (pyignite) 0.6.0. > > This new release is mostly the maintenance one. However, there are some > new important features and fixes: > > 1. Fixed non-intuitive automatically setting of flag use_ssl when the > authentication is enabled > 2. Added timeout support for cache operations in the async client. > 3. Fixed incorrect result setting of already completed futures in async > connection implementation > > For the full list of changes, you can look at the RELEASE_NOTES: > https://ignite.apache.org/releases/pyignite/0.6.0/release_notes.html > > You can install this version using pip > >> pip install pyignite=0.6.0 > > Alternatively, you can download sources and binary packages (wheels) from > here: > https://dist.apache.org/repos/dist/release/ignite/pyignite/0.6.0/ > > Please let us know if you have any problems > https://ignite.apache.org/community/resources.html#ask > > Regards, > Ivan Daschinsky on behalf of the Apache Ignite community. >
[ANNOUNCE] Apache IGNITE python thin client (pyignite) 0.6.0 have been released
The Apache Ignite Community is pleased to announce the release of Apache IGNITE python thin client (pyignite) 0.6.0. This new release is mostly the maintenance one. However, there are some new important features and fixes: 1. Fixed non-intuitive automatically setting of flag use_ssl when the authentication is enabled 2. Added timeout support for cache operations in the async client. 3. Fixed incorrect result setting of already completed futures in async connection implementation For the full list of changes, you can look at the RELEASE_NOTES: https://ignite.apache.org/releases/pyignite/0.6.0/release_notes.html You can install this version using pip >> pip install pyignite=0.6.0 Alternatively, you can download sources and binary packages (wheels) from here: https://dist.apache.org/repos/dist/release/ignite/pyignite/0.6.0/ Please let us know if you have any problems https://ignite.apache.org/community/resources.html#ask Regards, Ivan Daschinsky on behalf of the Apache Ignite community.
[ANNOUNCE] Apache IGNITE python thin client (pyignite) 0.5.2 released
The Apache Ignite Community is pleased to announce the release of Apache IGNITE python thin client (pyignite) 0.5.2. This new release of python thin client contains bugfixes and it is recommended to update. Namely: 1. Fixed incorrect partial read from socket in sync client 1. Fixed nested object arrays deserialization For the full list of changes, you can refer to the RELEASE_NOTES. Documentation is available here: https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2/ You can install this version using pip >> pip install pyignite==0.5.2 Alternatively, you can download sources and binary packages (wheels) from here: https://dist.apache.org/repos/dist/release/ignite/pyignite/0.5.2/ Please let us know if you have any problems https://ignite.apache.org/community/resources.html#ask Regards, Ivan Daschinsky on behalf of the Apache Ignite community.
[ANNOUNCE] Apache IGNITE python thin client (pyignite) 0.5.1 released
The Apache Ignite Community is pleased to announce the release of Apache IGNITE python thin client (pyignite) 0.5.1. This new release of python thin client contains bugfixes and a few new features Namely: 1. Event listeners to connection events and query events 2. Optional client's side handshake timeout 3. Logging of connection and queries events For the full list of changes, you can refer to the RELEASE_NOTES. https://ignite.apache.org/releases/pyignite/0.5.1/release_notes.html You can install this version using pip >> pip install pyignite==0.5.1 Alternatively, you can download sources and binary packages (wheels) from here: https://dist.apache.org/repos/dist/release/ignite/pyignite/0.5.1/ Please let us know if you have any problems https://ignite.apache.org/community/resources.html#ask Regards, Ivan Daschinsky on behalf of the Apache Ignite community.
[ANNOUNCE] Apache IGNITE python thin client (pyignite) 0.5.0 released
The Apache Ignite Community is pleased to announce the release of Apache IGNITE python thin client (pyignite) 0.5.0. This new release of python thin client contains new cool features: Namely: 1. Transactions support 2. Expiry policy support. 3. Performance improvement of asyncio version 4. Few bug-fixes For the full list of changes, you can refer to the RELEASE_NOTES. https://ignite.apache.org/releases/pyignite/0.5.0/release_notes.html You can install this version using pip >> pip install pyignite==0.5.0 Alternatively, you can download sources and binary packages (wheels) from here: https://dist.apache.org/repos/dist/release/ignite/pyignite/0.5.0/ Please let us know if you have any problems https://ignite.apache.org/community/resources.html#ask Regards, Ivan Daschinsky on behalf of the Apache Ignite community.
Re: Seconds and milliseconds confusion in python thin client
I have a compromise variant. 1. Large timeouts are set usually only for expire_policy. I suggest to support datetime.timedelta here and int as milliseconds 2. All others timeouts should accept only ints as milliseconds. 3. Only timeout in Connection as sockettimeout should remain float in seconds, because it corresponds to socket.settimeout method in python (accepts floating point number) This is a compromise between usability and backward compatibility WDYT? ср, 16 июн. 2021 г. в 11:09, Ivan Daschinsky : > > I've created ticket for it https://issues.apache.org/jira/browse/IGNITE-14911 > > ср, 16 июн. 2021 г. в 08:37, Ivan Daschinsky : > > > > Ops, i don't even know about it. I believe that this is so rarely used, i > > don't even noticed it. I am talking about transactions and expiry policy. I > > suppose that in the case of sql we can simply change it. > > > > ср, 16 июн. 2021 г., 00:46 Igor Sapego : > >> > >> Why is it not released? > >> > >> I can see client.sql(timeout) in 0.4.0 for example, which is int number of > >> ms. > >> > >> Best Regards, > >> Igor > >> > >> > >> On Tue, Jun 15, 2021 at 11:52 PM Ivan Daschinsky > >> wrote: > >> > >> > BTW, common approach is to treat both ints and floats as seconds. Floats > >> > are used to set timeout with millisecods precision. I.e. > >> > asyncio.sleep(1.0) > >> > and asyncio.sleep(1) pauses coroutine for 1 sec. Lets create ticket for > >> > it, > >> > stop voting for 0.5.0.rc0 and schedule next vote. > >> > > >> > вт, 15 июн. 2021 г., 23:49 Ivan Daschinsky : > >> > > >> > > Igor, I suppose that you are probably right. But there is no need to > >> > > notice or deprecate something. This functionality is not released yet > >> > > > >> > > вт, 15 июн. 2021 г., 23:41 Igor Sapego : > >> > > > >> > >> Hi Igniters, > >> > >> > >> > >> I've noticed a weird behaviour of python thin client. In those places > >> > >> where > >> > >> we have > >> > >> timeouts or any other parameters that take time in some places we > >> > >> treat > >> > it > >> > >> like integer > >> > >> number of milliseconds, in others it can take both floats (as a number > >> > of > >> > >> seconds) > >> > >> and ints (number of milliseconds). This approach looks very confusing > >> > >> to > >> > >> me > >> > >> as > >> > >> it leads to things where tx_start(1) and tx_start(1.0) are not > >> > >> actually > >> > >> the > >> > >> same thing. > >> > >> > >> > >> AFAIK in python the most common way to pass time to such functions is > >> > >> to > >> > >> use floats > >> > >> as a number of seconds. This is the approach I propose to use in our > >> > >> API > >> > >> as > >> > >> well. Let's > >> > >> deprecate usage of ints in those functions with the appropriate > >> > >> warning > >> > >> before getting rid > >> > >> of it. > >> > >> > >> > >> What do you think? > >> > >> > >> > >> Best Regards, > >> > >> Igor > >> > >> > >> > > > >> > > > > > -- > Sincerely yours, Ivan Daschinskiy -- Sincerely yours, Ivan Daschinskiy
Re: Seconds and milliseconds confusion in python thin client
I've created ticket for it https://issues.apache.org/jira/browse/IGNITE-14911 ср, 16 июн. 2021 г. в 08:37, Ivan Daschinsky : > > Ops, i don't even know about it. I believe that this is so rarely used, i > don't even noticed it. I am talking about transactions and expiry policy. I > suppose that in the case of sql we can simply change it. > > ср, 16 июн. 2021 г., 00:46 Igor Sapego : >> >> Why is it not released? >> >> I can see client.sql(timeout) in 0.4.0 for example, which is int number of >> ms. >> >> Best Regards, >> Igor >> >> >> On Tue, Jun 15, 2021 at 11:52 PM Ivan Daschinsky >> wrote: >> >> > BTW, common approach is to treat both ints and floats as seconds. Floats >> > are used to set timeout with millisecods precision. I.e. asyncio.sleep(1.0) >> > and asyncio.sleep(1) pauses coroutine for 1 sec. Lets create ticket for it, >> > stop voting for 0.5.0.rc0 and schedule next vote. >> > >> > вт, 15 июн. 2021 г., 23:49 Ivan Daschinsky : >> > >> > > Igor, I suppose that you are probably right. But there is no need to >> > > notice or deprecate something. This functionality is not released yet >> > > >> > > вт, 15 июн. 2021 г., 23:41 Igor Sapego : >> > > >> > >> Hi Igniters, >> > >> >> > >> I've noticed a weird behaviour of python thin client. In those places >> > >> where >> > >> we have >> > >> timeouts or any other parameters that take time in some places we treat >> > it >> > >> like integer >> > >> number of milliseconds, in others it can take both floats (as a number >> > of >> > >> seconds) >> > >> and ints (number of milliseconds). This approach looks very confusing to >> > >> me >> > >> as >> > >> it leads to things where tx_start(1) and tx_start(1.0) are not actually >> > >> the >> > >> same thing. >> > >> >> > >> AFAIK in python the most common way to pass time to such functions is to >> > >> use floats >> > >> as a number of seconds. This is the approach I propose to use in our API >> > >> as >> > >> well. Let's >> > >> deprecate usage of ints in those functions with the appropriate warning >> > >> before getting rid >> > >> of it. >> > >> >> > >> What do you think? >> > >> >> > >> Best Regards, >> > >> Igor >> > >> >> > > >> > -- Sincerely yours, Ivan Daschinskiy
Re: Seconds and milliseconds confusion in python thin client
Ops, i don't even know about it. I believe that this is so rarely used, i don't even noticed it. I am talking about transactions and expiry policy. I suppose that in the case of sql we can simply change it. ср, 16 июн. 2021 г., 00:46 Igor Sapego : > Why is it not released? > > I can see client.sql(timeout) in 0.4.0 for example, which is int number of > ms. > > Best Regards, > Igor > > > On Tue, Jun 15, 2021 at 11:52 PM Ivan Daschinsky > wrote: > > > BTW, common approach is to treat both ints and floats as seconds. Floats > > are used to set timeout with millisecods precision. I.e. > asyncio.sleep(1.0) > > and asyncio.sleep(1) pauses coroutine for 1 sec. Lets create ticket for > it, > > stop voting for 0.5.0.rc0 and schedule next vote. > > > > вт, 15 июн. 2021 г., 23:49 Ivan Daschinsky : > > > > > Igor, I suppose that you are probably right. But there is no need to > > > notice or deprecate something. This functionality is not released yet > > > > > > вт, 15 июн. 2021 г., 23:41 Igor Sapego : > > > > > >> Hi Igniters, > > >> > > >> I've noticed a weird behaviour of python thin client. In those places > > >> where > > >> we have > > >> timeouts or any other parameters that take time in some places we > treat > > it > > >> like integer > > >> number of milliseconds, in others it can take both floats (as a number > > of > > >> seconds) > > >> and ints (number of milliseconds). This approach looks very confusing > to > > >> me > > >> as > > >> it leads to things where tx_start(1) and tx_start(1.0) are not > actually > > >> the > > >> same thing. > > >> > > >> AFAIK in python the most common way to pass time to such functions is > to > > >> use floats > > >> as a number of seconds. This is the approach I propose to use in our > API > > >> as > > >> well. Let's > > >> deprecate usage of ints in those functions with the appropriate > warning > > >> before getting rid > > >> of it. > > >> > > >> What do you think? > > >> > > >> Best Regards, > > >> Igor > > >> > > > > > >
Re: Seconds and milliseconds confusion in python thin client
Why is it not released? I can see client.sql(timeout) in 0.4.0 for example, which is int number of ms. Best Regards, Igor On Tue, Jun 15, 2021 at 11:52 PM Ivan Daschinsky wrote: > BTW, common approach is to treat both ints and floats as seconds. Floats > are used to set timeout with millisecods precision. I.e. asyncio.sleep(1.0) > and asyncio.sleep(1) pauses coroutine for 1 sec. Lets create ticket for it, > stop voting for 0.5.0.rc0 and schedule next vote. > > вт, 15 июн. 2021 г., 23:49 Ivan Daschinsky : > > > Igor, I suppose that you are probably right. But there is no need to > > notice or deprecate something. This functionality is not released yet > > > > вт, 15 июн. 2021 г., 23:41 Igor Sapego : > > > >> Hi Igniters, > >> > >> I've noticed a weird behaviour of python thin client. In those places > >> where > >> we have > >> timeouts or any other parameters that take time in some places we treat > it > >> like integer > >> number of milliseconds, in others it can take both floats (as a number > of > >> seconds) > >> and ints (number of milliseconds). This approach looks very confusing to > >> me > >> as > >> it leads to things where tx_start(1) and tx_start(1.0) are not actually > >> the > >> same thing. > >> > >> AFAIK in python the most common way to pass time to such functions is to > >> use floats > >> as a number of seconds. This is the approach I propose to use in our API > >> as > >> well. Let's > >> deprecate usage of ints in those functions with the appropriate warning > >> before getting rid > >> of it. > >> > >> What do you think? > >> > >> Best Regards, > >> Igor > >> > > >
Re: Seconds and milliseconds confusion in python thin client
BTW, common approach is to treat both ints and floats as seconds. Floats are used to set timeout with millisecods precision. I.e. asyncio.sleep(1.0) and asyncio.sleep(1) pauses coroutine for 1 sec. Lets create ticket for it, stop voting for 0.5.0.rc0 and schedule next vote. вт, 15 июн. 2021 г., 23:49 Ivan Daschinsky : > Igor, I suppose that you are probably right. But there is no need to > notice or deprecate something. This functionality is not released yet > > вт, 15 июн. 2021 г., 23:41 Igor Sapego : > >> Hi Igniters, >> >> I've noticed a weird behaviour of python thin client. In those places >> where >> we have >> timeouts or any other parameters that take time in some places we treat it >> like integer >> number of milliseconds, in others it can take both floats (as a number of >> seconds) >> and ints (number of milliseconds). This approach looks very confusing to >> me >> as >> it leads to things where tx_start(1) and tx_start(1.0) are not actually >> the >> same thing. >> >> AFAIK in python the most common way to pass time to such functions is to >> use floats >> as a number of seconds. This is the approach I propose to use in our API >> as >> well. Let's >> deprecate usage of ints in those functions with the appropriate warning >> before getting rid >> of it. >> >> What do you think? >> >> Best Regards, >> Igor >> >
Re: Seconds and milliseconds confusion in python thin client
Igor, I suppose that you are probably right. But there is no need to notice or deprecate something. This functionality is not released yet вт, 15 июн. 2021 г., 23:41 Igor Sapego : > Hi Igniters, > > I've noticed a weird behaviour of python thin client. In those places where > we have > timeouts or any other parameters that take time in some places we treat it > like integer > number of milliseconds, in others it can take both floats (as a number of > seconds) > and ints (number of milliseconds). This approach looks very confusing to me > as > it leads to things where tx_start(1) and tx_start(1.0) are not actually the > same thing. > > AFAIK in python the most common way to pass time to such functions is to > use floats > as a number of seconds. This is the approach I propose to use in our API as > well. Let's > deprecate usage of ints in those functions with the appropriate warning > before getting rid > of it. > > What do you think? > > Best Regards, > Igor >
Seconds and milliseconds confusion in python thin client
Hi Igniters, I've noticed a weird behaviour of python thin client. In those places where we have timeouts or any other parameters that take time in some places we treat it like integer number of milliseconds, in others it can take both floats (as a number of seconds) and ints (number of milliseconds). This approach looks very confusing to me as it leads to things where tx_start(1) and tx_start(1.0) are not actually the same thing. AFAIK in python the most common way to pass time to such functions is to use floats as a number of seconds. This is the approach I propose to use in our API as well. Let's deprecate usage of ints in those functions with the appropriate warning before getting rid of it. What do you think? Best Regards, Igor
Re: [DISCUSS] Python thin client development approach.
Ivan, if Petr can`t help here, how can we change it out own ? May be we need some help from pmc chair or someone else ? > >> >>Igniters. Almost half of a year passed since this thread begun. We >>released 0.4.0, we adopted travis-ci and use it as primary source for >>test results but nothing changed in TC >> >>1. We use deprecated pytest-runner (even here >>( https://pypi.org/project/pytest-runner ) you can see deprecation >>notice and advice to use tox instead) in TC. >> This dependency is absolutely unnecessary and cause a bunch of >>problems when pyignite is installed as source from pypi.org local >>mirror >> We just need at least tox in linux agents on TC. >>2. We build release 0.4.0 manually on my and Igor Sapego's laptops. >>There are no release build on TC, despite the fact that all needed >>scripts are included in git repo and comprehensive instruction is >>available. >>For running these scripts, we just need docker available on linux >>agents and some additional packages on windows agents. >> >>I, Igor and Petr Ivanov had personal talk about this few months ago >>but nothing changed. >> >>пт, 22 янв. 2021 г. в 16:12, Ivan Daschinsky < ivanda...@gmail.com >: >>> >>> Igor, I've never talked about complete removal of TC builds. >>> I just suggested to add TC jobs for different python versions and use >>> travis heavily. >>> >>> Currently, we have done most of the tasks, except of run TC tests on >>> different python's versions. >>> >>> >>> >>> пт, 22 янв. 2021 г. в 15:16, Igor Sapego < isap...@apache.org >: >>>> >>>> Ivan, >>>> >>>> Though I generally agree with the approach you've suggested, I can see >>>> a problem here. Since we now have a separate repos for thin clients, for >>>> some features we may need to introduce changes to Ignite and python-thin >>>> repos in a single ticket and we should have an ability to run tests on >>>> with >>>> changes on both python client and server nodes. Current TC suites >>>> provide >>>> such ability, Travis does not. So I believe, it's too soon to abandon TC >>>> for thin >>>> clients, at least until we could solve the issue I've described. >>>> >>>> Best Regards, >>>> Igor >>>> >>>> >>>> On Fri, Dec 25, 2020 at 1:49 PM Nikolay Izhikov < nizhi...@apache.org > >>>> wrote: >>>> >>>> > Hello, Ivan. >>>> > >>>> > I’m +1 for your proposal. >>>> > >>>> > > 25 дек. 2020 г., в 13:14, Ivan Daschinsky < ivanda...@gmail.com > >>>> > написал(а): >>>> > > >>>> > > Hi folks! >>>> > > >>>> > > Since we already have a separate repo for thin-clients [1], [2] >>>> > > I'd like to propose some improvements in development process/ >>>> > > >>>> > > 1. We should simplify and automate unit tests run for different >>>> versions >>>> > of >>>> > > python >>>> > > 2. We should add travis integration per commit and pr. Tests could >>>> be run >>>> > > against latest dockered image of ignite >>>> > > 3. There should be ability to run tests against multiple pythons on >>>> TC >>>> > > 4. For thin client development process, travis should be the first >>>> > option. >>>> > > TC suite should be used only to check that compatibility is not >>>> broken >>>> > > and when new functionality is developed (rare case). >>>> > > >>>> > > I've prepared fix [3], you can see successful builds for travis. It >>>> uses >>>> > > tox [5], the most common tool to run tests in multiple environments. >>>> > > There are few environments set up in tox.ini -- with and without >>>> docker, >>>> > > with or without ssl, etc. This helped a lot >>>> > > to setup travis CI build (you can see in commits list of PR) and >>>> simplify >>>> > > run tests for developers. Also docker-compose was introduced >>>> > > to help python thin client developers. >>>> > > >>>> > > But I need some assistance for TC: >>>> > > 1. There is outdated python setuptools on TC agents, so tests >>>> cannot be >>>> > run >>>> > > with updated pytest etc. >>>> > > 2. Multiple pythons should be installed on TC agents (via >>>> > > https://github.com/pyenv/pyenv ), latest minor versions >>>> > > for 3.6, 3.7 and 3.8 >>>> > > 3. After that, TC job should be changed to utilize tox >>>> > > >>>> > > WDYT about this initiative? >>>> > > >>>> > > >>>> > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 >>>> > > [2] -- https://github.com/apache/ignite-python-thin-client >>>> > > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 >>>> > > [4] -- >>>> > https://github.com/apache/ignite-python-thin-client/pull/1/commits >>>> > > [5] -- https://tox.readthedocs.io/en/latest/ >>>> > > >>>> > > -- >>>> > > Sincerely yours, Ivan Daschinskiy >>>> > >>>> > >>> >>> >>> >>> -- >>> Sincerely yours, Ivan Daschinskiy > > > >
Re: [DISCUSS] Python thin client development approach.
Igniters. Almost half of a year passed since this thread begun. We released 0.4.0, we adopted travis-ci and use it as primary source for test results but nothing changed in TC 1. We use deprecated pytest-runner (even here (https://pypi.org/project/pytest-runner) you can see deprecation notice and advice to use tox instead) in TC. This dependency is absolutely unnecessary and cause a bunch of problems when pyignite is installed as source from pypi.org local mirror We just need at least tox in linux agents on TC. 2. We build release 0.4.0 manually on my and Igor Sapego's laptops. There are no release build on TC, despite the fact that all needed scripts are included in git repo and comprehensive instruction is available. For running these scripts, we just need docker available on linux agents and some additional packages on windows agents. I, Igor and Petr Ivanov had personal talk about this few months ago but nothing changed. пт, 22 янв. 2021 г. в 16:12, Ivan Daschinsky : > > Igor, I've never talked about complete removal of TC builds. > I just suggested to add TC jobs for different python versions and use travis > heavily. > > Currently, we have done most of the tasks, except of run TC tests on > different python's versions. > > > > пт, 22 янв. 2021 г. в 15:16, Igor Sapego : >> >> Ivan, >> >> Though I generally agree with the approach you've suggested, I can see >> a problem here. Since we now have a separate repos for thin clients, for >> some features we may need to introduce changes to Ignite and python-thin >> repos in a single ticket and we should have an ability to run tests on with >> changes on both python client and server nodes. Current TC suites provide >> such ability, Travis does not. So I believe, it's too soon to abandon TC >> for thin >> clients, at least until we could solve the issue I've described. >> >> Best Regards, >> Igor >> >> >> On Fri, Dec 25, 2020 at 1:49 PM Nikolay Izhikov wrote: >> >> > Hello, Ivan. >> > >> > I’m +1 for your proposal. >> > >> > > 25 дек. 2020 г., в 13:14, Ivan Daschinsky >> > написал(а): >> > > >> > > Hi folks! >> > > >> > > Since we already have a separate repo for thin-clients [1], [2] >> > > I'd like to propose some improvements in development process/ >> > > >> > > 1. We should simplify and automate unit tests run for different versions >> > of >> > > python >> > > 2. We should add travis integration per commit and pr. Tests could be run >> > > against latest dockered image of ignite >> > > 3. There should be ability to run tests against multiple pythons on TC >> > > 4. For thin client development process, travis should be the first >> > option. >> > > TC suite should be used only to check that compatibility is not broken >> > > and when new functionality is developed (rare case). >> > > >> > > I've prepared fix [3], you can see successful builds for travis. It uses >> > > tox [5], the most common tool to run tests in multiple environments. >> > > There are few environments set up in tox.ini -- with and without docker, >> > > with or without ssl, etc. This helped a lot >> > > to setup travis CI build (you can see in commits list of PR) and simplify >> > > run tests for developers. Also docker-compose was introduced >> > > to help python thin client developers. >> > > >> > > But I need some assistance for TC: >> > > 1. There is outdated python setuptools on TC agents, so tests cannot be >> > run >> > > with updated pytest etc. >> > > 2. Multiple pythons should be installed on TC agents (via >> > > https://github.com/pyenv/pyenv), latest minor versions >> > > for 3.6, 3.7 and 3.8 >> > > 3. After that, TC job should be changed to utilize tox >> > > >> > > WDYT about this initiative? >> > > >> > > >> > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 >> > > [2] -- https://github.com/apache/ignite-python-thin-client >> > > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 >> > > [4] -- >> > https://github.com/apache/ignite-python-thin-client/pull/1/commits >> > > [5] -- https://tox.readthedocs.io/en/latest/ >> > > >> > > -- >> > > Sincerely yours, Ivan Daschinskiy >> > >> > > > > > -- > Sincerely yours, Ivan Daschinskiy -- Sincerely yours, Ivan Daschinskiy
[ANNOUNCE] Apache IGNITE python thin client (pyignite) 0.4.0 released
The Apache Ignite Community is pleased to announce the release of Apache IGNITE python thin client (pyignite) 0.4.0. This new release of python thin client contains a lot of changes. Namely: 1. Partition awareness support. 2. Asyncio support. 3. Numerous performance fixes. 4. Numerous bug fixes. For the full list of changes, you can refer to the RELEASE_NOTES. https://ignite.apache.org/releases/pyignite/0.4.0/release_notes.html You can install this version using pip >> pip install pyignite==0.4.0 Alternatively, you can download sources and binary packages (wheels) from here: https://archive.apache.org/dist/ignite/pyignite/0.4.0/ Please let us know if you have any problems https://ignite.apache.org/community/resources.html#ask Regards, Ivan Daschinsky on behalf of the Apache Ignite community.
[jira] [Created] (IGNITE-14595) ExpiryPolicy support to python thin client
Ivan Daschinsky created IGNITE-14595: Summary: ExpiryPolicy support to python thin client Key: IGNITE-14595 URL: https://issues.apache.org/jira/browse/IGNITE-14595 Project: Ignite Issue Type: New Feature Components: python, thin client Reporter: Ivan Daschinsky It's of a crucial importance to add support for expiration policy in python client. This is on of the most used feature of any cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSSION] Release 0.4.0 version of ignite python thin client (pyignite)
Igniters! I suppose it is time to finally release new version of python thin client Recently, multiple bug fixes, features and other goodies have been implemented. For example: 1. Partition awareness. 2. Asyncio support. 3. Cluster API support. 4. Multiple performance improvements, especially for BinaryObjects and large data chunks. Full list of issues are here [1] I suppose that scope is already fixed and done, so it's time to test and release. Unfortunately, there is no ready infrastructure and TC jobs, so we probably should build artifacts on personal computers. Igor Sapego and me can prepare release artifacts on our own. Documentation is ready and is here [2]. Documentation for release will be built automatically after cutting the release branch. After finishing of voting procedure, signed artifacts will be uploaded to pypi.org by me or Igor Sapego. Please, share your opinions about subj. [1] -- https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%20python-0.4.0 [2] https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/ -- Sincerely yours, Ivan Daschinskiy
[jira] [Created] (IGNITE-14465) Add the ability to activate the cluster via the Python thin client
Igor Sapego created IGNITE-14465: Summary: Add the ability to activate the cluster via the Python thin client Key: IGNITE-14465 URL: https://issues.apache.org/jira/browse/IGNITE-14465 Project: Ignite Issue Type: Bug Components: python, thin client Affects Versions: python-0.3.4 Reporter: Igor Sapego Assignee: Igor Sapego Fix For: python-0.4.0 This feature will be extremely useful when working with clusters that have the "persistenceEnabled" option. Since they require activation to get started. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14429) Python thin client cache.get_size works not as expected and PeekModes are incorrect.
Ivan Daschinskiy created IGNITE-14429: - Summary: Python thin client cache.get_size works not as expected and PeekModes are incorrect. Key: IGNITE-14429 URL: https://issues.apache.org/jira/browse/IGNITE-14429 Project: Ignite Issue Type: Bug Components: python, thin client Reporter: Ivan Daschinskiy 1. PeekModes is now ByteArray, so class variables should be changed. Currently these values are incorrect, seems like masks. They should be changed to ordinal values in order to resemble java enum. 2. By default, peek_modes in get_size should be None, not 0 * If pass 0, behaviour is not if we use PeekModes.ALL, but PeekModes.PRIMARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14240) Handle authentication error on python thin client properly
Ivan Daschinskiy created IGNITE-14240: - Summary: Handle authentication error on python thin client properly Key: IGNITE-14240 URL: https://issues.apache.org/jira/browse/IGNITE-14240 Project: Ignite Issue Type: Bug Components: python, thin client Reporter: Ivan Daschinskiy -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14186) Develop C module for python thin client to speedup hashcode calculation
Ivan Daschinskiy created IGNITE-14186: - Summary: Develop C module for python thin client to speedup hashcode calculation Key: IGNITE-14186 URL: https://issues.apache.org/jira/browse/IGNITE-14186 Project: Ignite Issue Type: Improvement Components: python, thin client Reporter: Ivan Daschinskiy Pure python calculation of hashcode is very slow. It leads to inadequate performance of simple operation. For example, put object with 1Mb data takes 500ms. After rewriting hashcode in C, operation tooks only 7ms -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14058) Python Thin client: get boolean type return integers
Igor Sapego created IGNITE-14058:
Summary: Python Thin client: get boolean type return integers
Key: IGNITE-14058
URL: https://issues.apache.org/jira/browse/IGNITE-14058
Project: Ignite
Issue Type: Bug
Components: python, thin client
Affects Versions: python-0.3.4
Reporter: Igor Sapego
Assignee: Igor Sapego
Fix For: python-0.4.0
Steps:
- put boolean value in cluster
{code}
from pyignite import Client
from pyignite.datatypes import *
client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("test")
cache.put(1, [True, False], key_hint=IntObject, value_hint=BoolArrayObject)
{code}
- get value
{code}
from pyignite import Client
from pyignite.datatypes import *
client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("test")
result = cache.get(1, key_hint=IntObject)
print(*[str(arr_item).lower() for arr_item in result])
{code}
Actual:
- return integers
{code}
0 1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Python thin client development approach.
Igor, I've never talked about complete removal of TC builds. I just suggested to add TC jobs for different python versions and use travis heavily. Currently, we have done most of the tasks, except of run TC tests on different python's versions. пт, 22 янв. 2021 г. в 15:16, Igor Sapego : > Ivan, > > Though I generally agree with the approach you've suggested, I can see > a problem here. Since we now have a separate repos for thin clients, for > some features we may need to introduce changes to Ignite and python-thin > repos in a single ticket and we should have an ability to run tests on with > changes on both python client and server nodes. Current TC suites provide > such ability, Travis does not. So I believe, it's too soon to abandon TC > for thin > clients, at least until we could solve the issue I've described. > > Best Regards, > Igor > > > On Fri, Dec 25, 2020 at 1:49 PM Nikolay Izhikov > wrote: > > > Hello, Ivan. > > > > I’m +1 for your proposal. > > > > > 25 дек. 2020 г., в 13:14, Ivan Daschinsky > > написал(а): > > > > > > Hi folks! > > > > > > Since we already have a separate repo for thin-clients [1], [2] > > > I'd like to propose some improvements in development process/ > > > > > > 1. We should simplify and automate unit tests run for different > versions > > of > > > python > > > 2. We should add travis integration per commit and pr. Tests could be > run > > > against latest dockered image of ignite > > > 3. There should be ability to run tests against multiple pythons on TC > > > 4. For thin client development process, travis should be the first > > option. > > > TC suite should be used only to check that compatibility is not broken > > > and when new functionality is developed (rare case). > > > > > > I've prepared fix [3], you can see successful builds for travis. It > uses > > > tox [5], the most common tool to run tests in multiple environments. > > > There are few environments set up in tox.ini -- with and without > docker, > > > with or without ssl, etc. This helped a lot > > > to setup travis CI build (you can see in commits list of PR) and > simplify > > > run tests for developers. Also docker-compose was introduced > > > to help python thin client developers. > > > > > > But I need some assistance for TC: > > > 1. There is outdated python setuptools on TC agents, so tests cannot be > > run > > > with updated pytest etc. > > > 2. Multiple pythons should be installed on TC agents (via > > > https://github.com/pyenv/pyenv), latest minor versions > > > for 3.6, 3.7 and 3.8 > > > 3. After that, TC job should be changed to utilize tox > > > > > > WDYT about this initiative? > > > > > > > > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 > > > [2] -- https://github.com/apache/ignite-python-thin-client > > > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 > > > [4] -- > > https://github.com/apache/ignite-python-thin-client/pull/1/commits > > > [5] -- https://tox.readthedocs.io/en/latest/ > > > > > > -- > > > Sincerely yours, Ivan Daschinskiy > > > > > -- Sincerely yours, Ivan Daschinskiy
Re: [DISCUSS] Python thin client development approach.
Ivan, Though I generally agree with the approach you've suggested, I can see a problem here. Since we now have a separate repos for thin clients, for some features we may need to introduce changes to Ignite and python-thin repos in a single ticket and we should have an ability to run tests on with changes on both python client and server nodes. Current TC suites provide such ability, Travis does not. So I believe, it's too soon to abandon TC for thin clients, at least until we could solve the issue I've described. Best Regards, Igor On Fri, Dec 25, 2020 at 1:49 PM Nikolay Izhikov wrote: > Hello, Ivan. > > I’m +1 for your proposal. > > > 25 дек. 2020 г., в 13:14, Ivan Daschinsky > написал(а): > > > > Hi folks! > > > > Since we already have a separate repo for thin-clients [1], [2] > > I'd like to propose some improvements in development process/ > > > > 1. We should simplify and automate unit tests run for different versions > of > > python > > 2. We should add travis integration per commit and pr. Tests could be run > > against latest dockered image of ignite > > 3. There should be ability to run tests against multiple pythons on TC > > 4. For thin client development process, travis should be the first > option. > > TC suite should be used only to check that compatibility is not broken > > and when new functionality is developed (rare case). > > > > I've prepared fix [3], you can see successful builds for travis. It uses > > tox [5], the most common tool to run tests in multiple environments. > > There are few environments set up in tox.ini -- with and without docker, > > with or without ssl, etc. This helped a lot > > to setup travis CI build (you can see in commits list of PR) and simplify > > run tests for developers. Also docker-compose was introduced > > to help python thin client developers. > > > > But I need some assistance for TC: > > 1. There is outdated python setuptools on TC agents, so tests cannot be > run > > with updated pytest etc. > > 2. Multiple pythons should be installed on TC agents (via > > https://github.com/pyenv/pyenv), latest minor versions > > for 3.6, 3.7 and 3.8 > > 3. After that, TC job should be changed to utilize tox > > > > WDYT about this initiative? > > > > > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 > > [2] -- https://github.com/apache/ignite-python-thin-client > > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 > > [4] -- > https://github.com/apache/ignite-python-thin-client/pull/1/commits > > [5] -- https://tox.readthedocs.io/en/latest/ > > > > -- > > Sincerely yours, Ivan Daschinskiy > >
[jira] [Created] (IGNITE-13967) Refactor and imrpove performance of python thin client marshaller
Ivan Daschinskiy created IGNITE-13967: - Summary: Refactor and imrpove performance of python thin client marshaller Key: IGNITE-13967 URL: https://issues.apache.org/jira/browse/IGNITE-13967 Project: Ignite Issue Type: Improvement Components: thin client Reporter: Ivan Daschinskiy Assignee: Ivan Daschinskiy Currently implemented serialization has questionable design and suffers from some problems 1. It is tightly coupled with Client object 2. It doesn't use protocol feature that total length of message is in the header, thus it constantly load from Client some data instead of iteration over byte array. 3. It uses some tricky hacks and sometimes new connection is created when deserializing object. 4. It constantly allocates bytes (immutable data structure). I suggest to rewrite serialization and deserialization: 1. Pass to corresponding methods specific SerDe context + BytesIO 2. Context can be sync and async and contains specific flags and methods for loading/uploading binary object schemas 3. Refactor Client in order to retrieve full packet from socket at once then pass full packet futher. These steps can significantly improve performance, reduce amount of allocations and give foundation for implementing asyncio version of client. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Python thin client development approach.
Hello, Ivan. I’m +1 for your proposal. > 25 дек. 2020 г., в 13:14, Ivan Daschinsky написал(а): > > Hi folks! > > Since we already have a separate repo for thin-clients [1], [2] > I'd like to propose some improvements in development process/ > > 1. We should simplify and automate unit tests run for different versions of > python > 2. We should add travis integration per commit and pr. Tests could be run > against latest dockered image of ignite > 3. There should be ability to run tests against multiple pythons on TC > 4. For thin client development process, travis should be the first option. > TC suite should be used only to check that compatibility is not broken > and when new functionality is developed (rare case). > > I've prepared fix [3], you can see successful builds for travis. It uses > tox [5], the most common tool to run tests in multiple environments. > There are few environments set up in tox.ini -- with and without docker, > with or without ssl, etc. This helped a lot > to setup travis CI build (you can see in commits list of PR) and simplify > run tests for developers. Also docker-compose was introduced > to help python thin client developers. > > But I need some assistance for TC: > 1. There is outdated python setuptools on TC agents, so tests cannot be run > with updated pytest etc. > 2. Multiple pythons should be installed on TC agents (via > https://github.com/pyenv/pyenv), latest minor versions > for 3.6, 3.7 and 3.8 > 3. After that, TC job should be changed to utilize tox > > WDYT about this initiative? > > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 > [2] -- https://github.com/apache/ignite-python-thin-client > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 > [4] -- https://github.com/apache/ignite-python-thin-client/pull/1/commits > [5] -- https://tox.readthedocs.io/en/latest/ > > -- > Sincerely yours, Ivan Daschinskiy
[DISCUSS] Python thin client development approach.
Hi folks! Since we already have a separate repo for thin-clients [1], [2] I'd like to propose some improvements in development process/ 1. We should simplify and automate unit tests run for different versions of python 2. We should add travis integration per commit and pr. Tests could be run against latest dockered image of ignite 3. There should be ability to run tests against multiple pythons on TC 4. For thin client development process, travis should be the first option. TC suite should be used only to check that compatibility is not broken and when new functionality is developed (rare case). I've prepared fix [3], you can see successful builds for travis. It uses tox [5], the most common tool to run tests in multiple environments. There are few environments set up in tox.ini -- with and without docker, with or without ssl, etc. This helped a lot to setup travis CI build (you can see in commits list of PR) and simplify run tests for developers. Also docker-compose was introduced to help python thin client developers. But I need some assistance for TC: 1. There is outdated python setuptools on TC agents, so tests cannot be run with updated pytest etc. 2. Multiple pythons should be installed on TC agents (via https://github.com/pyenv/pyenv), latest minor versions for 3.6, 3.7 and 3.8 3. After that, TC job should be changed to utilize tox WDYT about this initiative? [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 [2] -- https://github.com/apache/ignite-python-thin-client [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 [4] -- https://github.com/apache/ignite-python-thin-client/pull/1/commits [5] -- https://tox.readthedocs.io/en/latest/ -- Sincerely yours, Ivan Daschinskiy
[GitHub] [ignite-python-thin-client] ivandasch commented on a change in pull request #1: IGNITE-13903 Add tox, docker-compose and travis integration, update pytest versions.
ivandasch commented on a change in pull request #1: URL: https://github.com/apache/ignite-python-thin-client/pull/1#discussion_r548627959 ## File path: docker-compose.yml ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +services: + ignite: +image: apacheignite/ignite:2.9.0 Review comment: Changed to :latest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [ignite-python-thin-client] isapego commented on a change in pull request #1: IGNITE-13903 Add tox, docker-compose and travis integration, update pytest versions.
isapego commented on a change in pull request #1: URL: https://github.com/apache/ignite-python-thin-client/pull/1#discussion_r548558576 ## File path: docker-compose.yml ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +services: + ignite: +image: apacheignite/ignite:2.9.0 Review comment: Are we going to change Ignite version here every time new Ignite is released? Can we somehow avoid specifying exact version here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [ignite-python-thin-client] ivandasch opened a new pull request #1: IGNITE-13903 Add tox, docker-compose and travis integration, update pytest versions.
ivandasch opened a new pull request #1: URL: https://github.com/apache/ignite-python-thin-client/pull/1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (IGNITE-13903) Python thin client tests automation.
Ivan Daschinskiy created IGNITE-13903: - Summary: Python thin client tests automation. Key: IGNITE-13903 URL: https://issues.apache.org/jira/browse/IGNITE-13903 Project: Ignite Issue Type: Improvement Components: python Reporter: Ivan Daschinskiy Assignee: Ivan Daschinskiy It would be nice to futher improve our development process of python-thin-client 1. Add docker-compose.yml to simplify local development 2. Add tox.ini to simplify test running automation 3. Integrate travis-ci build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13863) Python thin client hangs when object has Boolean field
Igor Sapego created IGNITE-13863:
Summary: Python thin client hangs when object has Boolean field
Key: IGNITE-13863
URL: https://issues.apache.org/jira/browse/IGNITE-13863
Project: Ignite
Issue Type: Bug
Components: thin client
Affects Versions: 2.9
Reporter: Igor Sapego
Assignee: Igor Sapego
The first run of python client returns empty result set, all consequence runs
just hang:
{code:java}
from pygridgain import Client, GenericObjectMeta
import collections
from pygridgain.datatypes import *
if __name__ == '__main__':
client = Client()
client.connect('localhost', 10800)
cache = client.get_cache('Batch')
#client.register_binary_type(BatchConfigurationPK)
#client.register_binary_type(BatchConfiguration)
result = cache.scan()
print([b for b in result])
{code}
{code:java}
public class ObjectTest {
@QuerySqlField
private String field1;
@QuerySqlField
private String field2;
@QuerySqlField
private String field3;
@QuerySqlField
private String field4;
@QuerySqlField
private Boolean enabled;
public ObjectTest(String field1, String field2, String field3) {
this.field1 = field1;
this.field2 = field2;
this.field3 = field3;
}
}
{code}
{code:java}
public static void main(String[] args) throws IgniteException {
Ignite ignite = Ignition.start("examples/config/example-ignite.xml");
IgniteCache cache = ignite.getOrCreateCache("Batch");
for (int i = 0; i < 100; i++) {
cache.put(i, new ObjectTest("" + i, "" + i, "" + i));
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13862) Python Thin Client: put_all operation on a huge dictionary (say 10000 keys) results in connection timeout, same happens with multiple consecutive put operations using a
Archita Sukhwani created IGNITE-13862: - Summary: Python Thin Client: put_all operation on a huge dictionary (say 1 keys) results in connection timeout, same happens with multiple consecutive put operations using a for loop unless a sleep statement is put. Key: IGNITE-13862 URL: https://issues.apache.org/jira/browse/IGNITE-13862 Project: Ignite Issue Type: Bug Components: python, thin client Reporter: Archita Sukhwani [Reproducible code|https://stackoverflow.com/questions/65303573/pyignite-connection-timeout-error-when-trying-to-load-json-into-cache-using-put] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13735) Move Python thin client to a separate git repo
Igor Sapego created IGNITE-13735: Summary: Move Python thin client to a separate git repo Key: IGNITE-13735 URL: https://issues.apache.org/jira/browse/IGNITE-13735 Project: Ignite Issue Type: Improvement Components: thin client Affects Versions: 2.9 Reporter: Igor Sapego Assignee: Igor Sapego Fix For: 2.10 We now have separate repo for Python thin client. Need to move python thin client from ignite/modules/platforms/python to [https://gitbox.apache.org/repos/asf/ignite-python-thin-client.git] preserving git history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[TeamCity] Python Thin Client test job improvement
Hello, Igniters. JFYI Currently, we don’t test SSL mode for Python Thin Client. I and Andrey Kuznetsov are working on IGNITE-12718 [1] «Python Thin Client add an ability to specify keyfile password» To test existed SSL support and new feature I need to update «ThinClient: Python» job [2] I will do it in the nearest time. Locally, I use this script. Will use similar approach on TC. ``` #!/bin/sh IGNITE_HOME=/Users/sbt-izhikov-nv/bin/apache-ignite-2.10.0-SNAPSHOT-bin export PYTHON_TEST_CONFIG_PATH=/Users/sbt-izhikov-nv/src/ignite/modules/platforms/python/tests/config $IGNITE_HOME/bin/ignite.sh $PYTHON_TEST_CONFIG_PATH/ssl.xml & IGNITE_PID=$! TEST_OPTS="--use-ssl=True --ssl-certfile=$PYTHON_TEST_CONFIG_PATH/ssl/client_full.pem" python3 setup.py pytest --addopts="$TEST_OPTS" TEST_OPTS="--use-ssl=True --ssl-certfile=$PYTHON_TEST_CONFIG_PATH/ssl/client_with_pass_full.pem --ssl-keyfile-password=654321" python3 setup.py pytest --addopts="$TEST_OPTS" pkill -P $IGNITE_PID ``` [1] https://github.com/apache/ignite/pull/8200 [2] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPython
[jira] [Created] (IGNITE-12867) Python thin client: support for DB-API
Stephen Darlington created IGNITE-12867: --- Summary: Python thin client: support for DB-API Key: IGNITE-12867 URL: https://issues.apache.org/jira/browse/IGNITE-12867 Project: Ignite Issue Type: Improvement Components: python Affects Versions: 2.8 Reporter: Stephen Darlington Python has a database connection standard called [DB-API|http://example.com/]. It would make the Ignite Python thin client a lot more useful if it could use this in addition to its current Ignite-specific APIs. Implementing this would enable other Apache tools such as SuperSet and Airflow to connect to and use Ignite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12718) Python Thin Client: add an ability to specify keyfile password
Andrey Kuznetsov created IGNITE-12718:
-
Summary: Python Thin Client: add an ability to specify keyfile
password
Key: IGNITE-12718
URL: https://issues.apache.org/jira/browse/IGNITE-12718
Project: Ignite
Issue Type: Improvement
Components: thin client
Affects Versions: 2.8
Reporter: Andrey Kuznetsov
In pyignite, there is no way to specify password for keyfile being used to
establish TLS connection to Ignite cluster. If keyfile is encrypted, then
OpenSSL library prompts for password interactively.
In order to add configurable password, one can set up explicit {{SSLContext}}
instead of {{ssl.wrap_socket}} call.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-11854) Serialization of arrays of primitives in python thin client is not optimal
Denis Mekhanikov created IGNITE-11854:
-
Summary: Serialization of arrays of primitives in python thin
client is not optimal
Key: IGNITE-11854
URL: https://issues.apache.org/jira/browse/IGNITE-11854
Project: Ignite
Issue Type: Bug
Components: thin client
Affects Versions: 2.7
Reporter: Denis Mekhanikov
The following code hangs indefinitely inside of invocation to {{my_cache.put()}}
{code:java}
from pyignite import Client
arr_len = 3_000_000
content = bytearray(arr_len)
for i in range(arr_len):
content[i] = i % 256
client = Client()
client.connect('127.0.0.1', 10800)
my_cache = client.get_or_create_cache('my cache')
my_cache.put("key_bin", content){code}
While the value is only 3MB in size. Implementation of serialization of
primitive arrays seems to be quadratic in length of the array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-11593) python thin client: insert and select VARBINARY data type through SQL
Stepan Pilschikov created IGNITE-11593: -- Summary: python thin client: insert and select VARBINARY data type through SQL Key: IGNITE-11593 URL: https://issues.apache.org/jira/browse/IGNITE-11593 Project: Ignite Issue Type: Bug Components: thin client Reporter: Stepan Pilschikov varbin data type select is getting not parsed array of integer values Example: https://gist.github.com/pilshchikov/b98ce5514209ef7009099b9c188398a6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11366) python thin client: add python examples in release build
Stepan Pilschikov created IGNITE-11366: -- Summary: python thin client: add python examples in release build Key: IGNITE-11366 URL: https://issues.apache.org/jira/browse/IGNITE-11366 Project: Ignite Issue Type: Improvement Affects Versions: 2.7 Reporter: Stepan Pilschikov Examples directory should be added in release build because they exists in sources and they really help to understand how to work with this client I think they understandable enough for end user Also they have readme.md which is contain link on examples documentation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11303) Python thin client: best effort affinity
Dmitry Melnichuk created IGNITE-11303: - Summary: Python thin client: best effort affinity Key: IGNITE-11303 URL: https://issues.apache.org/jira/browse/IGNITE-11303 Project: Ignite Issue Type: Task Components: thin client Reporter: Dmitry Melnichuk Assignee: Dmitry Melnichuk The goal is to implement [IEP-23|https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients] using background thread (`threading` module). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10230) Putting collection of timestamps fails on python thin client
Pavel Petroshenko created IGNITE-10230: -- Summary: Putting collection of timestamps fails on python thin client Key: IGNITE-10230 URL: https://issues.apache.org/jira/browse/IGNITE-10230 Project: Ignite Issue Type: Task Components: thin client Affects Versions: 2.6 Reporter: Pavel Petroshenko Assignee: Dmitry Melnichuk Fix For: 2.8 Here's the test that fails: https://gist.github.com/pilshchikov/7434805b7d95973afe600e1c0c83d17a -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Python thin client
Oh, nice Thanks for quick response I've check against JS and PHP, now its works properly Igor, can you approve this patch? сб, 20 окт. 2018 г. в 0:57, Dmitry Melnichuk : > Stepan, > > My bad. I got the Decimal part of Ignite thin client protocol > specification totally wrong. > > Just submitted a patch tested against Java thin client. Please evaluate. > > On 10/20/18 2:00 AM, Stepan Pilschikov wrote: > > Dmitry, > > > > One more thing > > Discover that Decimal data type can't be getting by others clients > > > > Make ticket with description: > > https://issues.apache.org/jira/browse/IGNITE-9950 > > > > Please, can you help to investigate this problem? > > > > ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov < > pilshchikov@gmail.com>: > > > >> Dmitry, > >> > >> Great, now client works properly on all OS > >> Thanks > >> > >> I hope PR will be merged soon > >> > >> Igor, can please help with that? > >> > >> ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk < > >> dmitry.melnic...@nobitlost.com>: > >> > >>> Stepan, > >>> > >>> Sadly, it was a C `long` data type size mismatch between Windows and > >>> other OSes. I investigated this issue and offered a pull request. > >>> > >>> https://github.com/apache/ignite/pull/5017 > >>> > >>> On 10/17/18 6:29 PM, Stepan Pilschikov wrote: > >>>> Dmitry, > >>>> > >>>> I've trying to use python thin client with Ignite 2.7 it dosn't work > >>>> Please help to investigate > >>>> Problem reproduced on several others machines > >>>> > >>>> Issue: > >>>> https://issues.apache.org/jira/browse/IGNITE-9908 > >>>> > >>> > >> > > > >
Re: Python thin client
Stepan, My bad. I got the Decimal part of Ignite thin client protocol specification totally wrong. Just submitted a patch tested against Java thin client. Please evaluate. On 10/20/18 2:00 AM, Stepan Pilschikov wrote: Dmitry, One more thing Discover that Decimal data type can't be getting by others clients Make ticket with description: https://issues.apache.org/jira/browse/IGNITE-9950 Please, can you help to investigate this problem? ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov : Dmitry, Great, now client works properly on all OS Thanks I hope PR will be merged soon Igor, can please help with that? ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk < dmitry.melnic...@nobitlost.com>: Stepan, Sadly, it was a C `long` data type size mismatch between Windows and other OSes. I investigated this issue and offered a pull request. https://github.com/apache/ignite/pull/5017 On 10/17/18 6:29 PM, Stepan Pilschikov wrote: Dmitry, I've trying to use python thin client with Ignite 2.7 it dosn't work Please help to investigate Problem reproduced on several others machines Issue: https://issues.apache.org/jira/browse/IGNITE-9908
Re: Python thin client
Dmitry, One more thing Discover that Decimal data type can't be getting by others clients Make ticket with description: https://issues.apache.org/jira/browse/IGNITE-9950 Please, can you help to investigate this problem? ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov : > Dmitry, > > Great, now client works properly on all OS > Thanks > > I hope PR will be merged soon > > Igor, can please help with that? > > ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk < > dmitry.melnic...@nobitlost.com>: > >> Stepan, >> >> Sadly, it was a C `long` data type size mismatch between Windows and >> other OSes. I investigated this issue and offered a pull request. >> >> https://github.com/apache/ignite/pull/5017 >> >> On 10/17/18 6:29 PM, Stepan Pilschikov wrote: >> > Dmitry, >> > >> > I've trying to use python thin client with Ignite 2.7 it dosn't work >> > Please help to investigate >> > Problem reproduced on several others machines >> > >> > Issue: >> > https://issues.apache.org/jira/browse/IGNITE-9908 >> > >> >
Re: Python thin client
Dmitry, Great, now client works properly on all OS Thanks I hope PR will be merged soon Igor, can please help with that? ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk < dmitry.melnic...@nobitlost.com>: > Stepan, > > Sadly, it was a C `long` data type size mismatch between Windows and > other OSes. I investigated this issue and offered a pull request. > > https://github.com/apache/ignite/pull/5017 > > On 10/17/18 6:29 PM, Stepan Pilschikov wrote: > > Dmitry, > > > > I've trying to use python thin client with Ignite 2.7 it dosn't work > > Please help to investigate > > Problem reproduced on several others machines > > > > Issue: > > https://issues.apache.org/jira/browse/IGNITE-9908 > > >
Re: Python thin client
Stepan, Sadly, it was a C `long` data type size mismatch between Windows and other OSes. I investigated this issue and offered a pull request. https://github.com/apache/ignite/pull/5017 On 10/17/18 6:29 PM, Stepan Pilschikov wrote: Dmitry, I've trying to use python thin client with Ignite 2.7 it dosn't work Please help to investigate Problem reproduced on several others machines Issue: https://issues.apache.org/jira/browse/IGNITE-9908
Re: Python thin client
Dmitry, I've trying to use python thin client with Ignite 2.7 it dosn't work Please help to investigate Problem reproduced on several others machines Issue: https://issues.apache.org/jira/browse/IGNITE-9908 вт, 16 окт. 2018 г. в 4:36, Dmitry Melnichuk : > Igor, > > I do not have access to edit wiki pages, so please fill it. > > All the features are supported, except for: > > - Query API: ScanQuery with filter, > - Binary API: Get object in BinaryObject form, BinaryObject building, > - Additional Features: Best effort affinity, > - Type registration: Enum registration, Type name registration. > > On 10/15/18 9:17 PM, Igor Sapego wrote: > > Dmitry, > > > > Since Python thin client is now in master, can you please add it to our > > "Thin clients features" wiki page, so we can track the feature parity > > of our clients? Or just tell me which features are supported, so I can > fill > > it myself. > > > > [1] - > > https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features > > > > Best Regards, > > Igor > > >
Re: Python thin client
Igor, I do not have access to edit wiki pages, so please fill it. All the features are supported, except for: - Query API: ScanQuery with filter, - Binary API: Get object in BinaryObject form, BinaryObject building, - Additional Features: Best effort affinity, - Type registration: Enum registration, Type name registration. On 10/15/18 9:17 PM, Igor Sapego wrote: Dmitry, Since Python thin client is now in master, can you please add it to our "Thin clients features" wiki page, so we can track the feature parity of our clients? Or just tell me which features are supported, so I can fill it myself. [1] - https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features Best Regards, Igor
Re: Python thin client
Dmitry, Since Python thin client is now in master, can you please add it to our "Thin clients features" wiki page, so we can track the feature parity of our clients? Or just tell me which features are supported, so I can fill it myself. [1] - https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features Best Regards, Igor On Mon, Oct 15, 2018 at 1:38 PM Igor Sapego wrote: > Hi, > > I've just merged Python thin client to the master and ignite-2.7 > > Also, I like how the python's results improved and now are comparable to > node.js. > > Great job, guys! > > Best Regards, > Igor > > > On Sat, Oct 13, 2018 at 3:53 PM Степан Пильщиков < > pilshchikov@gmail.com> wrote: > >> Hi >> >> Made new run and client definitely become faster >> I think its right direction for improvements >> >> Also rerun java bench with compiler=none option >> >> All results you can get in >> https://issues.apache.org/jira/browse/IGNITE-9824 >> (last comment) >> >> сб, 13 окт. 2018 г. в 4:56, Pavel Petroshenko : >> >> > Hi Stepan, >> > >> > Re your performance questions. >> > >> > Firstly, please pull out the latest sources from the ignite-7782 branch >> as >> > it has some performance optimizations recently implemented. >> > >> > Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and >> it's >> > not fair to compare it to a purely interpreted Python language >> > implementation. To get comparable results, please run your Java >> benchmark >> > with -Djava.compiler=NONE. It turns the dynamic compiler off at >> runtime. I >> > just did this exercise myself and got comparable scores. >> > >> > Please let me know if you still see any significant difference with the >> > flag set. >> > >> > Thanks, >> > P. >> > >> > On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk < >> > dmitry.melnic...@nobitlost.com> wrote: >> > >> >> Igor, >> >> >> >> I fixed my files. >> >> >> >> Dmitry >> >> >> >> On 10/13/18 12:14 AM, Igor Sapego wrote: >> >> > Dmitry, >> >> > >> >> > I've run the tests and it appears that there are a lot of files >> without >> >> > Apache >> >> > licenses in Python client: [1] (See Unapproved licenses list). >> >> > >> >> > Can you either add headers or exclude file from the check, if you can >> >> not >> >> > add one? I.e. to this list: [2] >> >> > >> >> > [1] - >> >> > >> >> >> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt >> >> > [2] - >> https://github.com/apache/ignite/blob/master/parent/pom.xml#L817 >> >> > >> >> > Best Regards, >> >> > Igor >> >> > >> >> > >> >> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego >> wrote: >> >> > >> >> >> Ok, I've filed a ticket [1] for the performance investigation and >> >> targeted >> >> >> it to 2.8. Feel free to assign it to yourself. >> >> >> >> >> >> I'm going to merge the ticket to master and ignite-2.7 branches by >> the >> >> >> end of the week, if no one have any objections. >> >> >> >> >> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850 >> >> >> >> >> >> Best Regards, >> >> >> Igor >> >> >> >> >> >> >> >> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < >> >> >> dmitry.melnic...@nobitlost.com> wrote: >> >> >> >> >> >>> Stepan, >> >> >>> >> >> >>> Thank you for answering my question and for the updated results. >> >> >>> >> >> >>> I have ran the profiler (cProfile) against the benchmark code you >> >> >>> provided and have not found any particular bottleneck. One cycle >> took >> >> me >> >> >>> 434 μs, which is the same order of magnitude as your result. >> >> >>> >> >> >>> I will need more time to analyze the performance of my client and >> >> find a >> >> >>> wa
Re: Python thin client
Hi, I've just merged Python thin client to the master and ignite-2.7 Also, I like how the python's results improved and now are comparable to node.js. Great job, guys! Best Regards, Igor On Sat, Oct 13, 2018 at 3:53 PM Степан Пильщиков wrote: > Hi > > Made new run and client definitely become faster > I think its right direction for improvements > > Also rerun java bench with compiler=none option > > All results you can get in > https://issues.apache.org/jira/browse/IGNITE-9824 > (last comment) > > сб, 13 окт. 2018 г. в 4:56, Pavel Petroshenko : > > > Hi Stepan, > > > > Re your performance questions. > > > > Firstly, please pull out the latest sources from the ignite-7782 branch > as > > it has some performance optimizations recently implemented. > > > > Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and it's > > not fair to compare it to a purely interpreted Python language > > implementation. To get comparable results, please run your Java benchmark > > with -Djava.compiler=NONE. It turns the dynamic compiler off at runtime. > I > > just did this exercise myself and got comparable scores. > > > > Please let me know if you still see any significant difference with the > > flag set. > > > > Thanks, > > P. > > > > On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk < > > dmitry.melnic...@nobitlost.com> wrote: > > > >> Igor, > >> > >> I fixed my files. > >> > >> Dmitry > >> > >> On 10/13/18 12:14 AM, Igor Sapego wrote: > >> > Dmitry, > >> > > >> > I've run the tests and it appears that there are a lot of files > without > >> > Apache > >> > licenses in Python client: [1] (See Unapproved licenses list). > >> > > >> > Can you either add headers or exclude file from the check, if you can > >> not > >> > add one? I.e. to this list: [2] > >> > > >> > [1] - > >> > > >> > https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt > >> > [2] - > https://github.com/apache/ignite/blob/master/parent/pom.xml#L817 > >> > > >> > Best Regards, > >> > Igor > >> > > >> > > >> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego > wrote: > >> > > >> >> Ok, I've filed a ticket [1] for the performance investigation and > >> targeted > >> >> it to 2.8. Feel free to assign it to yourself. > >> >> > >> >> I'm going to merge the ticket to master and ignite-2.7 branches by > the > >> >> end of the week, if no one have any objections. > >> >> > >> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850 > >> >> > >> >> Best Regards, > >> >> Igor > >> >> > >> >> > >> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < > >> >> dmitry.melnic...@nobitlost.com> wrote: > >> >> > >> >>> Stepan, > >> >>> > >> >>> Thank you for answering my question and for the updated results. > >> >>> > >> >>> I have ran the profiler (cProfile) against the benchmark code you > >> >>> provided and have not found any particular bottleneck. One cycle > took > >> me > >> >>> 434 μs, which is the same order of magnitude as your result. > >> >>> > >> >>> I will need more time to analyze the performance of my client and > >> find a > >> >>> way to improve it. But is it a real stopper/blocker for the client > >> >>> itself? If there are no other issues, may we merge and release the > >> >>> client? Plus submit a jira-improvement to further investigate the > >> >>> performance of the Python platform. > >> >>> > >> >>> Dmitry > >> >>> > >> >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote: > >> >>>> Dmitry, > >> >>>> > >> >>>> pip install -e from latest sources > >> >>>> > >> >>>> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, < > >> >>> dmitry.melnic...@nobitlost.com> > >> >>>> wrote: > >> >>>> > >> >>>>> Stepan! > >> >>
Re: Python thin client
Hi Made new run and client definitely become faster I think its right direction for improvements Also rerun java bench with compiler=none option All results you can get in https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) сб, 13 окт. 2018 г. в 4:56, Pavel Petroshenko : > Hi Stepan, > > Re your performance questions. > > Firstly, please pull out the latest sources from the ignite-7782 branch as > it has some performance optimizations recently implemented. > > Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and it's > not fair to compare it to a purely interpreted Python language > implementation. To get comparable results, please run your Java benchmark > with -Djava.compiler=NONE. It turns the dynamic compiler off at runtime. I > just did this exercise myself and got comparable scores. > > Please let me know if you still see any significant difference with the > flag set. > > Thanks, > P. > > On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk < > dmitry.melnic...@nobitlost.com> wrote: > >> Igor, >> >> I fixed my files. >> >> Dmitry >> >> On 10/13/18 12:14 AM, Igor Sapego wrote: >> > Dmitry, >> > >> > I've run the tests and it appears that there are a lot of files without >> > Apache >> > licenses in Python client: [1] (See Unapproved licenses list). >> > >> > Can you either add headers or exclude file from the check, if you can >> not >> > add one? I.e. to this list: [2] >> > >> > [1] - >> > >> https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt >> > [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817 >> > >> > Best Regards, >> > Igor >> > >> > >> > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego wrote: >> > >> >> Ok, I've filed a ticket [1] for the performance investigation and >> targeted >> >> it to 2.8. Feel free to assign it to yourself. >> >> >> >> I'm going to merge the ticket to master and ignite-2.7 branches by the >> >> end of the week, if no one have any objections. >> >> >> >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850 >> >> >> >> Best Regards, >> >> Igor >> >> >> >> >> >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < >> >> dmitry.melnic...@nobitlost.com> wrote: >> >> >> >>> Stepan, >> >>> >> >>> Thank you for answering my question and for the updated results. >> >>> >> >>> I have ran the profiler (cProfile) against the benchmark code you >> >>> provided and have not found any particular bottleneck. One cycle took >> me >> >>> 434 μs, which is the same order of magnitude as your result. >> >>> >> >>> I will need more time to analyze the performance of my client and >> find a >> >>> way to improve it. But is it a real stopper/blocker for the client >> >>> itself? If there are no other issues, may we merge and release the >> >>> client? Plus submit a jira-improvement to further investigate the >> >>> performance of the Python platform. >> >>> >> >>> Dmitry >> >>> >> >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote: >> Dmitry, >> >> pip install -e from latest sources >> >> On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, < >> >>> dmitry.melnic...@nobitlost.com> >> wrote: >> >> > Stepan! >> > >> > Please tell me one thing about how you created the environment for >> your >> > benchmarks, namely: how did you install the Python client. Did you >> just >> > do `pip install pyignite`, or did you pull my working branch from >> the >> > downstream repository and did `pip install -e >> > ignite/modules/platforms/python`? >> > >> > I highly recommend the latter, because PyPI version do not yet >> include >> > the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. >> > These improvements can have a noticeable positive effect on speed. >> > >> > Dmitry >> > >> > On 10/10/18 11:06 PM, Степан Пильщиков wrote: >> >> Dmitry, >> >> >> >> Thanks for review >> >> >> >> Agree with all suggestions. >> >> >> >> Made and run new benches without any redundant operations (prints, >> >>> time >> >> calculation), only with "put" in "while true" cycle (almost) >> >> Case description, environment, results and code in ticket >> >> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) >> >> >> >> Please review new metrics >> >> Because picture is not changed so much, python still have >> performance >> > issues >> >> >> > >> >> >>> >> >>> >> > >> >>
Re: Python thin client
Hi Stepan, Re your performance questions. Firstly, please pull out the latest sources from the ignite-7782 branch as it has some performance optimizations recently implemented. Secondly, Hotspot is a pretty advanced dynamic adaptive compiler and it's not fair to compare it to a purely interpreted Python language implementation. To get comparable results, please run your Java benchmark with -Djava.compiler=NONE. It turns the dynamic compiler off at runtime. I just did this exercise myself and got comparable scores. Please let me know if you still see any significant difference with the flag set. Thanks, P. On Fri, Oct 12, 2018 at 10:02 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > I fixed my files. > > Dmitry > > On 10/13/18 12:14 AM, Igor Sapego wrote: > > Dmitry, > > > > I've run the tests and it appears that there are a lot of files without > > Apache > > licenses in Python client: [1] (See Unapproved licenses list). > > > > Can you either add headers or exclude file from the check, if you can not > > add one? I.e. to this list: [2] > > > > [1] - > > > https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt > > [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817 > > > > Best Regards, > > Igor > > > > > > On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego wrote: > > > >> Ok, I've filed a ticket [1] for the performance investigation and > targeted > >> it to 2.8. Feel free to assign it to yourself. > >> > >> I'm going to merge the ticket to master and ignite-2.7 branches by the > >> end of the week, if no one have any objections. > >> > >> [1] - https://issues.apache.org/jira/browse/IGNITE-9850 > >> > >> Best Regards, > >> Igor > >> > >> > >> On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < > >> dmitry.melnic...@nobitlost.com> wrote: > >> > >>> Stepan, > >>> > >>> Thank you for answering my question and for the updated results. > >>> > >>> I have ran the profiler (cProfile) against the benchmark code you > >>> provided and have not found any particular bottleneck. One cycle took > me > >>> 434 μs, which is the same order of magnitude as your result. > >>> > >>> I will need more time to analyze the performance of my client and find > a > >>> way to improve it. But is it a real stopper/blocker for the client > >>> itself? If there are no other issues, may we merge and release the > >>> client? Plus submit a jira-improvement to further investigate the > >>> performance of the Python platform. > >>> > >>> Dmitry > >>> > >>> On 10/11/18 4:19 PM, Степан Пильщиков wrote: > Dmitry, > > pip install -e from latest sources > > On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, < > >>> dmitry.melnic...@nobitlost.com> > wrote: > > > Stepan! > > > > Please tell me one thing about how you created the environment for > your > > benchmarks, namely: how did you install the Python client. Did you > just > > do `pip install pyignite`, or did you pull my working branch from the > > downstream repository and did `pip install -e > > ignite/modules/platforms/python`? > > > > I highly recommend the latter, because PyPI version do not yet > include > > the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. > > These improvements can have a noticeable positive effect on speed. > > > > Dmitry > > > > On 10/10/18 11:06 PM, Степан Пильщиков wrote: > >> Dmitry, > >> > >> Thanks for review > >> > >> Agree with all suggestions. > >> > >> Made and run new benches without any redundant operations (prints, > >>> time > >> calculation), only with "put" in "while true" cycle (almost) > >> Case description, environment, results and code in ticket > >> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) > >> > >> Please review new metrics > >> Because picture is not changed so much, python still have > performance > > issues > >> > > > > >>> > >>> > > > >
Re: Python thin client
Igor, I fixed my files. Dmitry On 10/13/18 12:14 AM, Igor Sapego wrote: Dmitry, I've run the tests and it appears that there are a lot of files without Apache licenses in Python client: [1] (See Unapproved licenses list). Can you either add headers or exclude file from the check, if you can not add one? I.e. to this list: [2] [1] - https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817 Best Regards, Igor On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego wrote: Ok, I've filed a ticket [1] for the performance investigation and targeted it to 2.8. Feel free to assign it to yourself. I'm going to merge the ticket to master and ignite-2.7 branches by the end of the week, if no one have any objections. [1] - https://issues.apache.org/jira/browse/IGNITE-9850 Best Regards, Igor On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Stepan, Thank you for answering my question and for the updated results. I have ran the profiler (cProfile) against the benchmark code you provided and have not found any particular bottleneck. One cycle took me 434 μs, which is the same order of magnitude as your result. I will need more time to analyze the performance of my client and find a way to improve it. But is it a real stopper/blocker for the client itself? If there are no other issues, may we merge and release the client? Plus submit a jira-improvement to further investigate the performance of the Python platform. Dmitry On 10/11/18 4:19 PM, Степан Пильщиков wrote: Dmitry, pip install -e from latest sources On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, < dmitry.melnic...@nobitlost.com> wrote: Stepan! Please tell me one thing about how you created the environment for your benchmarks, namely: how did you install the Python client. Did you just do `pip install pyignite`, or did you pull my working branch from the downstream repository and did `pip install -e ignite/modules/platforms/python`? I highly recommend the latter, because PyPI version do not yet include the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. These improvements can have a noticeable positive effect on speed. Dmitry On 10/10/18 11:06 PM, Степан Пильщиков wrote: Dmitry, Thanks for review Agree with all suggestions. Made and run new benches without any redundant operations (prints, time calculation), only with "put" in "while true" cycle (almost) Case description, environment, results and code in ticket https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) Please review new metrics Because picture is not changed so much, python still have performance issues
Re: Python thin client
Dmitry, I've run the tests and it appears that there are a lot of files without Apache licenses in Python client: [1] (See Unapproved licenses list). Can you either add headers or exclude file from the check, if you can not add one? I.e. to this list: [2] [1] - https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt [2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817 Best Regards, Igor On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego wrote: > Ok, I've filed a ticket [1] for the performance investigation and targeted > it to 2.8. Feel free to assign it to yourself. > > I'm going to merge the ticket to master and ignite-2.7 branches by the > end of the week, if no one have any objections. > > [1] - https://issues.apache.org/jira/browse/IGNITE-9850 > > Best Regards, > Igor > > > On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < > dmitry.melnic...@nobitlost.com> wrote: > >> Stepan, >> >> Thank you for answering my question and for the updated results. >> >> I have ran the profiler (cProfile) against the benchmark code you >> provided and have not found any particular bottleneck. One cycle took me >> 434 μs, which is the same order of magnitude as your result. >> >> I will need more time to analyze the performance of my client and find a >> way to improve it. But is it a real stopper/blocker for the client >> itself? If there are no other issues, may we merge and release the >> client? Plus submit a jira-improvement to further investigate the >> performance of the Python platform. >> >> Dmitry >> >> On 10/11/18 4:19 PM, Степан Пильщиков wrote: >> > Dmitry, >> > >> > pip install -e from latest sources >> > >> > On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, < >> dmitry.melnic...@nobitlost.com> >> > wrote: >> > >> >> Stepan! >> >> >> >> Please tell me one thing about how you created the environment for your >> >> benchmarks, namely: how did you install the Python client. Did you just >> >> do `pip install pyignite`, or did you pull my working branch from the >> >> downstream repository and did `pip install -e >> >> ignite/modules/platforms/python`? >> >> >> >> I highly recommend the latter, because PyPI version do not yet include >> >> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. >> >> These improvements can have a noticeable positive effect on speed. >> >> >> >> Dmitry >> >> >> >> On 10/10/18 11:06 PM, Степан Пильщиков wrote: >> >>> Dmitry, >> >>> >> >>> Thanks for review >> >>> >> >>> Agree with all suggestions. >> >>> >> >>> Made and run new benches without any redundant operations (prints, >> time >> >>> calculation), only with "put" in "while true" cycle (almost) >> >>> Case description, environment, results and code in ticket >> >>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) >> >>> >> >>> Please review new metrics >> >>> Because picture is not changed so much, python still have performance >> >> issues >> >>> >> >> >> > >> >>
[jira] [Created] (IGNITE-9852) Create TeamCity suite for Python thin client
Igor Sapego created IGNITE-9852: --- Summary: Create TeamCity suite for Python thin client Key: IGNITE-9852 URL: https://issues.apache.org/jira/browse/IGNITE-9852 Project: Ignite Issue Type: Task Components: thin client Reporter: Igor Sapego Fix For: 2.7 Implementation of Python client is almost ready (IGNITE-7782). We need to figure out how to integrate it with TeamCity: 1) New suite for Python thin client is needed along with required environment 2) Suite should be able to run tests as described in the documentation [1] 3) Currently all tests rely on manually started external node. We need to create a set of scripts (bash?) to start/stop nodes locally just like in IGNITE-8463 [1] https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#testing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Python thin client
Ok, I've filed a ticket [1] for the performance investigation and targeted it to 2.8. Feel free to assign it to yourself. I'm going to merge the ticket to master and ignite-2.7 branches by the end of the week, if no one have any objections. [1] - https://issues.apache.org/jira/browse/IGNITE-9850 Best Regards, Igor On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Stepan, > > Thank you for answering my question and for the updated results. > > I have ran the profiler (cProfile) against the benchmark code you > provided and have not found any particular bottleneck. One cycle took me > 434 μs, which is the same order of magnitude as your result. > > I will need more time to analyze the performance of my client and find a > way to improve it. But is it a real stopper/blocker for the client > itself? If there are no other issues, may we merge and release the > client? Plus submit a jira-improvement to further investigate the > performance of the Python platform. > > Dmitry > > On 10/11/18 4:19 PM, Степан Пильщиков wrote: > > Dmitry, > > > > pip install -e from latest sources > > > > On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, < > dmitry.melnic...@nobitlost.com> > > wrote: > > > >> Stepan! > >> > >> Please tell me one thing about how you created the environment for your > >> benchmarks, namely: how did you install the Python client. Did you just > >> do `pip install pyignite`, or did you pull my working branch from the > >> downstream repository and did `pip install -e > >> ignite/modules/platforms/python`? > >> > >> I highly recommend the latter, because PyPI version do not yet include > >> the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. > >> These improvements can have a noticeable positive effect on speed. > >> > >> Dmitry > >> > >> On 10/10/18 11:06 PM, Степан Пильщиков wrote: > >>> Dmitry, > >>> > >>> Thanks for review > >>> > >>> Agree with all suggestions. > >>> > >>> Made and run new benches without any redundant operations (prints, time > >>> calculation), only with "put" in "while true" cycle (almost) > >>> Case description, environment, results and code in ticket > >>> https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) > >>> > >>> Please review new metrics > >>> Because picture is not changed so much, python still have performance > >> issues > >>> > >> > > > >
Re: Python thin client
Stepan, Thank you for answering my question and for the updated results. I have ran the profiler (cProfile) against the benchmark code you provided and have not found any particular bottleneck. One cycle took me 434 μs, which is the same order of magnitude as your result. I will need more time to analyze the performance of my client and find a way to improve it. But is it a real stopper/blocker for the client itself? If there are no other issues, may we merge and release the client? Plus submit a jira-improvement to further investigate the performance of the Python platform. Dmitry On 10/11/18 4:19 PM, Степан Пильщиков wrote: Dmitry, pip install -e from latest sources On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, wrote: Stepan! Please tell me one thing about how you created the environment for your benchmarks, namely: how did you install the Python client. Did you just do `pip install pyignite`, or did you pull my working branch from the downstream repository and did `pip install -e ignite/modules/platforms/python`? I highly recommend the latter, because PyPI version do not yet include the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. These improvements can have a noticeable positive effect on speed. Dmitry On 10/10/18 11:06 PM, Степан Пильщиков wrote: Dmitry, Thanks for review Agree with all suggestions. Made and run new benches without any redundant operations (prints, time calculation), only with "put" in "while true" cycle (almost) Case description, environment, results and code in ticket https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) Please review new metrics Because picture is not changed so much, python still have performance issues
Re: Python thin client
Dmitry, pip install -e from latest sources On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, wrote: > Stepan! > > Please tell me one thing about how you created the environment for your > benchmarks, namely: how did you install the Python client. Did you just > do `pip install pyignite`, or did you pull my working branch from the > downstream repository and did `pip install -e > ignite/modules/platforms/python`? > > I highly recommend the latter, because PyPI version do not yet include > the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. > These improvements can have a noticeable positive effect on speed. > > Dmitry > > On 10/10/18 11:06 PM, Степан Пильщиков wrote: > > Dmitry, > > > > Thanks for review > > > > Agree with all suggestions. > > > > Made and run new benches without any redundant operations (prints, time > > calculation), only with "put" in "while true" cycle (almost) > > Case description, environment, results and code in ticket > > https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) > > > > Please review new metrics > > Because picture is not changed so much, python still have performance > issues > > >
Re: Python thin client
Stepan! Please tell me one thing about how you created the environment for your benchmarks, namely: how did you install the Python client. Did you just do `pip install pyignite`, or did you pull my working branch from the downstream repository and did `pip install -e ignite/modules/platforms/python`? I highly recommend the latter, because PyPI version do not yet include the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. These improvements can have a noticeable positive effect on speed. Dmitry On 10/10/18 11:06 PM, Степан Пильщиков wrote: Dmitry, Thanks for review Agree with all suggestions. Made and run new benches without any redundant operations (prints, time calculation), only with "put" in "while true" cycle (almost) Case description, environment, results and code in ticket https://issues.apache.org/jira/browse/IGNITE-9824 (last comment) Please review new metrics Because picture is not changed so much, python still have performance issues
Re: Python thin client
Dmitry,
Thanks for review
Agree with all suggestions.
Made and run new benches without any redundant operations (prints, time
calculation), only with "put" in "while true" cycle (almost)
Case description, environment, results and code in ticket
https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)
Please review new metrics
Because picture is not changed so much, python still have performance issues
ср, 10 окт. 2018 г. в 12:30, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:
> Stepan!
>
> Can you please update the benchmarks based on my suggestions earlier in
> this thread: use a cleaner time profiling for the loop, remove
> unnecessary operations (string formatting, rounding operations), stick
> to the primitive int value for put operations (agree with Vladimir,
> let's keep it simple). And let me know the results.
>
> On 10/10/18 5:46 PM, Vladimir Ozerov wrote:
> > Hi Dmitry,
> >
> > I agree with your comments on benchmarking code. As more fair
> > alternative we may measure time of loading N elements into the cache, so
> > that it will be necessary to call time() only twice. However, provided
> > that we have real network here, and according to numbers one PUT in
> > Python client requires about 0.3 msec, I hardly can imagine that call to
> > time() or round() may dominate in that response time. But I said, we can
> > easily rule that out by slight benchmark rewrite.
> >
> > As far as serialization, I would prefer to keep primitive objects at the
> > moment, because AFAIK the purpose of the benchmark was to assess
> > underlying infrastructure, looking for some obvious performance issues.
> > Every platform have sockets which use more or less the same API of
> > underlying OS. To the contrast, performance of serialization mechanics
> > may differ widely between platforms depending on implementation. E.g. in
> > Java we spend a lot of time on fine-grained tuning, applied a lot of
> > speculative optimizations. Add to this JIT nature of Java, and you will
> > hardly ever beat Java serialization engine from any interpreted
> > language. So primitive data types make good sense to me.
> >
> > At this point our goal is not make Python equally fast with other
> > platforms, but rather to understand why is it slower than others. Ignite
> > is a product which brings speed to end users. If they do no have speed,
> > they will not use Ignite. So performance questions are always of great
> > importance for us.
> >
> > Vladimir.
> >
> > On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk
> > mailto:dmitry.melnic...@nobitlost.com>>
>
> > wrote:
> >
> > Hi, Stepan!
> >
> > I looked at the benchmark code and the overall methodics, discussed
> it
> > with fellow programmers, and came up with basically two remarks.
> >
> > First of all, I think, the key for the good (or, at least,
> > unobjectionable) measurement is to isolate the object being measured
> > from the influence of the measurement tool. The usual pattern we use
> in
> > Python looks like:
> >
> > ```
> > from time import time
> >
> >
> > bench_start = time()
> >
> > for _ in range(number_of_tests):
> > do_test()
> >
> > bench_end = time()
> >
> > print('Performance is {} tests per second'.format(
> > number_of_tests / (bench_end - bench_start)
> > ))
> > ```
> >
> > I think, you got the idea: the measurement consists almost solely of
> > the
> > time taken by our subject function `do_test`. As few other code as
> > possible influence the result.
> >
> > Now, let's take a look at your code:
> >
> > https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> >
> > Ideally, the `while` loop should include only `cache.put(last_key,
> > some_precomputed_value)`. But instead it also includes:
> >
> > - a series of the `time()` calls, which could be mostly excluded from
> > the measured time, if the measurement done right; each call is
> probably
> > addresses the HPET device, or network time, or both,
> >
> > - some floating-point calculations, including `round()`, which is
> > hardly
> > necessary,
> >
> > - formatting and output of the intermediate result.
> >
> > I suppose the measurement influence can be quite significant here,
> but
> > it is at least more or less constant for each test.
> >
> > But if we look at the other benchmarks:
> >
> > https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
> > [Node.js]
> > https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80
> > [PHP]
> > https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
> > [Java]
> >
> > The extra code that influence the measurement is not equivalent
> across
> > all platforms. For example, PHP's `time()` is most probably lighter
> > than
> > Python `time()`, since it do not give out milliseconds and may
> address
> > RTC, not
Re: Python thin client
Stepan!
Can you please update the benchmarks based on my suggestions earlier in
this thread: use a cleaner time profiling for the loop, remove
unnecessary operations (string formatting, rounding operations), stick
to the primitive int value for put operations (agree with Vladimir,
let's keep it simple). And let me know the results.
On 10/10/18 5:46 PM, Vladimir Ozerov wrote:
Hi Dmitry,
I agree with your comments on benchmarking code. As more fair
alternative we may measure time of loading N elements into the cache, so
that it will be necessary to call time() only twice. However, provided
that we have real network here, and according to numbers one PUT in
Python client requires about 0.3 msec, I hardly can imagine that call to
time() or round() may dominate in that response time. But I said, we can
easily rule that out by slight benchmark rewrite.
As far as serialization, I would prefer to keep primitive objects at the
moment, because AFAIK the purpose of the benchmark was to assess
underlying infrastructure, looking for some obvious performance issues.
Every platform have sockets which use more or less the same API of
underlying OS. To the contrast, performance of serialization mechanics
may differ widely between platforms depending on implementation. E.g. in
Java we spend a lot of time on fine-grained tuning, applied a lot of
speculative optimizations. Add to this JIT nature of Java, and you will
hardly ever beat Java serialization engine from any interpreted
language. So primitive data types make good sense to me.
At this point our goal is not make Python equally fast with other
platforms, but rather to understand why is it slower than others. Ignite
is a product which brings speed to end users. If they do no have speed,
they will not use Ignite. So performance questions are always of great
importance for us.
Vladimir.
On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk
mailto:dmitry.melnic...@nobitlost.com>>
wrote:
Hi, Stepan!
I looked at the benchmark code and the overall methodics, discussed it
with fellow programmers, and came up with basically two remarks.
First of all, I think, the key for the good (or, at least,
unobjectionable) measurement is to isolate the object being measured
from the influence of the measurement tool. The usual pattern we use in
Python looks like:
```
from time import time
bench_start = time()
for _ in range(number_of_tests):
do_test()
bench_end = time()
print('Performance is {} tests per second'.format(
number_of_tests / (bench_end - bench_start)
))
```
I think, you got the idea: the measurement consists almost solely of
the
time taken by our subject function `do_test`. As few other code as
possible influence the result.
Now, let's take a look at your code:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Ideally, the `while` loop should include only `cache.put(last_key,
some_precomputed_value)`. But instead it also includes:
- a series of the `time()` calls, which could be mostly excluded from
the measured time, if the measurement done right; each call is probably
addresses the HPET device, or network time, or both,
- some floating-point calculations, including `round()`, which is
hardly
necessary,
- formatting and output of the intermediate result.
I suppose the measurement influence can be quite significant here, but
it is at least more or less constant for each test.
But if we look at the other benchmarks:
https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
[Node.js]
https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80
[PHP]
https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
[Java]
The extra code that influence the measurement is not equivalent across
all platforms. For example, PHP's `time()` is most probably lighter
than
Python `time()`, since it do not give out milliseconds and may address
RTC, not HPET. So the platform-to-platform comparison in your benchmark
does not look completely fair to me.
The second remark concerns not the measurement procedure, but the
object
being measured.
The only client operation being used is OP_CACHE_PUT with a payload
of a
primitive type. (BTW the type is `Long` in case of the Python client;
what about the other clients? Is it `Int`?) I afraid that such an
object, even being properly isolated from the measurement tool, would
show mostly the throughput of the underlying platform's sockets
implementation, not the performance of the client's code. To show the
potential of the thin client itself, more varied and fancy tasks
needed,
i. e. serialization/deserialization of the Complex objects.
But it depends on the goal of the benchmarking. If showing off the raw
Re: Python thin client
Hi Dmitry,
I agree with your comments on benchmarking code. As more fair alternative
we may measure time of loading N elements into the cache, so that it will
be necessary to call time() only twice. However, provided that we have real
network here, and according to numbers one PUT in Python client requires
about 0.3 msec, I hardly can imagine that call to time() or round() may
dominate in that response time. But I said, we can easily rule that out by
slight benchmark rewrite.
As far as serialization, I would prefer to keep primitive objects at the
moment, because AFAIK the purpose of the benchmark was to assess underlying
infrastructure, looking for some obvious performance issues. Every platform
have sockets which use more or less the same API of underlying OS. To the
contrast, performance of serialization mechanics may differ widely between
platforms depending on implementation. E.g. in Java we spend a lot of time
on fine-grained tuning, applied a lot of speculative optimizations. Add to
this JIT nature of Java, and you will hardly ever beat Java serialization
engine from any interpreted language. So primitive data types make good
sense to me.
At this point our goal is not make Python equally fast with other
platforms, but rather to understand why is it slower than others. Ignite is
a product which brings speed to end users. If they do no have speed, they
will not use Ignite. So performance questions are always of great
importance for us.
Vladimir.
On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:
> Hi, Stepan!
>
> I looked at the benchmark code and the overall methodics, discussed it
> with fellow programmers, and came up with basically two remarks.
>
> First of all, I think, the key for the good (or, at least,
> unobjectionable) measurement is to isolate the object being measured
> from the influence of the measurement tool. The usual pattern we use in
> Python looks like:
>
> ```
> from time import time
>
>
> bench_start = time()
>
> for _ in range(number_of_tests):
> do_test()
>
> bench_end = time()
>
> print('Performance is {} tests per second'.format(
> number_of_tests / (bench_end - bench_start)
> ))
> ```
>
> I think, you got the idea: the measurement consists almost solely of the
> time taken by our subject function `do_test`. As few other code as
> possible influence the result.
>
> Now, let's take a look at your code:
>
> https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
>
> Ideally, the `while` loop should include only `cache.put(last_key,
> some_precomputed_value)`. But instead it also includes:
>
> - a series of the `time()` calls, which could be mostly excluded from
> the measured time, if the measurement done right; each call is probably
> addresses the HPET device, or network time, or both,
>
> - some floating-point calculations, including `round()`, which is hardly
> necessary,
>
> - formatting and output of the intermediate result.
>
> I suppose the measurement influence can be quite significant here, but
> it is at least more or less constant for each test.
>
> But if we look at the other benchmarks:
>
> https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
> [Node.js]
> https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
> https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
> [Java]
>
> The extra code that influence the measurement is not equivalent across
> all platforms. For example, PHP's `time()` is most probably lighter than
> Python `time()`, since it do not give out milliseconds and may address
> RTC, not HPET. So the platform-to-platform comparison in your benchmark
> does not look completely fair to me.
>
> The second remark concerns not the measurement procedure, but the object
> being measured.
>
> The only client operation being used is OP_CACHE_PUT with a payload of a
> primitive type. (BTW the type is `Long` in case of the Python client;
> what about the other clients? Is it `Int`?) I afraid that such an
> object, even being properly isolated from the measurement tool, would
> show mostly the throughput of the underlying platform's sockets
> implementation, not the performance of the client's code. To show the
> potential of the thin client itself, more varied and fancy tasks needed,
> i. e. serialization/deserialization of the Complex objects.
>
> But it depends on the goal of the benchmarking. If showing off the raw
> platform agility was intended, than this objection is removed.
>
> Dmitry
>
> On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
> > Hi, all
> >
> > Tried to compare new thin client performance and made similar benchmarks
> for
> > each one
> > And result not so good for python
> >
> > Ticket with results and bench code:
> > https://issues.apache.org/jira/browse/IGNITE-9824
> >
> > Py code src:
> > https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> >
> > Dmitry, please review results and bench code, maybe
Re: Python thin client
Hi, Stepan!
I looked at the benchmark code and the overall methodics, discussed it
with fellow programmers, and came up with basically two remarks.
First of all, I think, the key for the good (or, at least,
unobjectionable) measurement is to isolate the object being measured
from the influence of the measurement tool. The usual pattern we use in
Python looks like:
```
from time import time
bench_start = time()
for _ in range(number_of_tests):
do_test()
bench_end = time()
print('Performance is {} tests per second'.format(
number_of_tests / (bench_end - bench_start)
))
```
I think, you got the idea: the measurement consists almost solely of the
time taken by our subject function `do_test`. As few other code as
possible influence the result.
Now, let's take a look at your code:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Ideally, the `while` loop should include only `cache.put(last_key,
some_precomputed_value)`. But instead it also includes:
- a series of the `time()` calls, which could be mostly excluded from
the measured time, if the measurement done right; each call is probably
addresses the HPET device, or network time, or both,
- some floating-point calculations, including `round()`, which is hardly
necessary,
- formatting and output of the intermediate result.
I suppose the measurement influence can be quite significant here, but
it is at least more or less constant for each test.
But if we look at the other benchmarks:
https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
[Node.js]
https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce [Java]
The extra code that influence the measurement is not equivalent across
all platforms. For example, PHP's `time()` is most probably lighter than
Python `time()`, since it do not give out milliseconds and may address
RTC, not HPET. So the platform-to-platform comparison in your benchmark
does not look completely fair to me.
The second remark concerns not the measurement procedure, but the object
being measured.
The only client operation being used is OP_CACHE_PUT with a payload of a
primitive type. (BTW the type is `Long` in case of the Python client;
what about the other clients? Is it `Int`?) I afraid that such an
object, even being properly isolated from the measurement tool, would
show mostly the throughput of the underlying platform's sockets
implementation, not the performance of the client's code. To show the
potential of the thin client itself, more varied and fancy tasks needed,
i. e. serialization/deserialization of the Complex objects.
But it depends on the goal of the benchmarking. If showing off the raw
platform agility was intended, than this objection is removed.
Dmitry
On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
Hi, all
Tried to compare new thin client performance and made similar benchmarks for
each one
And result not so good for python
Ticket with results and bench code:
https://issues.apache.org/jira/browse/IGNITE-9824
Py code src:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Dmitry, please review results and bench code, maybe somthing wrong or it's
expected numbers?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Python thin client
Hi, all Tried to compare new thin client performance and made similar benchmarks for each one And result not so good for python Ticket with results and bench code: https://issues.apache.org/jira/browse/IGNITE-9824 Py code src: https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb Dmitry, please review results and bench code, maybe somthing wrong or it's expected numbers? -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Python thin client installation instructions
Yes, I will mention the prerequisites. -Prachi > On Sep 21, 2018, at 5:18 PM, Denis Magda wrote: > > I would add a disclaimer or a prerequisite step. That what other companies > do if a user needs to do some basic installation steps. At least mention it. > > -- > Denis > >> On Fri, Sep 21, 2018 at 3:04 PM Prachi Garg wrote: >> >> Hi Dmitry, >> >> Thank you for taking the time to explain me everything in such detail :) >> >> I am trying to do this because I have to document. In general, I am >> assuming that a Python thin client user would already have Python installed >> and be using it. So, I would not suggest adding any disclaimers regarding >> Python installation. >> >> The example works with python3 command :) >> >> >> -P >> >> >> >> On Thu, Sep 20, 2018 at 8:31 PM, Dmitry Melnichuk < >> dmitry.melnic...@nobitlost.com> wrote: >> >>> Prachi, >>> >>> I feel your struggle. It is easier for end user to perceive Python 2 and >>> Python 3 as different languages, not as versions of one language. They >>> usually installed alongside each other; their updates are handled >>> separately. On most systems they have their respective shell commands: >>> `python2` and `python3`. >>> >>> Shell command `python` can be viewed as an alias of either `python2` or >>> `python3`. I use an Arch Linux derivative, where `python` is `python3`. >>> Most other GNU/Linux OSes use `python` as an alias of `python2`. I am not >>> sure about MacOS. On Windows the latest Python distribution installed >>> overrule PATH environment variables, so it impossible to predict the >>> “default” Python version (2 or 3). >>> >>> Luckily, virtualenv was introduced to leverage all these issues. It is >>> able to handle multiple isolated Python environments, where the `python` >>> command is set upon the creation of the environment, while the >>> environment-specific package dependencies are handled transparently with >>> pip. >>> >>> But the use of pyignite should not be limited to virtualenv. There are >>> many cases when the use of virtualenv is discouraged or even impossible. >>> For example, when deploying Python app into an OS-level container or >>> similar isolating environment, virtualenv would be just a useless >> overhead. >>> There are also non-standard Python distributions (used mostly on Windows) >>> that do not support virtualenv. >>> >>> I am sorry, that users who are not proficient in Python can have so many >>> problems with following my documentation. But still it seems obvious for >>> me, that all the details of organizing user's own Python environment are >>> out of pyignite documentation's scope. The only thing I can suggest to >>> improve my documentation in this regard is putting a big bold foreword >> like >>> this: >>> >>> It is assumed in this document that you know how to install >>> and use Python 3 on your system. Please consult your OS manual pages >>> or documentations of your specific Python 3 distribution regarding >>> the details of organizing your Python 3 environment. The use of >>> virualenv for development with pyignite is highly recommended. >>> >>> But, frankly, I have not seen such disclaimers in the wild and not sure >> if >>> it would be useful. It is very vague and do not cover any of the >> potential >>> pitfalls. >>> >>> I am sorry for giving such a lengthy explanation here, though I've been >>> asked a very specific question. I understand you may not have time to >>> invest in learning virtualenv. If so, you did everything right, just use >>> `python3` command for launching examples: >>> >>> ``` >>> $ python3 get_and_put.py >>> ``` >>> >>>> On 9/21/18 10:34 AM, Prachi Garg wrote: >>>> >>>> Hi Dmitry, >>>> >>>> Sorry, I am not familiar with Python. >>>> >>>> So there are more issues... >>>> >>>> 1. The version on my mac remains 2.7.10 even though I tried to link to >>>> the new version. >>>> >>>> ~$ python --version >>>> Python 2.7.10 >>>> >>>> ~$ brew unlink python && brew link --overwrite python3 >>>> Unlinking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks >>>> removed >>>
Re: Python thin client installation instructions
I would add a disclaimer or a prerequisite step. That what other companies do if a user needs to do some basic installation steps. At least mention it. -- Denis On Fri, Sep 21, 2018 at 3:04 PM Prachi Garg wrote: > Hi Dmitry, > > Thank you for taking the time to explain me everything in such detail :) > > I am trying to do this because I have to document. In general, I am > assuming that a Python thin client user would already have Python installed > and be using it. So, I would not suggest adding any disclaimers regarding > Python installation. > > The example works with python3 command :) > > > -P > > > > On Thu, Sep 20, 2018 at 8:31 PM, Dmitry Melnichuk < > dmitry.melnic...@nobitlost.com> wrote: > > > Prachi, > > > > I feel your struggle. It is easier for end user to perceive Python 2 and > > Python 3 as different languages, not as versions of one language. They > > usually installed alongside each other; their updates are handled > > separately. On most systems they have their respective shell commands: > > `python2` and `python3`. > > > > Shell command `python` can be viewed as an alias of either `python2` or > > `python3`. I use an Arch Linux derivative, where `python` is `python3`. > > Most other GNU/Linux OSes use `python` as an alias of `python2`. I am not > > sure about MacOS. On Windows the latest Python distribution installed > > overrule PATH environment variables, so it impossible to predict the > > “default” Python version (2 or 3). > > > > Luckily, virtualenv was introduced to leverage all these issues. It is > > able to handle multiple isolated Python environments, where the `python` > > command is set upon the creation of the environment, while the > > environment-specific package dependencies are handled transparently with > > pip. > > > > But the use of pyignite should not be limited to virtualenv. There are > > many cases when the use of virtualenv is discouraged or even impossible. > > For example, when deploying Python app into an OS-level container or > > similar isolating environment, virtualenv would be just a useless > overhead. > > There are also non-standard Python distributions (used mostly on Windows) > > that do not support virtualenv. > > > > I am sorry, that users who are not proficient in Python can have so many > > problems with following my documentation. But still it seems obvious for > > me, that all the details of organizing user's own Python environment are > > out of pyignite documentation's scope. The only thing I can suggest to > > improve my documentation in this regard is putting a big bold foreword > like > > this: > > > > It is assumed in this document that you know how to install > > and use Python 3 on your system. Please consult your OS manual pages > > or documentations of your specific Python 3 distribution regarding > > the details of organizing your Python 3 environment. The use of > > virualenv for development with pyignite is highly recommended. > > > > But, frankly, I have not seen such disclaimers in the wild and not sure > if > > it would be useful. It is very vague and do not cover any of the > potential > > pitfalls. > > > > I am sorry for giving such a lengthy explanation here, though I've been > > asked a very specific question. I understand you may not have time to > > invest in learning virtualenv. If so, you did everything right, just use > > `python3` command for launching examples: > > > > ``` > > $ python3 get_and_put.py > > ``` > > > > On 9/21/18 10:34 AM, Prachi Garg wrote: > > > >> Hi Dmitry, > >> > >> Sorry, I am not familiar with Python. > >> > >> So there are more issues... > >> > >> 1. The version on my mac remains 2.7.10 even though I tried to link to > >> the new version. > >> > >> ~$ python --version > >> Python 2.7.10 > >> > >> ~$ brew unlink python && brew link --overwrite python3 > >> Unlinking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks > >> removed > >> Linking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks > >> created > >> > >> ~$ python --version > >> Python 2.7.10 > >> > >> 2. Then I tried to update /pip/, uninstall and re-install /pyignite/ > >> > >> ~$ pip install -U pip > >> -bash: /Library/Frameworks/Python.framework/Versions/3.7/bin/pip: No > >> such file or directory > >> ~$ pi
Re: Python thin client installation instructions
Hi Dmitry, Thank you for taking the time to explain me everything in such detail :) I am trying to do this because I have to document. In general, I am assuming that a Python thin client user would already have Python installed and be using it. So, I would not suggest adding any disclaimers regarding Python installation. The example works with python3 command :) -P On Thu, Sep 20, 2018 at 8:31 PM, Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Prachi, > > I feel your struggle. It is easier for end user to perceive Python 2 and > Python 3 as different languages, not as versions of one language. They > usually installed alongside each other; their updates are handled > separately. On most systems they have their respective shell commands: > `python2` and `python3`. > > Shell command `python` can be viewed as an alias of either `python2` or > `python3`. I use an Arch Linux derivative, where `python` is `python3`. > Most other GNU/Linux OSes use `python` as an alias of `python2`. I am not > sure about MacOS. On Windows the latest Python distribution installed > overrule PATH environment variables, so it impossible to predict the > “default” Python version (2 or 3). > > Luckily, virtualenv was introduced to leverage all these issues. It is > able to handle multiple isolated Python environments, where the `python` > command is set upon the creation of the environment, while the > environment-specific package dependencies are handled transparently with > pip. > > But the use of pyignite should not be limited to virtualenv. There are > many cases when the use of virtualenv is discouraged or even impossible. > For example, when deploying Python app into an OS-level container or > similar isolating environment, virtualenv would be just a useless overhead. > There are also non-standard Python distributions (used mostly on Windows) > that do not support virtualenv. > > I am sorry, that users who are not proficient in Python can have so many > problems with following my documentation. But still it seems obvious for > me, that all the details of organizing user's own Python environment are > out of pyignite documentation's scope. The only thing I can suggest to > improve my documentation in this regard is putting a big bold foreword like > this: > > It is assumed in this document that you know how to install > and use Python 3 on your system. Please consult your OS manual pages > or documentations of your specific Python 3 distribution regarding > the details of organizing your Python 3 environment. The use of > virualenv for development with pyignite is highly recommended. > > But, frankly, I have not seen such disclaimers in the wild and not sure if > it would be useful. It is very vague and do not cover any of the potential > pitfalls. > > I am sorry for giving such a lengthy explanation here, though I've been > asked a very specific question. I understand you may not have time to > invest in learning virtualenv. If so, you did everything right, just use > `python3` command for launching examples: > > ``` > $ python3 get_and_put.py > ``` > > On 9/21/18 10:34 AM, Prachi Garg wrote: > >> Hi Dmitry, >> >> Sorry, I am not familiar with Python. >> >> So there are more issues... >> >> 1. The version on my mac remains 2.7.10 even though I tried to link to >> the new version. >> >> ~$ python --version >> Python 2.7.10 >> >> ~$ brew unlink python && brew link --overwrite python3 >> Unlinking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks >> removed >> Linking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks >> created >> >> ~$ python --version >> Python 2.7.10 >> >> 2. Then I tried to update /pip/, uninstall and re-install /pyignite/ >> >> ~$ pip install -U pip >> -bash: /Library/Frameworks/Python.framework/Versions/3.7/bin/pip: No >> such file or directory >> ~$ pip3 install -U pip >> Requirement already up-to-date: pip in /Library/Frameworks/Python.fra >> mework/Versions/3.7/lib/python3.7/site-packages (18.0) >> >> ~$ pip3 uninstall pyignite >> Uninstalling pyignite-0.3.0: >>Would remove: >> /Library/Frameworks/Python.framework/Versions/3.7/lib/pytho >> n3.7/site-packages/pyignite-0.3.0.dist-info/* >> /Library/Frameworks/Python.framework/Versions/3.7/lib/pytho >> n3.7/site-packages/pyignite/* >> Proceed (y/n)? y >>Successfully uninstalled pyignite-0.3.0 >> >> ~$ pip3 install pyignite >> Collecting pyignite >>Using cached https://files.pythonhosted.org >> /packages/f1/0f/5669cd63fb37fa20
Re: Python thin client installation instructions
ou should either undo the previous
install ("pip uninstall "), use "--update" argument
("pip install --update "), or even better − use
virtualenv to create a disposable Python environment for every
experiment. I provided a link to virtualenv manual in the README
file and in docs, in the 'Basics → Testing' section.
> Also, the installation instruction you have provided are for unix users.
> What are the installation instructions for Windows users?
I tried to be OS-agnostic in the docs. Please tell me, what part of
my instructions does not work on Windows.
On 9/20/18 7:32 AM, Prachi Garg wrote:
Hi Dmitry,
I tried to follow the instructions for the Python thin client
installation [1].
~$ pip install pyignite
Requirement already satisfied: pyignite in
./Downloads/ignite-python/modules/platforms/python (0.3.1)
Requirement already satisfied: typing==3.6.4 in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(from pyignite) (3.6.4)
Requirement already satisfied: attrs==18.1.0 in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(from pyignite) (18.1.0)
But when I try to run an example, I get an error.
~/Downloads/ignite-python/modules/platforms/python/examples$
python get_and_put.py
Traceback (most recent call last):
File "get_and_put.py", line 16, in
from pyignite import Client
ImportError: No module named pyignite
What else need to be done? I am documenting the instructions on
readme.io <http://readme.io> <http://readme.io>, but I need to
be able to run a few examples first.
Also, the installation instruction you have provided are for
unix users. What are the installation instructions for Windows
users?
[1]
https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#installation
<https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#installation>
Re: Python thin client installation instructions
Hi Dmitry,
Sorry, I am not familiar with Python.
So there are more issues...
1. The version on my mac remains 2.7.10 even though I tried to link to the
new version.
~$ python --version
Python 2.7.10
~$ brew unlink python && brew link --overwrite python3
Unlinking /usr/local/Cellar/python/3.7.0... 25 symlinks removed
Linking /usr/local/Cellar/python/3.7.0... 25 symlinks created
~$ python --version
Python 2.7.10
2. Then I tried to update *pip*, uninstall and re-install *pyignite*
~$ pip install -U pip
-bash: /Library/Frameworks/Python.framework/Versions/3.7/bin/pip: No such
file or directory
~$ pip3 install -U pip
Requirement already up-to-date: pip in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(18.0)
~$ pip3 uninstall pyignite
Uninstalling pyignite-0.3.0:
Would remove:
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyignite-0.3.0.dist-info/*
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyignite/*
Proceed (y/n)? y
Successfully uninstalled pyignite-0.3.0
~$ pip3 install pyignite
Collecting pyignite
Using cached
https://files.pythonhosted.org/packages/f1/0f/5669cd63fb37fa2025110f61598450567d04a72c8cf5b76bb0ca20c21734/pyignite-0.3.0-py3-none-any.whl
Requirement already satisfied: attrs==18.1.0 in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(from pyignite) (18.1.0)
Requirement already satisfied: typing==3.6.4 in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(from pyignite) (3.6.4)
Installing collected packages: pyignite
Successfully installed pyignite-0.3.0
How can I fix this?
On Wed, Sep 19, 2018 at 6:43 PM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:
> Prachi,
>
> This line in your message
>
> > Requirement already satisfied: pyignite in
> > ./Downloads/ignite-python/modules/platforms/python (0.3.1)
>
> looks like you already did an installation of pyignite in this environment
> before (maybe with "pip install -e ") from
> 'Downloads' folder, then delete or move downloaded copy, and than tried to
> install pyignite again, this time from PyPI ("pip install ").
>
> pip does not work this way. You should either undo the previous install
> ("pip uninstall "), use "--update" argument ("pip install
> --update "), or even better − use virtualenv to create a
> disposable Python environment for every experiment. I provided a link to
> virtualenv manual in the README file and in docs, in the 'Basics → Testing'
> section.
>
> > Also, the installation instruction you have provided are for unix users.
> > What are the installation instructions for Windows users?
>
> I tried to be OS-agnostic in the docs. Please tell me, what part of my
> instructions does not work on Windows.
>
> On 9/20/18 7:32 AM, Prachi Garg wrote:
>
>> Hi Dmitry,
>>
>> I tried to follow the instructions for the Python thin client
>> installation [1].
>>
>> ~$ pip install pyignite
>> Requirement already satisfied: pyignite in
>> ./Downloads/ignite-python/modules/platforms/python
>> (0.3.1)
>> Requirement already satisfied: typing==3.6.4 in
>> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
>> (from pyignite) (3.6.4)
>> Requirement already satisfied: attrs==18.1.0 in
>> /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
>> (from pyignite) (18.1.0)
>>
>> But when I try to run an example, I get an error.
>>
>> ~/Downloads/ignite-python/modules/platforms/python/examples$ python
>> get_and_put.py
>> Traceback (most recent call last):
>>File "get_and_put.py", line 16, in
>> from pyignite import Client
>> ImportError: No module named pyignite
>>
>>
>> What else need to be done? I am documenting the instructions on readme.io
>> <http://readme.io>, but I need to be able to run a few examples first.
>>
>> Also, the installation instruction you have provided are for unix users.
>> What are the installation instructions for Windows users?
>>
>> [1] https://apache-ignite-binary-protocol-client.readthedocs.io/
>> en/latest/readme.html#installation
>>
>
>
Re: Python thin client installation instructions
Prachi,
This line in your message
> Requirement already satisfied: pyignite in
> ./Downloads/ignite-python/modules/platforms/python (0.3.1)
looks like you already did an installation of pyignite in this
environment before (maybe with "pip install -e ")
from 'Downloads' folder, then delete or move downloaded copy, and than
tried to install pyignite again, this time from PyPI ("pip install
").
pip does not work this way. You should either undo the previous install
("pip uninstall "), use "--update" argument ("pip install
--update "), or even better − use virtualenv to create a
disposable Python environment for every experiment. I provided a link to
virtualenv manual in the README file and in docs, in the 'Basics →
Testing' section.
> Also, the installation instruction you have provided are for unix users.
> What are the installation instructions for Windows users?
I tried to be OS-agnostic in the docs. Please tell me, what part of my
instructions does not work on Windows.
On 9/20/18 7:32 AM, Prachi Garg wrote:
Hi Dmitry,
I tried to follow the instructions for the Python thin client
installation [1].
~$ pip install pyignite
Requirement already satisfied: pyignite in
./Downloads/ignite-python/modules/platforms/python (0.3.1)
Requirement already satisfied: typing==3.6.4 in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(from pyignite) (3.6.4)
Requirement already satisfied: attrs==18.1.0 in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(from pyignite) (18.1.0)
But when I try to run an example, I get an error.
~/Downloads/ignite-python/modules/platforms/python/examples$ python
get_and_put.py
Traceback (most recent call last):
File "get_and_put.py", line 16, in
from pyignite import Client
ImportError: No module named pyignite
What else need to be done? I am documenting the instructions on
readme.io <http://readme.io>, but I need to be able to run a few
examples first.
Also, the installation instruction you have provided are for unix users.
What are the installation instructions for Windows users?
[1]
https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#installation
Python thin client installation instructions
Hi Dmitry, I tried to follow the instructions for the Python thin client installation [1]. ~$ pip install pyignite Requirement already satisfied: pyignite in ./Downloads/ignite-python/modules/platforms/python (0.3.1) Requirement already satisfied: typing==3.6.4 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from pyignite) (3.6.4) Requirement already satisfied: attrs==18.1.0 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from pyignite) (18.1.0) But when I try to run an example, I get an error. ~/Downloads/ignite-python/modules/platforms/python/examples$ python get_and_put.py Traceback (most recent call last): File "get_and_put.py", line 16, in from pyignite import Client ImportError: No module named pyignite What else need to be done? I am documenting the instructions on readme.io, but I need to be able to run a few examples first. Also, the installation instruction you have provided are for unix users. What are the installation instructions for Windows users? [1] https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#installation
Re: Python thin client
Hi Dmitry, Not sure if you are aware but all documentation for the thin clients will be moved to readme.io. Here is the ticket for Python thin clients - https://issues.apache.org/jira/browse/IGNITE-9522 I am already working on it. So if you make any changes to the docs in your repo, please let me know so that I don't miss bringing them over to readme.io -Prachi On Tue, Sep 18, 2018 at 4:08 AM, Igor Sapego wrote: > Great job. > > Best Regards, > Igor > > > On Tue, Sep 18, 2018 at 11:35 AM Dmitry Melnichuk < > dmitry.melnic...@nobitlost.com> wrote: > > > Igor, > > > > All examples are in 'ignite/modules/platforms/python/examples'. > > > > I put examples in separate Python files mostly to be able to > > automatically confirm their operability. All the lengthy explanations > > and cross-references are in the main documentation: > > > > > > https://apache-ignite-binary-protocol-client.readthedocs. > io/en/latest/examples.html > > > > To make it easier for end users to navigate in pyignite directories, I > > added a small README file to the 'examples' directory with a short > > description and a link to the docs: > > > > > > https://github.com/nobitlost/ignite/blob/ignite-7782/ > modules/platforms/python/examples/readme.md > > > > If you think this README is lacking something regardless of the main > > pyignite docs, please let me know. > > > > On 9/17/18 8:59 PM, Igor Sapego wrote: > > > Dmitry, > > > > > > Sorry, I was not clear enough. What I mean is that Ignite distributed > > > by both source and binary releases. Binary releases contain platforms > > > code, which is needed to write your own application in the language, > > > but does not contain developer stuff, such as tests, documentation > > > generating scripts, etc. > > > > > > Of course, it is more common to use package managers when > > > developing your application, and of course, we are going to support > > > this approach. But still, we provide a way for a user to get any client > > > without any package manager. > > > > > > I like the idea of supplying whl package in a binary release, though it > > > looks like it's going to take more effort to implement it. But I > believe > > > except for the whl package, we will need to supply examples and > > > instructions, how user can run them. > > > > > > What do you think? > > > > > > Best Regards, > > > Igor > > > > > > > > > On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk < > > > dmitry.melnic...@nobitlost.com> wrote: > > > > > >> Igor, > > >> > > >> I am in doubt here because I am not fully comprehend the meaning of > > >> "binary release". But if it is somehow related to the "distribution" > > >> thing, I would dare to suggest the following options: > > >> > > >> 1. Copy nothing. Just do > > >> > > >> ``` > > >> $ python setup.py bdist_wheel > > >> $ twine upload dist/* > > >> ``` > > >> > > >> during the build process (or separately) and let PyPI handle the > > >> distribution. > > >> > > >> This is the most natural and user-friendly way of distributing Python > > >> packages. End user might only do > > >> > > >> ``` > > >> $ pip install pyignite > > >> ``` > > >> > > >> as it is suggested by my readme file. > > >> > > >> 2. Supply the wheel package. It is the file 'pyignite-*.whl' from > 'dist' > > >> directory, that should appear there as the "python setup.py > bdist_wheel" > > >> command finishes. Actually, it can be combined with the first option. > > >> > > >> The wheel can then be installed by the end user: > > >> > > >> ``` > > >> $ pip install pyignite-*.whl > > >> ``` > > >> > > >> 3. Supply the following files: > > >> > > >> ignite/modules/platforms/python/pyignite/** > > >> ignite/modules/platforms/python/requirements/** > > >> ignite/modules/platforms/python/LICENSE > > >> ignite/modules/platforms/python/README.md > > >> ignite/modules/platforms/python/setup.py > > >> > > >> (** stands for "all the files and sub-folders recursively, starting > from > > >> this folder".) > > >> > > >> It is not uncommon or wrong to distribute Python programs as text > > >> without prior packaging, but, in my personal experience, this is a way > > >> more suitable for one-time scripts or proprietary programs. For > example, > > >> most of Python web apps relies on git for deployment, without any > > >> packaging subsystem. > > >> > > >> Here's a few words about wheels: > > >> > > >> https://wheel.readthedocs.io/ > > >> > > >> Here's about uploading to PyPI with twine: > > >> > > >> > > >> > > https://packaging.python.org/tutorials/packaging-projects/# > uploading-the-distribution-archives > > >> > > >> On 9/14/18 9:05 PM, Igor Sapego wrote: > > >>> Ok, good. > > >>> > > >>> Now, what is about installation? Which directories/files > > >>> need to be copied to ignite's binary release? > > >>> > > >>> Best Regards, > > >>> Igor > > >>> > > >> > > > > > > > >
Re: Python thin client
Great job. Best Regards, Igor On Tue, Sep 18, 2018 at 11:35 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > All examples are in 'ignite/modules/platforms/python/examples'. > > I put examples in separate Python files mostly to be able to > automatically confirm their operability. All the lengthy explanations > and cross-references are in the main documentation: > > > https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html > > To make it easier for end users to navigate in pyignite directories, I > added a small README file to the 'examples' directory with a short > description and a link to the docs: > > > https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/readme.md > > If you think this README is lacking something regardless of the main > pyignite docs, please let me know. > > On 9/17/18 8:59 PM, Igor Sapego wrote: > > Dmitry, > > > > Sorry, I was not clear enough. What I mean is that Ignite distributed > > by both source and binary releases. Binary releases contain platforms > > code, which is needed to write your own application in the language, > > but does not contain developer stuff, such as tests, documentation > > generating scripts, etc. > > > > Of course, it is more common to use package managers when > > developing your application, and of course, we are going to support > > this approach. But still, we provide a way for a user to get any client > > without any package manager. > > > > I like the idea of supplying whl package in a binary release, though it > > looks like it's going to take more effort to implement it. But I believe > > except for the whl package, we will need to supply examples and > > instructions, how user can run them. > > > > What do you think? > > > > Best Regards, > > Igor > > > > > > On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk < > > dmitry.melnic...@nobitlost.com> wrote: > > > >> Igor, > >> > >> I am in doubt here because I am not fully comprehend the meaning of > >> "binary release". But if it is somehow related to the "distribution" > >> thing, I would dare to suggest the following options: > >> > >> 1. Copy nothing. Just do > >> > >> ``` > >> $ python setup.py bdist_wheel > >> $ twine upload dist/* > >> ``` > >> > >> during the build process (or separately) and let PyPI handle the > >> distribution. > >> > >> This is the most natural and user-friendly way of distributing Python > >> packages. End user might only do > >> > >> ``` > >> $ pip install pyignite > >> ``` > >> > >> as it is suggested by my readme file. > >> > >> 2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist' > >> directory, that should appear there as the "python setup.py bdist_wheel" > >> command finishes. Actually, it can be combined with the first option. > >> > >> The wheel can then be installed by the end user: > >> > >> ``` > >> $ pip install pyignite-*.whl > >> ``` > >> > >> 3. Supply the following files: > >> > >> ignite/modules/platforms/python/pyignite/** > >> ignite/modules/platforms/python/requirements/** > >> ignite/modules/platforms/python/LICENSE > >> ignite/modules/platforms/python/README.md > >> ignite/modules/platforms/python/setup.py > >> > >> (** stands for "all the files and sub-folders recursively, starting from > >> this folder".) > >> > >> It is not uncommon or wrong to distribute Python programs as text > >> without prior packaging, but, in my personal experience, this is a way > >> more suitable for one-time scripts or proprietary programs. For example, > >> most of Python web apps relies on git for deployment, without any > >> packaging subsystem. > >> > >> Here's a few words about wheels: > >> > >> https://wheel.readthedocs.io/ > >> > >> Here's about uploading to PyPI with twine: > >> > >> > >> > https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives > >> > >> On 9/14/18 9:05 PM, Igor Sapego wrote: > >>> Ok, good. > >>> > >>> Now, what is about installation? Which directories/files > >>> need to be copied to ignite's binary release? > >>> > >>> Best Regards, > >>> Igor > >>> > >> > > > >
Re: Python thin client
Igor, All examples are in 'ignite/modules/platforms/python/examples'. I put examples in separate Python files mostly to be able to automatically confirm their operability. All the lengthy explanations and cross-references are in the main documentation: https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html To make it easier for end users to navigate in pyignite directories, I added a small README file to the 'examples' directory with a short description and a link to the docs: https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/readme.md If you think this README is lacking something regardless of the main pyignite docs, please let me know. On 9/17/18 8:59 PM, Igor Sapego wrote: Dmitry, Sorry, I was not clear enough. What I mean is that Ignite distributed by both source and binary releases. Binary releases contain platforms code, which is needed to write your own application in the language, but does not contain developer stuff, such as tests, documentation generating scripts, etc. Of course, it is more common to use package managers when developing your application, and of course, we are going to support this approach. But still, we provide a way for a user to get any client without any package manager. I like the idea of supplying whl package in a binary release, though it looks like it's going to take more effort to implement it. But I believe except for the whl package, we will need to supply examples and instructions, how user can run them. What do you think? Best Regards, Igor On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Igor, I am in doubt here because I am not fully comprehend the meaning of "binary release". But if it is somehow related to the "distribution" thing, I would dare to suggest the following options: 1. Copy nothing. Just do ``` $ python setup.py bdist_wheel $ twine upload dist/* ``` during the build process (or separately) and let PyPI handle the distribution. This is the most natural and user-friendly way of distributing Python packages. End user might only do ``` $ pip install pyignite ``` as it is suggested by my readme file. 2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist' directory, that should appear there as the "python setup.py bdist_wheel" command finishes. Actually, it can be combined with the first option. The wheel can then be installed by the end user: ``` $ pip install pyignite-*.whl ``` 3. Supply the following files: ignite/modules/platforms/python/pyignite/** ignite/modules/platforms/python/requirements/** ignite/modules/platforms/python/LICENSE ignite/modules/platforms/python/README.md ignite/modules/platforms/python/setup.py (** stands for "all the files and sub-folders recursively, starting from this folder".) It is not uncommon or wrong to distribute Python programs as text without prior packaging, but, in my personal experience, this is a way more suitable for one-time scripts or proprietary programs. For example, most of Python web apps relies on git for deployment, without any packaging subsystem. Here's a few words about wheels: https://wheel.readthedocs.io/ Here's about uploading to PyPI with twine: https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives On 9/14/18 9:05 PM, Igor Sapego wrote: Ok, good. Now, what is about installation? Which directories/files need to be copied to ignite's binary release? Best Regards, Igor
Re: Python thin client
Dmitry, Sorry, I was not clear enough. What I mean is that Ignite distributed by both source and binary releases. Binary releases contain platforms code, which is needed to write your own application in the language, but does not contain developer stuff, such as tests, documentation generating scripts, etc. Of course, it is more common to use package managers when developing your application, and of course, we are going to support this approach. But still, we provide a way for a user to get any client without any package manager. I like the idea of supplying whl package in a binary release, though it looks like it's going to take more effort to implement it. But I believe except for the whl package, we will need to supply examples and instructions, how user can run them. What do you think? Best Regards, Igor On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > I am in doubt here because I am not fully comprehend the meaning of > "binary release". But if it is somehow related to the "distribution" > thing, I would dare to suggest the following options: > > 1. Copy nothing. Just do > > ``` > $ python setup.py bdist_wheel > $ twine upload dist/* > ``` > > during the build process (or separately) and let PyPI handle the > distribution. > > This is the most natural and user-friendly way of distributing Python > packages. End user might only do > > ``` > $ pip install pyignite > ``` > > as it is suggested by my readme file. > > 2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist' > directory, that should appear there as the "python setup.py bdist_wheel" > command finishes. Actually, it can be combined with the first option. > > The wheel can then be installed by the end user: > > ``` > $ pip install pyignite-*.whl > ``` > > 3. Supply the following files: > > ignite/modules/platforms/python/pyignite/** > ignite/modules/platforms/python/requirements/** > ignite/modules/platforms/python/LICENSE > ignite/modules/platforms/python/README.md > ignite/modules/platforms/python/setup.py > > (** stands for "all the files and sub-folders recursively, starting from > this folder".) > > It is not uncommon or wrong to distribute Python programs as text > without prior packaging, but, in my personal experience, this is a way > more suitable for one-time scripts or proprietary programs. For example, > most of Python web apps relies on git for deployment, without any > packaging subsystem. > > Here's a few words about wheels: > > https://wheel.readthedocs.io/ > > Here's about uploading to PyPI with twine: > > > https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives > > On 9/14/18 9:05 PM, Igor Sapego wrote: > > Ok, good. > > > > Now, what is about installation? Which directories/files > > need to be copied to ignite's binary release? > > > > Best Regards, > > Igor > > >
Re: Python thin client
Igor, I am in doubt here because I am not fully comprehend the meaning of "binary release". But if it is somehow related to the "distribution" thing, I would dare to suggest the following options: 1. Copy nothing. Just do ``` $ python setup.py bdist_wheel $ twine upload dist/* ``` during the build process (or separately) and let PyPI handle the distribution. This is the most natural and user-friendly way of distributing Python packages. End user might only do ``` $ pip install pyignite ``` as it is suggested by my readme file. 2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist' directory, that should appear there as the "python setup.py bdist_wheel" command finishes. Actually, it can be combined with the first option. The wheel can then be installed by the end user: ``` $ pip install pyignite-*.whl ``` 3. Supply the following files: ignite/modules/platforms/python/pyignite/** ignite/modules/platforms/python/requirements/** ignite/modules/platforms/python/LICENSE ignite/modules/platforms/python/README.md ignite/modules/platforms/python/setup.py (** stands for "all the files and sub-folders recursively, starting from this folder".) It is not uncommon or wrong to distribute Python programs as text without prior packaging, but, in my personal experience, this is a way more suitable for one-time scripts or proprietary programs. For example, most of Python web apps relies on git for deployment, without any packaging subsystem. Here's a few words about wheels: https://wheel.readthedocs.io/ Here's about uploading to PyPI with twine: https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives On 9/14/18 9:05 PM, Igor Sapego wrote: Ok, good. Now, what is about installation? Which directories/files need to be copied to ignite's binary release? Best Regards, Igor
Re: Python thin client
Ok, good. Now, what is about installation? Which directories/files need to be copied to ignite's binary release? Best Regards, Igor On Fri, Sep 14, 2018 at 4:51 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > The commented code (lines 95-96) gives an error if executed. The error > is stated just below, in lines 98-100. It is explained here: > > > https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html#create > > I found out by trial and error that a cache, created with SQL DDL > statement (`CREATE TABLE`), can be deleted only with SQL DDL statement > (`DROP TABLE`), whereas a table, created with a binary protocol > operation (`OP_CACHE_CREATE_*` or `OP_CACHE_GET_OR_CREATE_*`) can be > deleted only with binary protocol operation (`OP_CACHE_DESTROY`). I miss > this fact in Ignite documentation. > > In this particular part of the example I try to stress this out with > regard to the cache created in the beginning of the example. The cache > behaves like an SQL table, but can not be dropped. You need to destroy > it instead (line 102). > > I would not go into such depths at all, but the examples are designed to > be runnable in automated environments, and thus must have some cleanup > code. And since the cleanup here is not absolutely trivial, I thought I > must explain it. > > I also could handle an error instead of commenting the erroneous code, like > > ``` > any_error = None > try: > DROP_QUERY = 'DROP TABLE Student' > client.sql(DROP_QUERY) > except Exception as e: > any_error = e > print(any_error) > > # pyignite.exceptions.SQLError: class > org.apache.ignite.IgniteCheckedException: > # Only cache created with CREATE TABLE may be removed with DROP TABLE > # [cacheName=SQL_PUBLIC_STUDENT] > > but it would complicate the cleanup part of the example to the point it > is lost any exemplariness. So I decided to simply use comments. > > On 9/14/18 2:48 AM, Igor Sapego wrote: > > Ok, now everything's running. > > > > API looks good to me. I have a single question about example code: > > What these comments are for - [1]? > > > > [1] - > > > https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/create_binary.py#L95 > > > > Best Regards, > > Igor > > > > > > On Thu, Sep 13, 2018 at 12:52 AM Dmitry Melnichuk < > > dmitry.melnic...@nobitlost.com> wrote: > > > >> Igor, > >> > >> Yes, it's my bad, sorry. Just merged the Ignite master with my branch. > >> > >> On 9/12/18 8:47 PM, Igor Sapego wrote: > >>> Pavel, > >>> Yes, I did. I tried completely clean environment, followed the same > >>> steps and got the same error. Then I removed attr, and out of sudden > >>> everything started working. > >>> > >>> Dmitry, > >>> Thanks, now it's more clear: > >>> Handshake error: Unsupported version. Server expects binary protocol > >>> version 1.1.0. Client provides 1.2.0. > >>> > >>> Why do you use version 1.2.0, but have outdated server code? I > >>> propose you to merge with or rebase onto Ignite's master branch. > >>> > >>> Best Regards, > >>> Igor > >>> > >>> > >>> On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < > >>> dmitry.melnic...@nobitlost.com> wrote: > >>> > Igor, > > I have just commited an improvment to the HandshakeError message > generation algorithm. I hope it is now easier to understand what > expects > what in case of binary protocol version mismatch. > > Thank you for pointing this out. > > On 9/12/18 2:13 AM, Igor Sapego wrote: > > I managed to start tests, and now I'm getting the following message: > > > > pyignite.exceptions.HandshakeError: Handshake error: Unsupported > > version. Expected protocol version: 1.1.0. > > > > It would be useful to print "Unexpected version" itself, because I > can > > not understand what is the issue. > > > > Best Regards, > > Igor > > >>> > >> > >> > > > >
Re: Python thin client
Igor, The commented code (lines 95-96) gives an error if executed. The error is stated just below, in lines 98-100. It is explained here: https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html#create I found out by trial and error that a cache, created with SQL DDL statement (`CREATE TABLE`), can be deleted only with SQL DDL statement (`DROP TABLE`), whereas a table, created with a binary protocol operation (`OP_CACHE_CREATE_*` or `OP_CACHE_GET_OR_CREATE_*`) can be deleted only with binary protocol operation (`OP_CACHE_DESTROY`). I miss this fact in Ignite documentation. In this particular part of the example I try to stress this out with regard to the cache created in the beginning of the example. The cache behaves like an SQL table, but can not be dropped. You need to destroy it instead (line 102). I would not go into such depths at all, but the examples are designed to be runnable in automated environments, and thus must have some cleanup code. And since the cleanup here is not absolutely trivial, I thought I must explain it. I also could handle an error instead of commenting the erroneous code, like ``` any_error = None try: DROP_QUERY = 'DROP TABLE Student' client.sql(DROP_QUERY) except Exception as e: any_error = e print(any_error) # pyignite.exceptions.SQLError: class org.apache.ignite.IgniteCheckedException: # Only cache created with CREATE TABLE may be removed with DROP TABLE # [cacheName=SQL_PUBLIC_STUDENT] but it would complicate the cleanup part of the example to the point it is lost any exemplariness. So I decided to simply use comments. On 9/14/18 2:48 AM, Igor Sapego wrote: Ok, now everything's running. API looks good to me. I have a single question about example code: What these comments are for - [1]? [1] - https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/create_binary.py#L95 Best Regards, Igor On Thu, Sep 13, 2018 at 12:52 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Igor, Yes, it's my bad, sorry. Just merged the Ignite master with my branch. On 9/12/18 8:47 PM, Igor Sapego wrote: Pavel, Yes, I did. I tried completely clean environment, followed the same steps and got the same error. Then I removed attr, and out of sudden everything started working. Dmitry, Thanks, now it's more clear: Handshake error: Unsupported version. Server expects binary protocol version 1.1.0. Client provides 1.2.0. Why do you use version 1.2.0, but have outdated server code? I propose you to merge with or rebase onto Ignite's master branch. Best Regards, Igor On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Igor, I have just commited an improvment to the HandshakeError message generation algorithm. I hope it is now easier to understand what expects what in case of binary protocol version mismatch. Thank you for pointing this out. On 9/12/18 2:13 AM, Igor Sapego wrote: I managed to start tests, and now I'm getting the following message: pyignite.exceptions.HandshakeError: Handshake error: Unsupported version. Expected protocol version: 1.1.0. It would be useful to print "Unexpected version" itself, because I can not understand what is the issue. Best Regards, Igor
Re: Python thin client
Ok, now everything's running. API looks good to me. I have a single question about example code: What these comments are for - [1]? [1] - https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/create_binary.py#L95 Best Regards, Igor On Thu, Sep 13, 2018 at 12:52 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > Yes, it's my bad, sorry. Just merged the Ignite master with my branch. > > On 9/12/18 8:47 PM, Igor Sapego wrote: > > Pavel, > > Yes, I did. I tried completely clean environment, followed the same > > steps and got the same error. Then I removed attr, and out of sudden > > everything started working. > > > > Dmitry, > > Thanks, now it's more clear: > > Handshake error: Unsupported version. Server expects binary protocol > > version 1.1.0. Client provides 1.2.0. > > > > Why do you use version 1.2.0, but have outdated server code? I > > propose you to merge with or rebase onto Ignite's master branch. > > > > Best Regards, > > Igor > > > > > > On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < > > dmitry.melnic...@nobitlost.com> wrote: > > > >> Igor, > >> > >> I have just commited an improvment to the HandshakeError message > >> generation algorithm. I hope it is now easier to understand what expects > >> what in case of binary protocol version mismatch. > >> > >> Thank you for pointing this out. > >> > >> On 9/12/18 2:13 AM, Igor Sapego wrote: > >>> I managed to start tests, and now I'm getting the following message: > >>> > >>> pyignite.exceptions.HandshakeError: Handshake error: Unsupported > >>> version. Expected protocol version: 1.1.0. > >>> > >>> It would be useful to print "Unexpected version" itself, because I can > >>> not understand what is the issue. > >>> > >>> Best Regards, > >>> Igor > >> > > > >
Re: Python thin client
Igor, Yes, it's my bad, sorry. Just merged the Ignite master with my branch. On 9/12/18 8:47 PM, Igor Sapego wrote: Pavel, Yes, I did. I tried completely clean environment, followed the same steps and got the same error. Then I removed attr, and out of sudden everything started working. Dmitry, Thanks, now it's more clear: Handshake error: Unsupported version. Server expects binary protocol version 1.1.0. Client provides 1.2.0. Why do you use version 1.2.0, but have outdated server code? I propose you to merge with or rebase onto Ignite's master branch. Best Regards, Igor On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Igor, I have just commited an improvment to the HandshakeError message generation algorithm. I hope it is now easier to understand what expects what in case of binary protocol version mismatch. Thank you for pointing this out. On 9/12/18 2:13 AM, Igor Sapego wrote: I managed to start tests, and now I'm getting the following message: pyignite.exceptions.HandshakeError: Handshake error: Unsupported version. Expected protocol version: 1.1.0. It would be useful to print "Unexpected version" itself, because I can not understand what is the issue. Best Regards, Igor
Re: Python thin client
Pavel, Yes, I did. I tried completely clean environment, followed the same steps and got the same error. Then I removed attr, and out of sudden everything started working. Dmitry, Thanks, now it's more clear: Handshake error: Unsupported version. Server expects binary protocol version 1.1.0. Client provides 1.2.0. Why do you use version 1.2.0, but have outdated server code? I propose you to merge with or rebase onto Ignite's master branch. Best Regards, Igor On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > I have just commited an improvment to the HandshakeError message > generation algorithm. I hope it is now easier to understand what expects > what in case of binary protocol version mismatch. > > Thank you for pointing this out. > > On 9/12/18 2:13 AM, Igor Sapego wrote: > > I managed to start tests, and now I'm getting the following message: > > > > pyignite.exceptions.HandshakeError: Handshake error: Unsupported > > version. Expected protocol version: 1.1.0. > > > > It would be useful to print "Unexpected version" itself, because I can > > not understand what is the issue. > > > > Best Regards, > > Igor >
Re: Python thin client
Igor, I have just commited an improvment to the HandshakeError message generation algorithm. I hope it is now easier to understand what expects what in case of binary protocol version mismatch. Thank you for pointing this out. On 9/12/18 2:13 AM, Igor Sapego wrote: I managed to start tests, and now I'm getting the following message: pyignite.exceptions.HandshakeError: Handshake error: Unsupported version. Expected protocol version: 1.1.0. It would be useful to print "Unexpected version" itself, because I can not understand what is the issue. Best Regards, Igor
Re: Python thin client
Hi Igor,
Did you have to do anything special to make the tests eventually run?
p.
On Tue, Sep 11, 2018 at 9:13 AM, Igor Sapego wrote:
> I managed to start tests, and now I'm getting the following message:
>
> pyignite.exceptions.HandshakeError: Handshake error: Unsupported
> version. Expected protocol version: 1.1.0.
>
> It would be useful to print "Unexpected version" itself, because I can
> not understand what is the issue.
>
> Best Regards,
> Igor
>
>
> On Tue, Sep 11, 2018 at 10:34 AM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
>
> > Igor,
> >
> > I literally followed the steps you describe, but unfortunately could not
> > reproduce the error.
> >
> > The only clue I got is that your site-packages already have the newer
> > version of attrs (18.2.0 against 18.1.0 as of pyignite requirements).
> > But this should not be an issue, since pytest-runner in this command
> > creates its own test environment, independent of the one it runs into.
> > It worked for me when I installed `attrs==18.2.0` locally and run tests
> > − everything went smooth.
> >
> > You can try uninstalling or downgrading your local attrs with
> >
> > ```
> > $ pip3 uninstall attrs
> > ```
> >
> > but I don't think it will actually help. Maybe it will raise another
> > error, so we could dig deeper into the problem.
> >
> > Alternatively, you can give me more details to recreate it (OS, Python
> > and pip versions). I'm not ruling out a bug in setuptools.
> >
> > On 9/11/18 2:13 AM, Igor Sapego wrote:
> > > Guys, I've cloned your repository, run pip3 install -e .
> > > then run pip3 install -r requirements/* over all the requirements,
> > > and finally run python ./setup.py pytest, but get the following output:
> > >
> > > running pytest
> > > Searching for attrs==18.1.0
> > > Best match: attrs 18.1.0
> > >
> > > Using /home/isapego/.local/lib/python3.6/site-packages
> > > running egg_info
> > > writing pyignite.egg-info/PKG-INFO
> > > writing dependency_links to pyignite.egg-info/dependency_links.txt
> > > writing requirements to pyignite.egg-info/requires.txt
> > > writing top-level names to pyignite.egg-info/top_level.txt
> > > reading manifest file 'pyignite.egg-info/SOURCES.txt'
> > > writing manifest file 'pyignite.egg-info/SOURCES.txt'
> > > running build_ext
> > > Traceback (most recent call last):
> > >File "./setup.py", line 98, in
> > > 'Operating System :: OS Independent',
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/
> setuptools/__init__.py",
> > > line 140, in setup
> > > return distutils.core.setup(**attrs)
> > >File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
> > > dist.run_commands()
> > >File "/usr/lib/python3.6/distutils/dist.py", line 955, in
> run_commands
> > > self.run_command(cmd)
> > >File "/usr/lib/python3.6/distutils/dist.py", line 974, in
> run_command
> > > cmd_obj.run()
> > >File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py",
> line
> > 175,
> > > in run
> > > with self.project_on_sys_path():
> > >File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
> > > return next(self.gen)
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/
> setuptools/command/test.py",
> > > line 166, in project_on_sys_path
> > > require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/pkg_
> resources/__init__.py",
> > > line 895, in require
> > > needed = self.resolve(parse_requirements(requirements))
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/pkg_
> resources/__init__.py",
> > > line 786, in resolve
> > > raise VersionConflict(dist, req).with_context(dependent_req)
> > > pkg_resources.ContextualVersionConflict: (attrs 18.2.0
> > > (/home/isapego/.local/lib/python3.6/site-packages),
> > > Requirement.parse('attrs==18.1.0'), {'pyignite'})
> > >
> > > What am I doing wrong?
> > >
> > > Best Regards,
> > > Igor
> > >
> >
>
Re: Python thin client
I managed to start tests, and now I'm getting the following message:
pyignite.exceptions.HandshakeError: Handshake error: Unsupported
version. Expected protocol version: 1.1.0.
It would be useful to print "Unexpected version" itself, because I can
not understand what is the issue.
Best Regards,
Igor
On Tue, Sep 11, 2018 at 10:34 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:
> Igor,
>
> I literally followed the steps you describe, but unfortunately could not
> reproduce the error.
>
> The only clue I got is that your site-packages already have the newer
> version of attrs (18.2.0 against 18.1.0 as of pyignite requirements).
> But this should not be an issue, since pytest-runner in this command
> creates its own test environment, independent of the one it runs into.
> It worked for me when I installed `attrs==18.2.0` locally and run tests
> − everything went smooth.
>
> You can try uninstalling or downgrading your local attrs with
>
> ```
> $ pip3 uninstall attrs
> ```
>
> but I don't think it will actually help. Maybe it will raise another
> error, so we could dig deeper into the problem.
>
> Alternatively, you can give me more details to recreate it (OS, Python
> and pip versions). I'm not ruling out a bug in setuptools.
>
> On 9/11/18 2:13 AM, Igor Sapego wrote:
> > Guys, I've cloned your repository, run pip3 install -e .
> > then run pip3 install -r requirements/* over all the requirements,
> > and finally run python ./setup.py pytest, but get the following output:
> >
> > running pytest
> > Searching for attrs==18.1.0
> > Best match: attrs 18.1.0
> >
> > Using /home/isapego/.local/lib/python3.6/site-packages
> > running egg_info
> > writing pyignite.egg-info/PKG-INFO
> > writing dependency_links to pyignite.egg-info/dependency_links.txt
> > writing requirements to pyignite.egg-info/requires.txt
> > writing top-level names to pyignite.egg-info/top_level.txt
> > reading manifest file 'pyignite.egg-info/SOURCES.txt'
> > writing manifest file 'pyignite.egg-info/SOURCES.txt'
> > running build_ext
> > Traceback (most recent call last):
> >File "./setup.py", line 98, in
> > 'Operating System :: OS Independent',
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/setuptools/__init__.py",
> > line 140, in setup
> > return distutils.core.setup(**attrs)
> >File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
> > dist.run_commands()
> >File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
> > self.run_command(cmd)
> >File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
> > cmd_obj.run()
> >File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py", line
> 175,
> > in run
> > with self.project_on_sys_path():
> >File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
> > return next(self.gen)
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/setuptools/command/test.py",
> > line 166, in project_on_sys_path
> > require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 895, in require
> > needed = self.resolve(parse_requirements(requirements))
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 786, in resolve
> > raise VersionConflict(dist, req).with_context(dependent_req)
> > pkg_resources.ContextualVersionConflict: (attrs 18.2.0
> > (/home/isapego/.local/lib/python3.6/site-packages),
> > Requirement.parse('attrs==18.1.0'), {'pyignite'})
> >
> > What am I doing wrong?
> >
> > Best Regards,
> > Igor
> >
>
Re: Python thin client
Igor,
I literally followed the steps you describe, but unfortunately could not
reproduce the error.
The only clue I got is that your site-packages already have the newer
version of attrs (18.2.0 against 18.1.0 as of pyignite requirements).
But this should not be an issue, since pytest-runner in this command
creates its own test environment, independent of the one it runs into.
It worked for me when I installed `attrs==18.2.0` locally and run tests
− everything went smooth.
You can try uninstalling or downgrading your local attrs with
```
$ pip3 uninstall attrs
```
but I don't think it will actually help. Maybe it will raise another
error, so we could dig deeper into the problem.
Alternatively, you can give me more details to recreate it (OS, Python
and pip versions). I'm not ruling out a bug in setuptools.
On 9/11/18 2:13 AM, Igor Sapego wrote:
Guys, I've cloned your repository, run pip3 install -e .
then run pip3 install -r requirements/* over all the requirements,
and finally run python ./setup.py pytest, but get the following output:
running pytest
Searching for attrs==18.1.0
Best match: attrs 18.1.0
Using /home/isapego/.local/lib/python3.6/site-packages
running egg_info
writing pyignite.egg-info/PKG-INFO
writing dependency_links to pyignite.egg-info/dependency_links.txt
writing requirements to pyignite.egg-info/requires.txt
writing top-level names to pyignite.egg-info/top_level.txt
reading manifest file 'pyignite.egg-info/SOURCES.txt'
writing manifest file 'pyignite.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "./setup.py", line 98, in
'Operating System :: OS Independent',
File
"/home/isapego/.local/lib/python3.6/site-packages/setuptools/__init__.py",
line 140, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py", line 175,
in run
with self.project_on_sys_path():
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File
"/home/isapego/.local/lib/python3.6/site-packages/setuptools/command/test.py",
line 166, in project_on_sys_path
require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
File
"/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 895, in require
needed = self.resolve(parse_requirements(requirements))
File
"/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 786, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (attrs 18.2.0
(/home/isapego/.local/lib/python3.6/site-packages),
Requirement.parse('attrs==18.1.0'), {'pyignite'})
What am I doing wrong?
Best Regards,
Igor
[jira] [Created] (IGNITE-9522) Document Python thin client
Prachi Garg created IGNITE-9522: --- Summary: Document Python thin client Key: IGNITE-9522 URL: https://issues.apache.org/jira/browse/IGNITE-9522 Project: Ignite Issue Type: Task Reporter: Prachi Garg Assignee: Prachi Garg Fix For: 2.7 Bring documentation from here - [https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html] into readme.io -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Python thin client
Guys, I've cloned your repository, run pip3 install -e .
then run pip3 install -r requirements/* over all the requirements,
and finally run python ./setup.py pytest, but get the following output:
running pytest
Searching for attrs==18.1.0
Best match: attrs 18.1.0
Using /home/isapego/.local/lib/python3.6/site-packages
running egg_info
writing pyignite.egg-info/PKG-INFO
writing dependency_links to pyignite.egg-info/dependency_links.txt
writing requirements to pyignite.egg-info/requires.txt
writing top-level names to pyignite.egg-info/top_level.txt
reading manifest file 'pyignite.egg-info/SOURCES.txt'
writing manifest file 'pyignite.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "./setup.py", line 98, in
'Operating System :: OS Independent',
File
"/home/isapego/.local/lib/python3.6/site-packages/setuptools/__init__.py",
line 140, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py", line 175,
in run
with self.project_on_sys_path():
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File
"/home/isapego/.local/lib/python3.6/site-packages/setuptools/command/test.py",
line 166, in project_on_sys_path
require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
File
"/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 895, in require
needed = self.resolve(parse_requirements(requirements))
File
"/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 786, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (attrs 18.2.0
(/home/isapego/.local/lib/python3.6/site-packages),
Requirement.parse('attrs==18.1.0'), {'pyignite'})
What am I doing wrong?
Best Regards,
Igor
On Wed, Sep 5, 2018 at 3:19 AM Dmitriy Setrakyan
wrote:
> Got it, sounds good!
>
> On Tue, Sep 4, 2018 at 10:54 AM, Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
>
> > Dmitriy,
> >
> > It would be quite messy to implement with Python modular system.
> >
> > First of all, Python 2 and Python 3 are different languages with a small
> > common subset of syntax rules. That's why what we see in a stack trace
> is a
> > syntax error, and not a “missing feature” error.
> >
> > Second, there is no single entry point in Python code. User is allowed to
> > import any name, from any module, in any order. In fact the module is run
> > when it first discovered by CPython during any `import` operation, and
> that
> > is how the imported entities are created and initialized: by the chain of
> > imports.
> >
> > It means for us, that implementing even a simple compatibility message in
> > Python 2 requires any module in our program (or at least all the modules,
> > that represent the public API of our library) to consist entirely of a
> > valid Python 2 code.
> >
> > It can be achieved by writing stubs with a version check, putting said
> > stubs in place of real modules, and proxying all the calls through the
> > conditional imports. It would take a small effort once, but make code
> less
> > readable and harder to maintain for the rest of its life cycle. Should
> we,
> > for example, provide two testing environments − for the main program and
> > for Python 2-compatible stubs?
> >
> > As far as I know, no Python developer is making such efforts nowadays.
> > There are some projects with a long history, that achieve
> 2/3-compatibility
> > through the use of restricted syntax and external packages like `six`, or
> > simply support two separate versions. Most of the new projects are
> creating
> > with the latest Python 3, pip and virtualenv in mind.
> >
> > I took the idea of my `setup.py` solution from the Django project, which
> > is dropped Python 2 support not long ago. Its setup relies on
> `setuptools`
> > 9+ option, or otherwise displays a simple banner; the banner is likely to
> > be buried under the further cryptic output of old setuptools, but it is
> > better than nothing.
> >
> >
> > On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote:
> >
> >> Dmitriy,
> >>
> >> setuptools sounds like an installation step. Does it make sense to add a
> >> check during startup of a client?
> >>
> >> D.
> >>
> >> On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
> >> dmitry.melnic...@nobitlost.com> wrote:
> >>
> >> Nikolay,
> >>>
> >>> There is indeed a feature in `setuptools` I just learned about, which
> >>> would help in this case (and I believe the case you demonstrated can be
> >>> quite typical, thank you for pointing it out). It gives user a clever
> >>> message in the end of a stack trace like
> >>>
> >>>
Re: Python thin client
Got it, sounds good! On Tue, Sep 4, 2018 at 10:54 AM, Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Dmitriy, > > It would be quite messy to implement with Python modular system. > > First of all, Python 2 and Python 3 are different languages with a small > common subset of syntax rules. That's why what we see in a stack trace is a > syntax error, and not a “missing feature” error. > > Second, there is no single entry point in Python code. User is allowed to > import any name, from any module, in any order. In fact the module is run > when it first discovered by CPython during any `import` operation, and that > is how the imported entities are created and initialized: by the chain of > imports. > > It means for us, that implementing even a simple compatibility message in > Python 2 requires any module in our program (or at least all the modules, > that represent the public API of our library) to consist entirely of a > valid Python 2 code. > > It can be achieved by writing stubs with a version check, putting said > stubs in place of real modules, and proxying all the calls through the > conditional imports. It would take a small effort once, but make code less > readable and harder to maintain for the rest of its life cycle. Should we, > for example, provide two testing environments − for the main program and > for Python 2-compatible stubs? > > As far as I know, no Python developer is making such efforts nowadays. > There are some projects with a long history, that achieve 2/3-compatibility > through the use of restricted syntax and external packages like `six`, or > simply support two separate versions. Most of the new projects are creating > with the latest Python 3, pip and virtualenv in mind. > > I took the idea of my `setup.py` solution from the Django project, which > is dropped Python 2 support not long ago. Its setup relies on `setuptools` > 9+ option, or otherwise displays a simple banner; the banner is likely to > be buried under the further cryptic output of old setuptools, but it is > better than nothing. > > > On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote: > >> Dmitriy, >> >> setuptools sounds like an installation step. Does it make sense to add a >> check during startup of a client? >> >> D. >> >> On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk < >> dmitry.melnic...@nobitlost.com> wrote: >> >> Nikolay, >>> >>> There is indeed a feature in `setuptools` I just learned about, which >>> would help in this case (and I believe the case you demonstrated can be >>> quite typical, thank you for pointing it out). It gives user a clever >>> message in the end of a stack trace like >>> >>> UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the running Python is 2.7.15 >>> >>> This feature is not 100% bullet-proof, and it will not help users who >>> will >>> try to install my client other way than with `pip` or `setup.py`. >>> It will also be less helpful with old versions of `pip` (before 9.0). >>> However, it should cover most situations. >>> >>> On 9/4/18 7:15 PM, Nikolay Izhikov wrote: >>> >>> Hello, Dmitry. I understand that for experienced Python developer it obvious from stack trace I send. But can we check python version on startup? And print big fat error message "You are using wrong python version". >>> From my experience, there are some tickets in Ignite that should be implemented in various thin clients. It a very trivial changes, but lack of testability makes this task harder then steel. I think a .Net DEVNOTES are very good example. Please, be gentle with your fellow contributors and make DEVNOTES as clear as possible. >>> >> >
Re: Python thin client
Dmitriy, It would be quite messy to implement with Python modular system. First of all, Python 2 and Python 3 are different languages with a small common subset of syntax rules. That's why what we see in a stack trace is a syntax error, and not a “missing feature” error. Second, there is no single entry point in Python code. User is allowed to import any name, from any module, in any order. In fact the module is run when it first discovered by CPython during any `import` operation, and that is how the imported entities are created and initialized: by the chain of imports. It means for us, that implementing even a simple compatibility message in Python 2 requires any module in our program (or at least all the modules, that represent the public API of our library) to consist entirely of a valid Python 2 code. It can be achieved by writing stubs with a version check, putting said stubs in place of real modules, and proxying all the calls through the conditional imports. It would take a small effort once, but make code less readable and harder to maintain for the rest of its life cycle. Should we, for example, provide two testing environments − for the main program and for Python 2-compatible stubs? As far as I know, no Python developer is making such efforts nowadays. There are some projects with a long history, that achieve 2/3-compatibility through the use of restricted syntax and external packages like `six`, or simply support two separate versions. Most of the new projects are creating with the latest Python 3, pip and virtualenv in mind. I took the idea of my `setup.py` solution from the Django project, which is dropped Python 2 support not long ago. Its setup relies on `setuptools` 9+ option, or otherwise displays a simple banner; the banner is likely to be buried under the further cryptic output of old setuptools, but it is better than nothing. On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote: Dmitriy, setuptools sounds like an installation step. Does it make sense to add a check during startup of a client? D. On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Nikolay, There is indeed a feature in `setuptools` I just learned about, which would help in this case (and I believe the case you demonstrated can be quite typical, thank you for pointing it out). It gives user a clever message in the end of a stack trace like UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the running Python is 2.7.15 This feature is not 100% bullet-proof, and it will not help users who will try to install my client other way than with `pip` or `setup.py`. It will also be less helpful with old versions of `pip` (before 9.0). However, it should cover most situations. On 9/4/18 7:15 PM, Nikolay Izhikov wrote: Hello, Dmitry. I understand that for experienced Python developer it obvious from stack trace I send. But can we check python version on startup? And print big fat error message "You are using wrong python version". From my experience, there are some tickets in Ignite that should be implemented in various thin clients. It a very trivial changes, but lack of testability makes this task harder then steel. I think a .Net DEVNOTES are very good example. Please, be gentle with your fellow contributors and make DEVNOTES as clear as possible.
Re: Python thin client
Dmitriy, setuptools sounds like an installation step. Does it make sense to add a check during startup of a client? D. On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Nikolay, > > There is indeed a feature in `setuptools` I just learned about, which > would help in this case (and I believe the case you demonstrated can be > quite typical, thank you for pointing it out). It gives user a clever > message in the end of a stack trace like > > > UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the > > running Python is 2.7.15 > > This feature is not 100% bullet-proof, and it will not help users who will > try to install my client other way than with `pip` or `setup.py`. > It will also be less helpful with old versions of `pip` (before 9.0). > However, it should cover most situations. > > On 9/4/18 7:15 PM, Nikolay Izhikov wrote: > >> Hello, Dmitry. >> >> I understand that for experienced Python developer it obvious from >> stack trace I send. >> >> But can we check python version on startup? And print big fat error >> message "You are using wrong python version". >> > > > >> From my experience, there are some tickets in Ignite that should be >> implemented in various thin clients. It a very trivial changes, but >> lack of testability makes this task harder then steel. >> >> I think a .Net DEVNOTES are very good example. >> >> Please, be gentle with your fellow contributors and make DEVNOTES as >> clear as possible. >> >
Re: Python thin client
Nikolay, There is indeed a feature in `setuptools` I just learned about, which would help in this case (and I believe the case you demonstrated can be quite typical, thank you for pointing it out). It gives user a clever message in the end of a stack trace like > UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the > running Python is 2.7.15 This feature is not 100% bullet-proof, and it will not help users who will try to install my client other way than with `pip` or `setup.py`. It will also be less helpful with old versions of `pip` (before 9.0). However, it should cover most situations. On 9/4/18 7:15 PM, Nikolay Izhikov wrote: Hello, Dmitry. I understand that for experienced Python developer it obvious from stack trace I send. But can we check python version on startup? And print big fat error message "You are using wrong python version". > From my experience, there are some tickets in Ignite that should be implemented in various thin clients. It a very trivial changes, but lack of testability makes this task harder then steel. I think a .Net DEVNOTES are very good example. Please, be gentle with your fellow contributors and make DEVNOTES as clear as possible.
Re: Python thin client
Hello, Dmitry. I understand that for experienced Python developer it obvious from stack trace I send. But can we check python version on startup? And print big fat error message "You are using wrong python version". From my experience, there are some tickets in Ignite that should be implemented in various thin clients. It a very trivial changes, but lack of testability makes this task harder then steel. I think a .Net DEVNOTES are very good example. Please, be gentle with your fellow contributors and make DEVNOTES as clear as possible. В Пн, 03/09/2018 в 17:07 +1000, Dmitry Melnichuk пишет: > Hello, Nikolay! > > Thank you for your interest in Python thin client. > > The traceback suggests that you using Python 2.7. Unfortunately, my > client supports only Python versions 3.4 and later. > > Since you are using the latest Ubuntu OS, chances are you already have a > recent Python 3 installed. Try `pip3`. > > On 9/3/18 4:25 PM, Nikolay Izhikov wrote: > > Hello, Dmitry. > > > > I tried to build your lib locally and it failed. > > Error message and pip version are below. > > > > Seems, we have to fix developer instructions. > > Do we need specific version of pip or something? > > I tried to use default versions. > > I using Ubuntu Linux. > > > > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v > > -e . > > Obtaining file:///home/dragon/src/ignite/modules/platforms/python > >Running setup.py > > (path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info > > for package from file:///home/dragon/src/ignite/modules/platforms/python > > Running command python setup.py egg_info > > Traceback (most recent call last): > >File "", line 1, in > >File "/home/dragon/src/ignite/modules/platforms/python/setup.py", > > line 20 > > def is_a_requirement(line: str) -> bool: > > ^ > > SyntaxError: invalid syntax > > Cleaning up... > > Command "python setup.py egg_info" failed with error code 1 in > > /home/dragon/src/ignite/modules/platforms/python/ > > Exception information: > > Traceback (most recent call last): > >File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in > > main > > status = self.run(options, args) > >File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line > > 353, in run > > wb.build(autobuilding=True) > >File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build > > self.requirement_set.prepare_files(self.finder) > >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in > > prepare_files > > ignore_dependencies=self.ignore_dependencies)) > >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in > > _prepare_file > > abstract_dist.prep_for_dist() > >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in > > prep_for_dist > > self.req_to_install.run_egg_info() > >File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line > > 439, in run_egg_info > > command_desc='python setup.py egg_info') > >File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, > > in call_subprocess > > % (command_desc, proc.returncode, cwd)) > > InstallationError: Command "python setup.py egg_info" failed with error > > code 1 in /home/dragon/src/ignite/modules/platforms/python/ > > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache > > policy python-pip > > python-pip: > >Установлен: 9.0.1-2.3~ubuntu1 > >Кандидат: 9.0.1-2.3~ubuntu1 > >Таблица версий: > > *** 9.0.1-2.3~ubuntu1 500 > > 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe > > amd64 Packages > > 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe > > i386 Packages > > 100 /var/lib/dpkg/status > > 9.0.1-2 500 > > 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 > > Packages > > 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 > > Packages > > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a > > Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC > > 2018 x86_64 x86_64 x86_64 GNU/Linux > > signature.asc Description: This is a digitally signed message part
Re: Python thin client
Hello, Nikolay! Thank you for your interest in Python thin client. The traceback suggests that you using Python 2.7. Unfortunately, my client supports only Python versions 3.4 and later. Since you are using the latest Ubuntu OS, chances are you already have a recent Python 3 installed. Try `pip3`. On 9/3/18 4:25 PM, Nikolay Izhikov wrote: Hello, Dmitry. I tried to build your lib locally and it failed. Error message and pip version are below. Seems, we have to fix developer instructions. Do we need specific version of pip or something? I tried to use default versions. I using Ubuntu Linux. dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v -e . Obtaining file:///home/dragon/src/ignite/modules/platforms/python Running setup.py (path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info for package from file:///home/dragon/src/ignite/modules/platforms/python Running command python setup.py egg_info Traceback (most recent call last): File "", line 1, in File "/home/dragon/src/ignite/modules/platforms/python/setup.py", line 20 def is_a_requirement(line: str) -> bool: ^ SyntaxError: invalid syntax Cleaning up... Command "python setup.py egg_info" failed with error code 1 in /home/dragon/src/ignite/modules/platforms/python/ Exception information: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run wb.build(autobuilding=True) File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build self.requirement_set.prepare_files(self.finder) File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in _prepare_file abstract_dist.prep_for_dist() File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in prep_for_dist self.req_to_install.run_egg_info() File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 439, in run_egg_info command_desc='python setup.py egg_info') File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, in call_subprocess % (command_desc, proc.returncode, cwd)) InstallationError: Command "python setup.py egg_info" failed with error code 1 in /home/dragon/src/ignite/modules/platforms/python/ dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache policy python-pip python-pip: Установлен: 9.0.1-2.3~ubuntu1 Кандидат: 9.0.1-2.3~ubuntu1 Таблица версий: *** 9.0.1-2.3~ubuntu1 500 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages 100 /var/lib/dpkg/status 9.0.1-2 500 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 Packages dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Re: Python thin client
Hello, Dmitry. I tried to build your lib locally and it failed. Error message and pip version are below. Seems, we have to fix developer instructions. Do we need specific version of pip or something? I tried to use default versions. I using Ubuntu Linux. dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v -e . Obtaining file:///home/dragon/src/ignite/modules/platforms/python Running setup.py (path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info for package from file:///home/dragon/src/ignite/modules/platforms/python Running command python setup.py egg_info Traceback (most recent call last): File "", line 1, in File "/home/dragon/src/ignite/modules/platforms/python/setup.py", line 20 def is_a_requirement(line: str) -> bool: ^ SyntaxError: invalid syntax Cleaning up... Command "python setup.py egg_info" failed with error code 1 in /home/dragon/src/ignite/modules/platforms/python/ Exception information: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run wb.build(autobuilding=True) File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build self.requirement_set.prepare_files(self.finder) File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in _prepare_file abstract_dist.prep_for_dist() File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in prep_for_dist self.req_to_install.run_egg_info() File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 439, in run_egg_info command_desc='python setup.py egg_info') File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, in call_subprocess % (command_desc, proc.returncode, cwd)) InstallationError: Command "python setup.py egg_info" failed with error code 1 in /home/dragon/src/ignite/modules/platforms/python/ dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache policy python-pip python-pip: Установлен: 9.0.1-2.3~ubuntu1 Кандидат: 9.0.1-2.3~ubuntu1 Таблица версий: *** 9.0.1-2.3~ubuntu1 500 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages 100 /var/lib/dpkg/status 9.0.1-2 500 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 Packages dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux В Пн, 03/09/2018 в 16:10 +1000, Dmitry Melnichuk пишет: > Hello, Igniters! > > Please review my work on Ignite thin client library written in Python 3. > > Pull request: > https://github.com/apache/ignite/pull/4278 > > Jira issue with initial proposal: > https://issues.apache.org/jira/browse/IGNITE-7782 > > The documentation is temporarily available at: > https://apache-ignite-binary-protocol-client.readthedocs.io/ > > It covers installation, requirements, API specification, coding > examples, instructions on how to run tests and build the documentation > itself. > > Dmitry signature.asc Description: This is a digitally signed message part
Python thin client
Hello, Igniters! Please review my work on Ignite thin client library written in Python 3. Pull request: https://github.com/apache/ignite/pull/4278 Jira issue with initial proposal: https://issues.apache.org/jira/browse/IGNITE-7782 The documentation is temporarily available at: https://apache-ignite-binary-protocol-client.readthedocs.io/ It covers installation, requirements, API specification, coding examples, instructions on how to run tests and build the documentation itself. Dmitry
