[jira] [Created] (IGNITE-9343) ignite-geospatial is not on maven central

[Grant Haywood (JIRA)]

Grant Haywood created IGNITE-9343:
-

 Summary: ignite-geospatial is not on maven central
 Key: IGNITE-9343
 URL: https://issues.apache.org/jira/browse/IGNITE-9343
 Project: Ignite
  Issue Type: Bug
Reporter: Grant Haywood


I am wondering why ignite-geospatial is not available on maven central? 

 

[https://apacheignite-sql.readme.io/docs/geospatial-support#including-ignite-geospatial-library]

 

the docs here would lead me to believe I just need to add it to my POM

 

is it expected that users need to compile ignite locally to use geospatial 
index?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[MTCGA]: new failures in builds [1703435] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New test failure in master SchemaExchangeSelfTest.testNonEmptyDynamic 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1783392241523438551=%3Cdefault%3E=testDetails
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dpav...@apache.org or write to 
dev.list 
Best Regards,
MTCGA.Bot 
Notification generated at Wed Aug 22 04:07:37 MSK 2018 

Re: Does Ignite support java.sql.Array?

[Denis Magda]

I've removed the ARRAY data type from the supported list.

Regardless of this, Vova, could you answer to Dmiry's question?

--
Denis

On Tue, Aug 14, 2018 at 8:28 AM Dmitriy Setrakyan 
wrote:

>
>
> On Tue, Aug 14, 2018 at 3:34 AM, Vladimir Ozerov 
> wrote:
>
>> Hi,
>>
>> No, we do not support ARRAY data type at the moment. Let's remove it from
>> docs.
>>
>
> Vladimir, what is the reason for not supporting it? ARRAY is a basic SQL
> type and not supporting it presents a serious limitation.
>
> D.
>

Re: Apache ignite contribution

[Denis Magda]

Hi,

Please tell your JIRA id so that we grant required permissions.

--
Denis

On Tue, Aug 21, 2018 at 7:59 PM SUMIT KESARWANI 
wrote:

> Hi All,
> wanted to contribute in apache ignite project.
> i request to admin , please provide me the access for jira to assign issue
> for myself.
>
>
> -Sumit Kesarwani
> India
>

[MTCGA]: new failures in builds [1703433] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New test failure in master 
IgniteCacheAtomicProtocolTest.testPutReaderUpdate1 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5377747406804372686=%3Cdefault%3E=testDetails
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dpav...@apache.org or write to 
dev.list 
Best Regards,
MTCGA.Bot 
Notification generated at Wed Aug 22 03:22:33 MSK 2018 

Apache ignite contribution

[SUMIT KESARWANI]

Hi All,
wanted to contribute in apache ignite project.
i request to admin , please provide me the access for jira to assign issue
for myself.


-Sumit Kesarwani
India

[MTCGA]: new failures in builds [1701592] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New test failure in master 
IgniteClusterActivateDeactivateTestWithPersistence.testClientReconnectClusterActivated
 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-390174676041430081=%3Cdefault%3E=testDetails
 Changes may led to failure were done by 
 - dmitrievanthony 
http://ci.ignite.apache.org/viewModification.html?modId=829030=false
 - zaleslaw.sin 
http://ci.ignite.apache.org/viewModification.html?modId=829029=false
 - michael.cherkasov 
http://ci.ignite.apache.org/viewModification.html?modId=829026=false
 - alexey.goncharuk 
http://ci.ignite.apache.org/viewModification.html?modId=829012=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dpav...@apache.org or write to 
dev.list 
Best Regards,
MTCGA.Bot 
Notification generated at Wed Aug 22 01:22:37 MSK 2018 

Re: Table Names in Spark Catalog

[Valentin Kulichenko]

Stuart,

Thanks for pointing this out, I was not aware that we use Spark database
concept this way. Actually, this confuses me a lot. As far as I understand,
catalog is created in the scope of a particular IgniteSparkSession, which
in turn is assigned to a particular IgniteContext and therefore single
Ignite client. If that's the case, I don't think it should be aware of
other Ignite clients that are connected to other clusters. This doesn't
look like correct behavior to me, not to mention that with this approach
having multiple databases would be a very rare case. I believe we should
get rid of this logic and use Ignite schema name as database name in
Spark's catalog.

Nikolay, what do you think?

-Val

On Tue, Aug 21, 2018 at 8:17 AM Stuart Macdonald  wrote:

> Nikolay, Val,
>
> The JDBC Spark datasource[1] -- as far as I can tell -- has no
> ExternalCatalog implementation, it just uses the database specified in the
> JDBC URL. So I don't believe there is any way to call listTables() or
> listDatabases() for JDBC provider.
>
> The Hive ExternalCatalog[2] makes the distinction between database and
> table using the actual database and table mechanisms built into the
> catalog, which is fine because Hive has the clear distinction and hierarchy
> of databases and tables.
>
> *However* Ignite already uses the "database" concept in the Ignite
> ExternalCatalog[3] to mean the name of an Ignite instance. So in Ignite we
> have instances containing schemas containing tables, and Spark only has the
> concept of databases and tables so it seems like either we ignore one of
> the three Ignite concepts or combine two of them into database or table.
> The current implementation in the pull request combines Ignite schema and
> table attributes into the Spark table attribute.
>
> Stuart.
>
> [1]
>
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
> [2]
>
> https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
> [3]
>
> https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/spark/sql/ignite/IgniteExternalCatalog.scala
>
> On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov 
> wrote:
>
> > Hello, Stuart.
> >
> > Can you do some research and find out how schema is handled in Data
> Frames
> > for a regular RDBMS such as Oracle, MySQL, etc?
> >
> > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> > > Stuart, Nikolay,
> > >
> > > I see that the 'Table' class (returned by listTables method) has a
> > 'database' field. Can we use this one to report schema name?
> > >
> > > In any case, I think we should look into how this is done in data
> source
> > implementations for other databases. Any relational database has a notion
> > of schema, and I'm sure Spark integrations take this into account
> somehow.
> > >
> > > -Val
> > >
> > > On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov 
> > wrote:
> > > > Hello, Stuart.
> > > >
> > > > Personally, I think we should change current tables naming and return
> > table in form of `schema.table`.
> > > >
> > > > Valentin, could you share your opinion?
> > > >
> > > >
> > > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет:
> > > > > Igniters,
> > > > >
> > > > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I
> are
> > > > > discussing whether to introduce a change which may impact backwards
> > > > > compatibility; Nikolay suggested we take the discussion to this
> list.
> > > > >
> > > > > Ignite implements a custom Spark catalog which provides an API by
> > which
> > > > > Spark users can list the tables which are available in Ignite which
> > can be
> > > > > queried via Spark SQL. Currently that table name list includes just
> > the
> > > > > names of the tables, but IGNITE-9228 is introducing a change which
> > allows
> > > > > optional prefixing of schema names to table names to disambiguate
> > multiple
> > > > > tables with the same name in different schemas. For the "list
> > tables" API
> > > > > we therefore have two options:
> > > > >
> > > > > 1. List the tables using both their table names and
> schema-qualified
> > table
> > > > > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are
> > the same
> > > > > underlying table. This retains backwards compatibility with users
> who
> > > > > expect "myTable" to appear in the catalog.
> > > > > 2. List the tables using only their schema-qualified names. This
> > eliminates
> > > > > duplication of names in the catalog but will potentially break
> > > > > compatibility with users who expect the table name in the catalog.
> > > > >
> > > > > With either option we will allow for  Spark SQL SELECT statements
> to
> > use
> > > > > either table name or schema-qualified table names, this change
> would
> > purely
> > > > > impact the API which is used to list available tables.
> > > > >
> > > > > Any opinions 

Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

[Alex Plehanov]

Dmitriy, yes, you correctly understood my idea.

вт, 21 авг. 2018 г. в 22:03, Vladimir Ozerov :

> Guys,
>
> Do we have any numbers for real workloads? Or at least Yardstick.
>
> вт, 21 авг. 2018 г. в 21:04, Dmitriy Pavlov :
>
> > Let us double-check current options. Maybe I'm mistaken about Alex
> > Plehanov's idea.
> >
> > Let's introduce the following variables and functions:
> > data - our block.
> > OldImpl(data) = X
> > NewImpl(data) = Y
> > NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR
> operation.
> > And NewImpl(data) ^ C  = X = OldImpl(data)
> >
> > So new and old implementations give us X in all cases, we don't need any
> > compatibility mode.
> >
> > Let's try to adopt new fast implementation with C constant.
> >
> > Evgeniy, would you like to try?
> >
> > вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov :
> >
> > > Alex
> > >
> > > In that case the data becomes in unpredictable state (mix of new and
> old
> > > methods) and there's a chance that some partitions will be never
> touched
> > > and never converted. It will force us always support old compression.
> > > I suppose the explicit conversion that clearly say what is happening
> now
> > > and may be warn that the db files should backed up before version
> > upgrade.
> > >
> > > Also I think AI 3.0 will have set of incompatible changes that give us
> a
> > > chance to add an upgrade procedure where compression method update
> among
> > > other stuff will take into account.
> > >
> > > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov <
> plehanov.a...@gmail.com
> > >
> > > wrote:
> > >
> > > > Sergey, converting data also force us to introduce some flag to WAL
> > > > segments/records or convert WAL segments too. WAL segments can be
> > > archived,
> > > > they also can be compressed.
> > > > IMO we better should not introduce any compatibility modes, left data
> > as
> > > is
> > > > and always just convert crc value returned by zip.CRC32 to old format
> > > (xor
> > > > it) at runtime.
> > > >
> > > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov :
> > > >
> > > > > Dmitriy
> > > > >
> > > > > Due to significant improvement and to reduce the number supported
> > > > > modes/options would be good to convert the data at the moment of
> > > upgrade.
> > > > >
> > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Sergey, that was precisely my comment in the ticket:
> > > > > >
> > > > > > Can we add this option without breaking compatibility with
> previous
> > > > > > page/storage formats? If not, then this should support both
> > > > > implementation.
> > > > > > The default should be the new fastest implementation, but if we
> > > > encounter
> > > > > > the older, slower one, then we should print out a warning in the
> > log
> > > > and
> > > > > > automatically switch to the older implementation.
> > > > > >
> > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov <
> > skoz...@gridgain.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Igniters
> > > > > > >
> > > > > > > I suppose that'll break compatibility for LFS (PDS).
> > > > > > >
> > > > > > > Do we plan to provide a migration guide w/o data loss for
> upgrade
> > > AI
> > > > > 2.x
> > > > > > to
> > > > > > > 3.0?
> > > > > > >
> > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I commented in the ticket: https://issues.apache.org/
> > > > > > > > jira/browse/IGNITE-9272
> > > > > > > >
> > > > > > > > It if can integrate it correctly, according to my comment, in
> > 2.7
> > > > > > > release,
> > > > > > > > it would be great. Otherwise, let's plan this change for 3.0
> > > > release.
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
> > > > > > > > eduard.shangar...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I have checked the benchmark and it shows great performance
> > > boost
> > > > > on
> > > > > > my
> > > > > > > > > laptop!
> > > > > > > > >
> > > > > > > > > +1 for this change.
> > > > > > > > >
> > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov <
> > > > > > dpavlov@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Evgeniy,
> > > > > > > > > >
> > > > > > > > > > Thank you. I see that the ticket is unassigned.
> > > > > > > > > >
> > > > > > > > > > Would you like to contribute PR to be macro-benchmarked
> > with
> > > > > > Ignite?
> > > > > > > > > >
> > > > > > > > > > Sincerely,
> > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > >
> > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > > > > > > > > > :
> > > > > > > > > >
> > > > > > > > > > > I fill the ticket, bench code attached there.
> > > > > > > > > > > 

[GitHub] ignite pull request #4590: Ignite 2.4.9

[slukyano]

GitHub user slukyano opened a pull request:

https://github.com/apache/ignite/pull/4590

Ignite 2.4.9



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4590.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4590


commit aae4e97b786e1bc9b2353ddee61ff7077b8d759c
Author: Pavel Kovalenko 
Date:   2018-03-06T18:22:40Z

IGNITE-6113 Return back old implementation instead of 'removeIf' - Fixes 
#3609.

Signed-off-by: Alexey Goncharuk 
(cherry picked from commit 8c3731e83296e1750c7c098b283c3fd8584b1be4)

commit d60d4fd8f6abe652a5618e5644fb978bcb1495b0
Author: Denis Mekhanikov 
Date:   2018-03-20T02:19:35Z

IGNITE-7794 - Share persisted marshaller mappings when connecting - Fixes 
#3620.

Signed-off-by: Valentin Kulichenko 
(cherry picked from commit c4ccf55dd6eb22102c56e211812eb4ccc206f4a7)

commit 70efde3305a1559231583f453aeb228ed2d40896
Author: Valentin Kulichenko 
Date:   2018-03-20T02:20:20Z

Added new test to suite

Signed-off-by: Valentin Kulichenko 
(cherry picked from commit baf08bc9501676e418dab2431211a659ae421d25)

commit 002e4d650c072c0927a93075e79d2ab1652b475d
Author: EdShangGG 
Date:   2018-03-22T14:58:39Z

IGNITE-8007 We should treat as empty any partition as empty if it doesn't 
have any data - Fixes #3677.

(cherry picked from commit 14f7bce)

commit 699561265ae616b5eb75897343be84d8c83be804
Author: Anton Kalashnikov 
Date:   2018-03-21T15:53:15Z

GG-13631 fix GridDhtPartitionDemandLegacyMessage

(cherry picked from commit ffbc56e)

commit 37c8033c72ac4fd1ec1e419ba68142c4fe11ad8b
Author: tledkov-gridgain 
Date:   2018-03-13T09:37:29Z

IGNITE-7860: JDBC thin: changed default socket buffer size to 64Kb. This 
closes #3600.

commit 130adcf29ddb61f8e9baa784b81454d3ed7c3b75
Author: Pavel Tupitsyn 
Date:   2018-01-26T08:48:14Z

IGNITE-7530 .NET: Fix GetAll memory usage and performance in binary mode

This closes #3436

commit 824004909b23a9a7d599500967af34103acb8c47
Author: Igor Sapego 
Date:   2018-01-30T12:56:17Z

IGNITE-6810: Implemented SSL support for ODBC.

This closes #3361

commit 82da0b5e9dc2ee2339c3fb1023e35d415bf1b647
Author: Pavel Kuznetsov 
Date:   2018-02-07T12:37:52Z

IGNITE-6217: Added benchmark to compare JDBC vs native SQL

This closes #2558

commit 701c6f141f6812ad7bc050a86266e696cf5863ed
Author: tledkov-gridgain 
Date:   2018-02-08T12:29:42Z

IGNITE-6625: JDBC thin: support SSL connection to Ignite node

This closes #3233

commit 2d729bf5c6f6fca9d07be2d57850642ba4b55004
Author: tledkov-gridgain 
Date:   2018-02-09T11:08:15Z

IGNITE-6625: SSL support for thin JDBC: additional fix test; change error 
message

commit 8994f847d7f5f15db73706d9210cdccb1cf3fb26
Author: devozerov 
Date:   2018-02-12T13:34:24Z

IGNITE-7003: Fixed faulty test 
WalModeChangeAdvancedSelfTest.testJoinCoordinator.

commit b142712264007d7397d1594541681a4a7e3d4b93
Author: Igor Sapego 
Date:   2018-02-26T12:02:07Z

IGNITE-7362: Fixed PDO ODBC integration issue

commit a2b2aee52cc65d01f2ecaf9462adc4bd368438ea
Author: Igor Sapego 
Date:   2018-02-28T10:23:12Z

IGNITE-7763: Fixed 32bit tests configurations to prevent OOM

This closes #3557

commit 652f3c4cdbaad40f5de25b06f0c13710aa7f2fd9
Author: Pavel Kuznetsov 
Date:   2018-03-13T12:46:36Z

IGNITE-7531: Data load benchmarks. This closes #3571.

commit 9337a53d9fcd62af87f6760080d350b43e275105
Author: tledkov-gridgain 
Date:   2018-03-16T11:38:38Z

IGNITE-7879: Fixed splitter logic for DISTINCT with subqueries. This closes 
#3634.

commit 7bec8b13cb373002d2a6b1b268d410338259bac2
Author: Igor Sapego 
Date:   2018-03-19T11:17:33Z

IGNITE-7811: Implemented connection failover for ODBC

This closes #3643

commit e512e5e0a2602df0ecfb69b2b5efabce836b04db
Author: Igor Sapego 
Date:   2018-03-20T10:37:02Z

IGNITE-7888: Implemented login timeout for ODBC.

This closes #3657

commit 211fca3a55e84b78ff0c1af04d91e25d6fc846c4
Author: devozerov 
Date:   2018-03-20T11:13:46Z

IGNITE-7984: SQL: improved deadlock handling in DML. This closes #3655.

commit bcd2888d27afe65f1a060e35b99a05ea420979c7
Author: Roman Guseinov 
Date:   2018-02-16T09:57:26Z

IGNITE-7192: Implemented JDBC support FQDN to multiple IPs

This closes #3439

commit d2659d0ec9f6e1a0b905fc7bf23b65fd5522c80a
Author: Alexander Paschenko 
Date:   2018-03-14T09:23:37Z

IGNITE-7253: JDBC thin driver: implemented streaming. This closes #3499. 
This closes #3591.

commit bc9018ef8b116f81b8e06d2ff7651ba2b6c7beae
Author: tledkov-gridgain 
Date:   2018-03-19T08:01:26Z

IGNITE-7029: JDBC thin driver now supports 

Re: Apache Ignite 2.7 release

[Dmitriy Setrakyan]

Thanks, Nikolay!

I think it is important to include the links to all important Jira tickets
in this thread, so that the community can track them.

D.

On Tue, Aug 21, 2018 at 12:06 AM, Nikolay Izhikov 
wrote:

> Hello, Dmitriy.
>
> I think Transparent Data Encryption will be available in 2.7
>
> В Пн, 20/08/2018 в 13:20 -0700, Dmitriy Setrakyan пишет:
> > Hi Nikolay,
> >
> > Thanks for being the release manager!
> >
> > I am getting a bit lost in all these tickets. Can we specify some
> > high-level tickets, that are not plain bug fixes, which will be
> interesting
> > for the community to notice?
> >
> > For example, here are some significant tasks that the community is either
> > working on or has been working on:
> >
> > - Node.JS client
> > - Python client
> > - Transactional SQL (MVCC)
> > - service grid stabilization
> > - SQL memory utilization improvements
> > - more?
> >
> > Can you please solicit status from the community for these tasks?
> >
> > D.
> >
> > On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov 
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I'm release manager of Apache Ignite 2.7.
> > >
> > > It's time to start discussion of release. [1]
> > >
> > > Current code freeze date is September, 30.
> > > If you have any objections - please, responsd to this thread.
> > >
> > > [1] https://cwiki.apache.org/confluence/display/IGNITE/
> Apache+Ignite+2.7
>

Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

[Vladimir Ozerov]

Guys,

Do we have any numbers for real workloads? Or at least Yardstick.

вт, 21 авг. 2018 г. в 21:04, Dmitriy Pavlov :

> Let us double-check current options. Maybe I'm mistaken about Alex
> Plehanov's idea.
>
> Let's introduce the following variables and functions:
> data - our block.
> OldImpl(data) = X
> NewImpl(data) = Y
> NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR operation.
> And NewImpl(data) ^ C  = X = OldImpl(data)
>
> So new and old implementations give us X in all cases, we don't need any
> compatibility mode.
>
> Let's try to adopt new fast implementation with C constant.
>
> Evgeniy, would you like to try?
>
> вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov :
>
> > Alex
> >
> > In that case the data becomes in unpredictable state (mix of new and old
> > methods) and there's a chance that some partitions will be never touched
> > and never converted. It will force us always support old compression.
> > I suppose the explicit conversion that clearly say what is happening now
> > and may be warn that the db files should backed up before version
> upgrade.
> >
> > Also I think AI 3.0 will have set of incompatible changes that give us a
> > chance to add an upgrade procedure where compression method update among
> > other stuff will take into account.
> >
> > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov  >
> > wrote:
> >
> > > Sergey, converting data also force us to introduce some flag to WAL
> > > segments/records or convert WAL segments too. WAL segments can be
> > archived,
> > > they also can be compressed.
> > > IMO we better should not introduce any compatibility modes, left data
> as
> > is
> > > and always just convert crc value returned by zip.CRC32 to old format
> > (xor
> > > it) at runtime.
> > >
> > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov :
> > >
> > > > Dmitriy
> > > >
> > > > Due to significant improvement and to reduce the number supported
> > > > modes/options would be good to convert the data at the moment of
> > upgrade.
> > > >
> > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > Sergey, that was precisely my comment in the ticket:
> > > > >
> > > > > Can we add this option without breaking compatibility with previous
> > > > > page/storage formats? If not, then this should support both
> > > > implementation.
> > > > > The default should be the new fastest implementation, but if we
> > > encounter
> > > > > the older, slower one, then we should print out a warning in the
> log
> > > and
> > > > > automatically switch to the older implementation.
> > > > >
> > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov <
> skoz...@gridgain.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Igniters
> > > > > >
> > > > > > I suppose that'll break compatibility for LFS (PDS).
> > > > > >
> > > > > > Do we plan to provide a migration guide w/o data loss for upgrade
> > AI
> > > > 2.x
> > > > > to
> > > > > > 3.0?
> > > > > >
> > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I commented in the ticket: https://issues.apache.org/
> > > > > > > jira/browse/IGNITE-9272
> > > > > > >
> > > > > > > It if can integrate it correctly, according to my comment, in
> 2.7
> > > > > > release,
> > > > > > > it would be great. Otherwise, let's plan this change for 3.0
> > > release.
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
> > > > > > > eduard.shangar...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have checked the benchmark and it shows great performance
> > boost
> > > > on
> > > > > my
> > > > > > > > laptop!
> > > > > > > >
> > > > > > > > +1 for this change.
> > > > > > > >
> > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov <
> > > > > dpavlov@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Evgeniy,
> > > > > > > > >
> > > > > > > > > Thank you. I see that the ticket is unassigned.
> > > > > > > > >
> > > > > > > > > Would you like to contribute PR to be macro-benchmarked
> with
> > > > > Ignite?
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > > Dmitriy Pavlov
> > > > > > > > >
> > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > > > > > > > > :
> > > > > > > > >
> > > > > > > > > > I fill the ticket, bench code attached there.
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >Has anyone else run the benchmark and reproduced the
> > > > performance
> > > > > > > > > > >difference?
> > > > > > > > > > >
> > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov <
> > > > > > > > dpavlov@gmail.com
> > > > > > > > > >
> > > > > > > > > > >wrote:
> > > > > > > > > > >
> > > > > > > > > > >> 

Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

[Dmitriy Pavlov]

Let us double-check current options. Maybe I'm mistaken about Alex
Plehanov's idea.

Let's introduce the following variables and functions:
data - our block.
OldImpl(data) = X
NewImpl(data) = Y
NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR operation.
And NewImpl(data) ^ C  = X = OldImpl(data)

So new and old implementations give us X in all cases, we don't need any
compatibility mode.

Let's try to adopt new fast implementation with C constant.

Evgeniy, would you like to try?

вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov :

> Alex
>
> In that case the data becomes in unpredictable state (mix of new and old
> methods) and there's a chance that some partitions will be never touched
> and never converted. It will force us always support old compression.
> I suppose the explicit conversion that clearly say what is happening now
> and may be warn that the db files should backed up before version upgrade.
>
> Also I think AI 3.0 will have set of incompatible changes that give us a
> chance to add an upgrade procedure where compression method update among
> other stuff will take into account.
>
> On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov 
> wrote:
>
> > Sergey, converting data also force us to introduce some flag to WAL
> > segments/records or convert WAL segments too. WAL segments can be
> archived,
> > they also can be compressed.
> > IMO we better should not introduce any compatibility modes, left data as
> is
> > and always just convert crc value returned by zip.CRC32 to old format
> (xor
> > it) at runtime.
> >
> > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov :
> >
> > > Dmitriy
> > >
> > > Due to significant improvement and to reduce the number supported
> > > modes/options would be good to convert the data at the moment of
> upgrade.
> > >
> > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > wrote:
> > >
> > > > Sergey, that was precisely my comment in the ticket:
> > > >
> > > > Can we add this option without breaking compatibility with previous
> > > > page/storage formats? If not, then this should support both
> > > implementation.
> > > > The default should be the new fastest implementation, but if we
> > encounter
> > > > the older, slower one, then we should print out a warning in the log
> > and
> > > > automatically switch to the older implementation.
> > > >
> > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov  >
> > > > wrote:
> > > >
> > > > > Hi Igniters
> > > > >
> > > > > I suppose that'll break compatibility for LFS (PDS).
> > > > >
> > > > > Do we plan to provide a migration guide w/o data loss for upgrade
> AI
> > > 2.x
> > > > to
> > > > > 3.0?
> > > > >
> > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I commented in the ticket: https://issues.apache.org/
> > > > > > jira/browse/IGNITE-9272
> > > > > >
> > > > > > It if can integrate it correctly, according to my comment, in 2.7
> > > > > release,
> > > > > > it would be great. Otherwise, let's plan this change for 3.0
> > release.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
> > > > > > eduard.shangar...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have checked the benchmark and it shows great performance
> boost
> > > on
> > > > my
> > > > > > > laptop!
> > > > > > >
> > > > > > > +1 for this change.
> > > > > > >
> > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov <
> > > > dpavlov@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Evgeniy,
> > > > > > > >
> > > > > > > > Thank you. I see that the ticket is unassigned.
> > > > > > > >
> > > > > > > > Would you like to contribute PR to be macro-benchmarked with
> > > > Ignite?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > > > > > > > :
> > > > > > > >
> > > > > > > > > I fill the ticket, bench code attached there.
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >Has anyone else run the benchmark and reproduced the
> > > performance
> > > > > > > > > >difference?
> > > > > > > > > >
> > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov <
> > > > > > > dpavlov@gmail.com
> > > > > > > > >
> > > > > > > > > >wrote:
> > > > > > > > > >
> > > > > > > > > >> It depends.
> > > > > > > > > >>
> > > > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and
> > page
> > > > > store
> > > > > > > > write
> > > > > > > > > >> are mostly about IO speed.
> > > > > > > > > >>
> > > > > > > > > >> In the same time, it can make the huge impact on
> machines
> > > with
> > > > > > fast
> > > > > > > IO
> > > > > > > > > >> and
> > > > > > > > > >> slow CPU. So if we can apply change 

Re: Workflow improvement

[Dmitriy Pavlov]

Hi Dmitriy,

The Bot is able to detect a frequent change of test status, but currently
only 50 last runs count. Same is true for the failure rate.

This value can be easily changed to 70 or 100, moreover, the auto
trigger feature gives us much more builds.

We can improve these rules. We can add not only status change, but status
change without any code changes. We can somehow save this data in RunStat
class. Let's create a better rule, and later we can code it.

Sincerely,
Dmitriy Pavlov

вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov :

> I think plugin will be more pretty looking, but comments can contain any
> information, so they can be more usefull. I agree with your idea to create
> bot instead of plugin.
>
> As for fail rate - I'm not sure it is working as you describe.
> I'm looking on my runAll [1]. There is
> `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in
> master branch [2].
>
> [1]
>
> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F4519%2Fhead=Latest
> [2]
>
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8==testDetails=5628470782089555961=TEST_STATUS_DESC_IgniteTests24Java8=__all_branches__=50
>
> 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov :
>
> > Hi Dmitrii,
> >
> > I'm not sure we're able to install Github apps to Apache mirrors.
> >
> > The simplest solution, what can be as efficient as a plugin, is fake
> MTCGA
> > bot account in Github, which will provide PR comments using Github
> program
> > interface. What do you think?
> >
> > A new test failure can be identified by the Ignite TC Bot by master
> recent
> > fail rate = 0.0%. The same rule can be applied to timed out suites.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov :
> >
> > > Hello, Igniters!
> > >
> > > I want to suggest improvement for TeamCity Helper [1] – we need an easy
> > way
> > > to get list of failed tests that don’t fall in the master branch. These
> > > tests should:
> > > * fail in the PR
> > > * not fail in the master
> > > * not be flaky.
> > >
> > > Also, I want to suggest to create a GitHub plugin, which will notify PR
> > if
> > > it has such tests. PR will have a marker, which allows/prohibits merge.
> > > This marker will be shown near PR conflicts.
> > >
> > > Allowing marker will be shown in case:
> > > * no new fails.
> > >
> > > Prohibiting marker will be shown in cases:
> > > * new fails – tests must be fixed.
> > > * new timed out test suite – suite should be restarted or tests must be
> > > fixed.
> > > * runAll wasn’t launched – tests must be launched.
> > >
> > > This will make test checks much faster and easier. Also, this will
> > decrease
> > > the number of merges with new failed tests made by inattention to the
> > > tests.
> > >
> > > Further, we can expand the plugin by adding new checks, showing PR
> > quality.
> > >
> > > [1] https://github.com/apache/ignite-teamcity-bot
> > >
> >
>

Storage Class Memory and Persistent Collections

[Steve Hostettler]

Hello,

 

Clearly Storage Class Memory represents a breakthrough for "in memory" grids
and some people already tried it on Ignite :
http://dmagda.blogspot.com/2017/10/3d-xpoint-outperforms-ssds-verified-on.ht
ml

I would like to know what is the position of the community towards  the
Persistent Collections (

https://github.com/pmem/pcj?utm_source=ISTV_medium=Video_campaign=IS
TV_2017,   https://pcollections.org/)? 
According to https://www.youtube.com/watch?v=ZuLAF3ppCzs , it could
massively reduce the marshalling/unmarshalling time. `

1)   Do you guys plan to test it and to support SCM such as 3DXPoint
natively?

2)   Would it mean that ignite would not serialize the objects off heap
anymore?

 

Hope it makes sense.

 

Best Regards

[GitHub] ignite pull request #4589: Replace TcpDiscoveryMulticastIpFinder with TcpDis...

[dmekhanikov]

GitHub user dmekhanikov opened a pull request:

https://github.com/apache/ignite/pull/4589

Replace TcpDiscoveryMulticastIpFinder with TcpDiscoveryVmIpFinder in tests

Created for testing

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-test-ip-finder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4589.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4589


commit 385950a6bb7e7e00bce784bab0e0708b8561c4d6
Author: Denis Mekhanikov 
Date:   2018-08-21T15:43:20Z

Search for all tests, that use multicast IP finder.

commit a12b8daaec037247620df79b9ec90b6e2023f783
Author: Denis Mekhanikov 
Date:   2018-08-21T16:27:52Z

Calculate multicast usage statistics per suite

commit e303f70d525882e5b97b27c0369603098c52a382
Author: Denis Mekhanikov 
Date:   2018-08-21T16:34:01Z

Use TcpDiscoveryVmIpFinder by default




---

Re: Workflow improvement

[Dmitrii Ryabov]

I think plugin will be more pretty looking, but comments can contain any
information, so they can be more usefull. I agree with your idea to create
bot instead of plugin.

As for fail rate - I'm not sure it is working as you describe.
I'm looking on my runAll [1]. There is
`IgniteCacheGroupsTest.testCacheApiTxReplicated`
in `Cache 3` suite with fail rate = 0.0%. But it is flaky and fails in
master branch [2].

[1]
https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F4519%2Fhead=Latest
[2]
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8==testDetails=5628470782089555961=TEST_STATUS_DESC_IgniteTests24Java8=__all_branches__=50

2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov :

> Hi Dmitrii,
>
> I'm not sure we're able to install Github apps to Apache mirrors.
>
> The simplest solution, what can be as efficient as a plugin, is fake MTCGA
> bot account in Github, which will provide PR comments using Github program
> interface. What do you think?
>
> A new test failure can be identified by the Ignite TC Bot by master recent
> fail rate = 0.0%. The same rule can be applied to timed out suites.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov :
>
> > Hello, Igniters!
> >
> > I want to suggest improvement for TeamCity Helper [1] – we need an easy
> way
> > to get list of failed tests that don’t fall in the master branch. These
> > tests should:
> > * fail in the PR
> > * not fail in the master
> > * not be flaky.
> >
> > Also, I want to suggest to create a GitHub plugin, which will notify PR
> if
> > it has such tests. PR will have a marker, which allows/prohibits merge.
> > This marker will be shown near PR conflicts.
> >
> > Allowing marker will be shown in case:
> > * no new fails.
> >
> > Prohibiting marker will be shown in cases:
> > * new fails – tests must be fixed.
> > * new timed out test suite – suite should be restarted or tests must be
> > fixed.
> > * runAll wasn’t launched – tests must be launched.
> >
> > This will make test checks much faster and easier. Also, this will
> decrease
> > the number of merges with new failed tests made by inattention to the
> > tests.
> >
> > Further, we can expand the plugin by adding new checks, showing PR
> quality.
> >
> > [1] https://github.com/apache/ignite-teamcity-bot
> >
>

[GitHub] ignite pull request #4588: IGNITE-9326 Fixed node failure processor invocati...

[agoncharuk]

GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/4588

IGNITE-9326 Fixed node failure processor invocation when EntryProcess…

…or result serialization throws an exception

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9326

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4588.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4588


commit df67c7826de826605baeb183c0007b548020756c
Author: Alexey Goncharuk 
Date:   2018-08-21T15:06:17Z

IGNITE-9326 Fixed node failure processor invocation when EntryProcessor 
result serialization throws an exception




---

Re: Table Names in Spark Catalog

[Stuart Macdonald]

Nikolay, Val,

The JDBC Spark datasource[1] -- as far as I can tell -- has no
ExternalCatalog implementation, it just uses the database specified in the
JDBC URL. So I don't believe there is any way to call listTables() or
listDatabases() for JDBC provider.

The Hive ExternalCatalog[2] makes the distinction between database and
table using the actual database and table mechanisms built into the
catalog, which is fine because Hive has the clear distinction and hierarchy
of databases and tables.

*However* Ignite already uses the "database" concept in the Ignite
ExternalCatalog[3] to mean the name of an Ignite instance. So in Ignite we
have instances containing schemas containing tables, and Spark only has the
concept of databases and tables so it seems like either we ignore one of
the three Ignite concepts or combine two of them into database or table.
The current implementation in the pull request combines Ignite schema and
table attributes into the Spark table attribute.

Stuart.

[1]
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
[2]
https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
[3]
https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/spark/sql/ignite/IgniteExternalCatalog.scala

On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov 
wrote:

> Hello, Stuart.
>
> Can you do some research and find out how schema is handled in Data Frames
> for a regular RDBMS such as Oracle, MySQL, etc?
>
> В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> > Stuart, Nikolay,
> >
> > I see that the 'Table' class (returned by listTables method) has a
> 'database' field. Can we use this one to report schema name?
> >
> > In any case, I think we should look into how this is done in data source
> implementations for other databases. Any relational database has a notion
> of schema, and I'm sure Spark integrations take this into account somehow.
> >
> > -Val
> >
> > On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov 
> wrote:
> > > Hello, Stuart.
> > >
> > > Personally, I think we should change current tables naming and return
> table in form of `schema.table`.
> > >
> > > Valentin, could you share your opinion?
> > >
> > >
> > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет:
> > > > Igniters,
> > > >
> > > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I are
> > > > discussing whether to introduce a change which may impact backwards
> > > > compatibility; Nikolay suggested we take the discussion to this list.
> > > >
> > > > Ignite implements a custom Spark catalog which provides an API by
> which
> > > > Spark users can list the tables which are available in Ignite which
> can be
> > > > queried via Spark SQL. Currently that table name list includes just
> the
> > > > names of the tables, but IGNITE-9228 is introducing a change which
> allows
> > > > optional prefixing of schema names to table names to disambiguate
> multiple
> > > > tables with the same name in different schemas. For the "list
> tables" API
> > > > we therefore have two options:
> > > >
> > > > 1. List the tables using both their table names and schema-qualified
> table
> > > > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are
> the same
> > > > underlying table. This retains backwards compatibility with users who
> > > > expect "myTable" to appear in the catalog.
> > > > 2. List the tables using only their schema-qualified names. This
> eliminates
> > > > duplication of names in the catalog but will potentially break
> > > > compatibility with users who expect the table name in the catalog.
> > > >
> > > > With either option we will allow for  Spark SQL SELECT statements to
> use
> > > > either table name or schema-qualified table names, this change would
> purely
> > > > impact the API which is used to list available tables.
> > > >
> > > > Any opinions would be welcome.
> > > >
> > > > Thanks,
> > > > Stuart.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9228
> > > > [2] https://github.com/apache/ignite/pull/4551
>

Re: Workflow improvement

[Dmitriy Pavlov]

Hi Dmitrii,

I'm not sure we're able to install Github apps to Apache mirrors.

The simplest solution, what can be as efficient as a plugin, is fake MTCGA
bot account in Github, which will provide PR comments using Github program
interface. What do you think?

A new test failure can be identified by the Ignite TC Bot by master recent
fail rate = 0.0%. The same rule can be applied to timed out suites.

Sincerely,
Dmitriy Pavlov

вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov :

> Hello, Igniters!
>
> I want to suggest improvement for TeamCity Helper [1] – we need an easy way
> to get list of failed tests that don’t fall in the master branch. These
> tests should:
> * fail in the PR
> * not fail in the master
> * not be flaky.
>
> Also, I want to suggest to create a GitHub plugin, which will notify PR if
> it has such tests. PR will have a marker, which allows/prohibits merge.
> This marker will be shown near PR conflicts.
>
> Allowing marker will be shown in case:
> * no new fails.
>
> Prohibiting marker will be shown in cases:
> * new fails – tests must be fixed.
> * new timed out test suite – suite should be restarted or tests must be
> fixed.
> * runAll wasn’t launched – tests must be launched.
>
> This will make test checks much faster and easier. Also, this will decrease
> the number of merges with new failed tests made by inattention to the
> tests.
>
> Further, we can expand the plugin by adding new checks, showing PR quality.
>
> [1] https://github.com/apache/ignite-teamcity-bot
>

[GitHub] ignite pull request #4581: IGNITE-9313 Update chief and user script processe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4581


---

[GitHub] ignite pull request #4585: IGNITE-9336: Remove random test dataset generatio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4585


---

[GitHub] ignite pull request #4582: IGNITE-6167: Ability to enabled TLS protocols and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4582


---

[GitHub] ignite pull request #4440: IGNITE-6167: Ability to enabled TLS protocols and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4440


---

Workflow improvement

[Dmitrii Ryabov]

Hello, Igniters!

I want to suggest improvement for TeamCity Helper [1] – we need an easy way
to get list of failed tests that don’t fall in the master branch. These
tests should:
* fail in the PR
* not fail in the master
* not be flaky.

Also, I want to suggest to create a GitHub plugin, which will notify PR if
it has such tests. PR will have a marker, which allows/prohibits merge.
This marker will be shown near PR conflicts.

Allowing marker will be shown in case:
* no new fails.

Prohibiting marker will be shown in cases:
* new fails – tests must be fixed.
* new timed out test suite – suite should be restarted or tests must be
fixed.
* runAll wasn’t launched – tests must be launched.

This will make test checks much faster and easier. Also, this will decrease
the number of merges with new failed tests made by inattention to the tests.

Further, we can expand the plugin by adding new checks, showing PR quality.

[1] https://github.com/apache/ignite-teamcity-bot

[GitHub] ignite pull request #4587: IGNITE-9270 Thread-per-partition during Tx commit

[Jokser]

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/4587

IGNITE-9270 Thread-per-partition during Tx commit



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9270

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4587


commit c396b4e7d682c45aa2ce70bbe1aa22a30a668a63
Author: Pavel Kovalenko 
Date:   2018-08-16T16:09:01Z

IGNITE-9270 WIP

commit 4c8ebbd5e8176c93583e1d6dd808608a0fa69c3d
Author: Pavel Kovalenko 
Date:   2018-08-16T17:23:43Z

IGNITE-9270 WIP

commit 1778b8e8b71d748f5305cc3f0e4a21c35c7b3067
Author: Pavel Kovalenko 
Date:   2018-08-16T17:41:21Z

IGNITE-9270 WIP

commit f6c7621704a63b411e71e87fd0d048ff1a80755e
Author: Pavel Kovalenko 
Date:   2018-08-21T13:10:04Z

IGNITE-9270 WIP




---

[jira] [Created] (IGNITE-9342) The same SQL request with multiple statements produces different result

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-9342:
---

 Summary: The same SQL request with multiple statements produces 
different result
 Key: IGNITE-9342
 URL: https://issues.apache.org/jira/browse/IGNITE-9342
 Project: Ignite
  Issue Type: Bug
  Components: jdbc, odbc, sql
Affects Versions: 2.6
Reporter: Igor Sapego


The bug affects ODBC and JDBC. Simply speaking, the following code:
{code:java}
public static void main(String[] args) throws Exception {
IgniteConfiguration cfg = new IgniteConfiguration();

cfg.setLocalHost("127.0.0.1");

try (Ignite ignite = Ignition.start(cfg)) {

try (Connection conn = 
DriverManager.getConnection("jdbc:ignite:thin://127.0.0.1/")) {

// Populate City table with PreparedStatement.
try (PreparedStatement stmt = conn.prepareStatement("SELECT 1; 
SELECT 2")) {
System.out.println("First run:");

executeAndFetch(stmt);

System.out.println();
System.out.println("Second run:");

executeAndFetch(stmt);
}
}
}
}

static void executeAndFetch(PreparedStatement stmt) throws SQLException {
ResultSet set = stmt.executeQuery();

System.out.println(">>>  First result set:");

while (set.next())
System.out.println(">>>" + set.getInt(1));

System.out.println(">>>  Next result set:");

boolean nextRsPresent = stmt.getMoreResults();

if (!nextRsPresent)
System.out.println(">>>Is not present");
else
{
set = stmt.getResultSet();

while (set.next())
System.out.println(">>>" + set.getInt(1));
}
}
{code}
Produces the following output:
{noformat}
First run:
>>>  First result set:
>>>1
>>>  Next result set:
>>>2

Second run:
>>>  First result set:
>>>1
>>>  Next result set:
>>>Is not present{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

[Sergey Kozlov]

Alex

In that case the data becomes in unpredictable state (mix of new and old
methods) and there's a chance that some partitions will be never touched
and never converted. It will force us always support old compression.
I suppose the explicit conversion that clearly say what is happening now
and may be warn that the db files should backed up before version upgrade.

Also I think AI 3.0 will have set of incompatible changes that give us a
chance to add an upgrade procedure where compression method update among
other stuff will take into account.

On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov 
wrote:

> Sergey, converting data also force us to introduce some flag to WAL
> segments/records or convert WAL segments too. WAL segments can be archived,
> they also can be compressed.
> IMO we better should not introduce any compatibility modes, left data as is
> and always just convert crc value returned by zip.CRC32 to old format (xor
> it) at runtime.
>
> 2018-08-21 0:12 GMT+03:00 Sergey Kozlov :
>
> > Dmitriy
> >
> > Due to significant improvement and to reduce the number supported
> > modes/options would be good to convert the data at the moment of upgrade.
> >
> > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > wrote:
> >
> > > Sergey, that was precisely my comment in the ticket:
> > >
> > > Can we add this option without breaking compatibility with previous
> > > page/storage formats? If not, then this should support both
> > implementation.
> > > The default should be the new fastest implementation, but if we
> encounter
> > > the older, slower one, then we should print out a warning in the log
> and
> > > automatically switch to the older implementation.
> > >
> > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov 
> > > wrote:
> > >
> > > > Hi Igniters
> > > >
> > > > I suppose that'll break compatibility for LFS (PDS).
> > > >
> > > > Do we plan to provide a migration guide w/o data loss for upgrade AI
> > 2.x
> > > to
> > > > 3.0?
> > > >
> > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > I commented in the ticket: https://issues.apache.org/
> > > > > jira/browse/IGNITE-9272
> > > > >
> > > > > It if can integrate it correctly, according to my comment, in 2.7
> > > > release,
> > > > > it would be great. Otherwise, let's plan this change for 3.0
> release.
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
> > > > > eduard.shangar...@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have checked the benchmark and it shows great performance boost
> > on
> > > my
> > > > > > laptop!
> > > > > >
> > > > > > +1 for this change.
> > > > > >
> > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov <
> > > dpavlov@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Evgeniy,
> > > > > > >
> > > > > > > Thank you. I see that the ticket is unassigned.
> > > > > > >
> > > > > > > Would you like to contribute PR to be macro-benchmarked with
> > > Ignite?
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > > > > > > :
> > > > > > >
> > > > > > > > I fill the ticket, bench code attached there.
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > >
> > > > > > > > >Has anyone else run the benchmark and reproduced the
> > performance
> > > > > > > > >difference?
> > > > > > > > >
> > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov <
> > > > > > dpavlov@gmail.com
> > > > > > > >
> > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > >> It depends.
> > > > > > > > >>
> > > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and
> page
> > > > store
> > > > > > > write
> > > > > > > > >> are mostly about IO speed.
> > > > > > > > >>
> > > > > > > > >> In the same time, it can make the huge impact on machines
> > with
> > > > > fast
> > > > > > IO
> > > > > > > > >> and
> > > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy
> and
> > > > Alexey
> > > > > > it
> > > > > > > > >> could
> > > > > > > > >> benefit performance because we save CPU. Later we can use
> > it's
> > > > > power
> > > > > > > in
> > > > > > > > a
> > > > > > > > >> more efficient manner (e.g. with compression).
> > > > > > > > >>
> > > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov <
> > > > yzhda...@apache.org
> > > > > >:
> > > > > > > > >>
> > > > > > > > >> > Guys, what time in % does crc calculation take in WAL
> > > logging
> > > > > > > process?
> > > > > > > > >> >
> > > > > > > > >> > --Yakov
> > > > > > > > >> >
> > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov <
> > > > > dpavlov@gmail.com
> > > > > > > >:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Alex, thank you for this idea.
> > > > > > > > >> > 

[jira] [Created] (IGNITE-9341) Notify metastorage listeners righ before start of discovery processor

[Ivan Rakov (JIRA)]

Ivan Rakov created IGNITE-9341:
--

 Summary: Notify metastorage listeners righ before start of 
discovery processor
 Key: IGNITE-9341
 URL: https://issues.apache.org/jira/browse/IGNITE-9341
 Project: Ignite
  Issue Type: Improvement
  Components: general
Reporter: Ivan Rakov
 Fix For: 2.7


onReadyForRead() is called only for inheritors of MetastorageLifecycleListener 
interface which are started prior to GridCacheProcessor. Listeners are notified 
at the moment of ReadOnlyMetastorage initialization, which in turn occur during 
GridCacheDatabaseSharedManager start.
We can split ReadOnlyMetastorage initialization and notification of listeners - 
this will allow all components to subscribe for read-only metastorage ready 
event.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4586: IGNITE-9279

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/4586

IGNITE-9279



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9279-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4586


commit 549f2fd5e5d053acfc015c408b5ea495cc5ef90d
Author: devozerov 
Date:   2018-08-21T09:51:30Z

Better schema counters.

commit 99887f6f70cf7a32051d880ab7780b3c44f7a03b
Author: devozerov 
Date:   2018-08-21T10:13:58Z

Retries.

commit 17f4b617f43e9989aa4b98d4345bc3b09a384466
Author: devozerov 
Date:   2018-08-21T10:25:04Z

Removed restriction.

commit bfface47fe8e791086b849457021aae7bc981920
Author: devozerov 
Date:   2018-08-21T10:26:05Z

Removed restrictions.




---

Re: QueryDetailMetrics for cache-less SQL queries

[Evgenii Zhuravlev]

Hi Alex,

I agree that we can't move all metrics to ignite.metrics() and SPI metrics
is a good example here. I propose to move at least dataRegionMetrics,
dataStorageMetrics to it, since the main API class is not a good place for
such things. If we will decide to choose option 1 from my first message,
then, I will also add QueryDetailMetrics for cacheless queries to this new
class.

пн, 20 авг. 2018 г. в 23:28, Alex Plehanov :

> Hi Evgeny,
>
> Do you propose to move into IgniteMetrics absolutely all Ignite metrics or
> just dataRegionMetrics, dataStorageMetrics and queryDetailMetrics? I think
> you can't move all metrics into one place. Pluggable components and
> different SPI implementations may have their own metric sets, and perhaps
> it's not such a good idea to try to fit them in one common fixed interface.
>
> 2018-08-20 18:14 GMT+03:00 Evgenii Zhuravlev :
>
> > As for now, metrics are accumulated for cache only when the query is run
> > directly over this cache, for example, using ignite.cache("Some
> > cache").sqlFieldsQuery("select ... from .."). When a query is started
> using
> > other APIs, it doesn't detect cache, to which this table belongs and
> > doesn't save any metrics.
> >
> >
> >
> > 2018-08-17 16:44 GMT+03:00 Vladimir Ozerov :
> >
> > > Query is not executed on specific cache. It is executed on many caches.
> > >
> > > On Fri, Aug 17, 2018 at 6:10 AM Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > >
> > > > But internally the SQL query still runs on some cache, no? What
> happens
> > > to
> > > > the metrics accumulated on that cache?
> > > >
> > > > D.
> > > >
> > > > On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov 
> > > > wrote:
> > > >
> > > > > Dima,
> > > > >
> > > > > "cache-less" means that SQL executed directly on SQL engine.
> > > > >
> > > > > In previous version of Ignite we execute queries via cache:
> > > > >
> > > > > ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
> > > > >
> > > > > In current Ignite we can execute query directly without using cache
> > as
> > > > > "gateway".
> > > > >
> > > > > And if we execute query directly, metrics not update.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Evgeny, what is a "cache-less" SQL query?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > > > > > e.zhuravlev...@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi Igniters,
> > > > > > >
> > > > > > > I've started to work on adding QueryDetailMetrics for
> cache-less
> > > SQL
> > > > > > > queries(issue
> https://issues.apache.org/jira/browse/IGNITE-6677)
> > > and
> > > > > > found
> > > > > > > that it's required to change API. I don't think that adding
> > methods
> > > > > like
> > > > > > > queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache
> to
> > > > > Ignite
> > > > > > > class is a good idea. So, I see 2 possible solutions here:
> > > > > > >
> > > > > > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> > > > > > > Ignite(like dataRegionMetrics and dataStorageMetrics) and add a
> > new
> > > > > > > metric "queryDetailMetrics" to it. Of course, old methods will
> be
> > > > > > > deprecated.
> > > > > > >
> > > > > > > 2. Finally create Ignite.sql() API, which was already discussed
> > > here:
> > > > > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > > > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > > > > > > and place "queryDetailMetrics" metric there. Here is the ticket
> > for
> > > > > this
> > > > > > > change: https://issues.apache.org/jira/browse/IGNITE-4701
> > > > > > >
> > > > > > > Personally, I think that the second solution looks better in
> this
> > > > case,
> > > > > > > however, moving dataRegionMetrics and dataStorageMetrics to
> > > > > > > ignite.matrics() is still a good idea - IMO, Ignite class is
> not
> > > the
> > > > > > right
> > > > > > > place for them - we shouldn't change our main API class so
> often.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Evgenii
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Alexey Kuznetsov
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[GitHub] ignite pull request #4585: IGNITE-9336: Remove random test dataset generatio...

[zaleslaw]

GitHub user zaleslaw opened a pull request:

https://github.com/apache/ignite/pull/4585

IGNITE-9336: Remove random test dataset generation

1. Remove ThreadLocalRandom usages in test dataset data generation
2. Refactor Trainer tests
3. Propose a few static datasets generated manually according testing tasks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9336

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4585


commit 1f456c55016265242f606f9711a700069174bbc5
Author: zaleslaw 
Date:   2018-08-21T07:50:55Z

Rename method

commit 440a63c0d2737503425bbfde306aa7c32b4830c2
Author: zaleslaw 
Date:   2018-08-21T09:14:17Z

Fixed LogRegMultiClassTrainerTest

commit e8dcf0ce72c28a8c4273e9a7371d4d63d124857f
Author: zaleslaw 
Date:   2018-08-21T09:53:50Z

Extract Trainer test and partitions divisions into the separate test




---

[GitHub] ignite pull request #4138: IGNITE-7339 RENTING partition is not evicted afte...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4138


---

Re: Wrong off-heap size is reported for a node

[Pavel Pereslegin]

Hello, Igniters.

I assigned ticket [1] created by Denis and want to clarify how to log
committed size.
The metric offHeapSize (in DataRegionMetricsImpl) is always
calculated, but getOffHeapSize returns zero if memory metrics are
disabled for this data region.

So I see the following options:
1. Modify method getOffHeapSize so that it always returns actual value
offHeapSize.
2. Add another offHeapSize() method.
3. Output to log max size instead of committed (change "comm" to "max"
in log output).
4. Don't bother about disabling metrics and output to log value
returned by getOffHeapSize.

Any thoughts?

[1] https://issues.apache.org/jira/browse/IGNITE-9305
сб, 18 авг. 2018 г. в 3:17, Denis Magda :
>
> Vova, the things are even simpler - we have this
>
> ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the
> number equal/comparabel to pageNumber X pageSize.
>
>
> Igniters, if you believe that we need to do more work here then let's
> do it iteratively. Let's fix the off-heap occupied size the way above
> (just print out getPhysicalMemorySize() for every data region). Then
> do the rest. This needs to be fixed in 2.7.
>
>
> --
>
> Denis
>
>
> On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov 
> wrote:
>
> > Folks,
> >
> > We already have this:
> > >>> PageMemory [pages=6997377]
> >
> > Then we can multiply it by page size and get occupied memory. Am I wrong?
> >
> > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov 
> > wrote:
> >
> > > Hi Maxim,
> > >
> > > thank you for stepping in and for finding these issues. Yes, these
> > tickets
> > > are correct.
> > >
> > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to
> > unassigned
> > > if someone would like to implement this change. I will not have enough
> > time
> > > to complete it in 1 month (before 2.7 release).
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov :
> > >
> > > > Igniters,
> > > >
> > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap
> > > memory
> > > > metrics
> > > > to new page memory semantics` and related previous disscustions to it
> > > [4].
> > > >
> > > > Also we have some additional improvements to CacheMetrics:
> > > > IGNITE-5490 [2] - `Implement replacement for obsolete
> > > > CacheMetrics#getOffHeapAllocatedSize`
> > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and
> > > fixes`
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490
> > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765
> > > > [4]
> > > >
> > > >
> > >
> > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html
> > > >
> > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov 
> > > wrote:
> > > >
> > > > > Hi Igniters,
> > > > >
> > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7.
> > > > >
> > > > > Offheap size is not reported by VM (it returns -1). To implement it
> > we
> > > > need
> > > > > totally migrate off-heap memory metrics to durable memory data.
> > > > >
> > > > > I think this issue was reported and I'll find the duplicate.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda :
> > > > >
> > > > > > Yes, it was at the end of my wordy email :)
> > > > > > https://issues.apache.org/jira/browse/IGNITE-9305
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Is there a blocker ticket for 2.7?
> > > > > > >
> > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda 
> > wrote:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > Was troubleshooting an Ignite deployment today and couldn't
> > find
> > > > out
> > > > > > from
> > > > > > > > the logs what was the actual off-heap space used.
> > > > > > > >
> > > > > > > > Those were the given memory resoures (Ignite 2.6):
> > > > > > > >
> > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager]
> > > > Topology
> > > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64,
> > *offheap=30.0GB*,
> > > > > > > > heap=24.0GB]
> > > > > > > >
> > > > > > > > And that weird stuff was reported by the node (pay attention to
> > > the
> > > > > > last
> > > > > > > > line):
> > > > > > > >
> > > > > > > > [2018-08-16 15:45:50,211][INFO
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
> > > > > > > > Metrics for local node (to disable set 'metricsLogFrequency' to
> > > 0)
> > > > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017,
> > > > > > uptime=00:38:00.257]
> > > > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64]
> > > > > > > > ^-- CPU [cur=0.03%, 

Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.

[Alex Plehanov]

Sergey, converting data also force us to introduce some flag to WAL
segments/records or convert WAL segments too. WAL segments can be archived,
they also can be compressed.
IMO we better should not introduce any compatibility modes, left data as is
and always just convert crc value returned by zip.CRC32 to old format (xor
it) at runtime.

2018-08-21 0:12 GMT+03:00 Sergey Kozlov :

> Dmitriy
>
> Due to significant improvement and to reduce the number supported
> modes/options would be good to convert the data at the moment of upgrade.
>
> On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan  >
> wrote:
>
> > Sergey, that was precisely my comment in the ticket:
> >
> > Can we add this option without breaking compatibility with previous
> > page/storage formats? If not, then this should support both
> implementation.
> > The default should be the new fastest implementation, but if we encounter
> > the older, slower one, then we should print out a warning in the log and
> > automatically switch to the older implementation.
> >
> > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov 
> > wrote:
> >
> > > Hi Igniters
> > >
> > > I suppose that'll break compatibility for LFS (PDS).
> > >
> > > Do we plan to provide a migration guide w/o data loss for upgrade AI
> 2.x
> > to
> > > 3.0?
> > >
> > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > wrote:
> > >
> > > > I commented in the ticket: https://issues.apache.org/
> > > > jira/browse/IGNITE-9272
> > > >
> > > > It if can integrate it correctly, according to my comment, in 2.7
> > > release,
> > > > it would be great. Otherwise, let's plan this change for 3.0 release.
> > > >
> > > > D.
> > > >
> > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev <
> > > > eduard.shangar...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have checked the benchmark and it shows great performance boost
> on
> > my
> > > > > laptop!
> > > > >
> > > > > +1 for this change.
> > > > >
> > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov <
> > dpavlov@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Evgeniy,
> > > > > >
> > > > > > Thank you. I see that the ticket is unassigned.
> > > > > >
> > > > > > Would you like to contribute PR to be macro-benchmarked with
> > Ignite?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский
> > > > > > :
> > > > > >
> > > > > > > I fill the ticket, bench code attached there.
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272
> > > > > > > Thanks!
> > > > > > >
> > > > > > >
> > > > > > > >Has anyone else run the benchmark and reproduced the
> performance
> > > > > > > >difference?
> > > > > > > >
> > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov <
> > > > > dpavlov@gmail.com
> > > > > > >
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> It depends.
> > > > > > > >>
> > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page
> > > store
> > > > > > write
> > > > > > > >> are mostly about IO speed.
> > > > > > > >>
> > > > > > > >> In the same time, it can make the huge impact on machines
> with
> > > > fast
> > > > > IO
> > > > > > > >> and
> > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and
> > > Alexey
> > > > > it
> > > > > > > >> could
> > > > > > > >> benefit performance because we save CPU. Later we can use
> it's
> > > > power
> > > > > > in
> > > > > > > a
> > > > > > > >> more efficient manner (e.g. with compression).
> > > > > > > >>
> > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov <
> > > yzhda...@apache.org
> > > > >:
> > > > > > > >>
> > > > > > > >> > Guys, what time in % does crc calculation take in WAL
> > logging
> > > > > > process?
> > > > > > > >> >
> > > > > > > >> > --Yakov
> > > > > > > >> >
> > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov <
> > > > dpavlov@gmail.com
> > > > > > >:
> > > > > > > >> >
> > > > > > > >> > > Hi Alex, thank you for this idea.
> > > > > > > >> > >
> > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with
> > > > bypassing
> > > > > > > >> > > implementation differences to keep compatibility?
> > > > > > > >> > >
> > > > > > > >> > > Sincerely,
> > > > > > > >> > > Dmitriy Pavlov
> > > > > > > >> > >
> > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov <
> > > > > > > plehanov.a...@gmail.com >:
> > > > > > > >> > >
> > > > > > > >> > > > Hello, Igniters!
> > > > > > > >> > > >
> > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic,
> > > > > moreover
> > > > > > > new
> > > > > > > >> > > > "update" method, which use ByteBuffer was introduced.
> > > Since
> > > > we
> > > > > > > >> moved
> > > > > > > >> to
> > > > > > > >> > > > java8, perhaps we really can get performance boost by
> > > using
> > > > > > > >> standard
> > > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32.
> > > 

[jira] [Created] (IGNITE-9339) Web console: form-field-size improvements

[Ilya Borisov (JIRA)]

Ilya Borisov created IGNITE-9339:


 Summary: Web console: form-field-size improvements
 Key: IGNITE-9339
 URL: https://issues.apache.org/jira/browse/IGNITE-9339
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Ilya Borisov
Assignee: Dmitriy Shabalin


I think there some changes due for {{form-field-size}} component:
1. [~pkonstantinov] expressed his confusion multiple times about the fact that 
{{form-field-size}} converts visible value when user changes scale. He suggests 
to keep the view value the same and only scale the model value.
2. It wasn't the case at the moment {{form-field-size}} was added, but now some 
form fields use different default scale (i.e. it's bytes vs megabytes). 
[~Dmitriyff] introduced redundant "time" scale in IGNITE-9286, surely we can 
figure out a better component API for cases like this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4584: IGNITE-2.4.8.b5

[dmekhanikov]

GitHub user dmekhanikov opened a pull request:

https://github.com/apache/ignite/pull/4584

IGNITE-2.4.8.b5

Created for testing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.8.b5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4584.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4584


commit 8994f847d7f5f15db73706d9210cdccb1cf3fb26
Author: devozerov 
Date:   2018-02-12T13:34:24Z

IGNITE-7003: Fixed faulty test 
WalModeChangeAdvancedSelfTest.testJoinCoordinator.

commit b142712264007d7397d1594541681a4a7e3d4b93
Author: Igor Sapego 
Date:   2018-02-26T12:02:07Z

IGNITE-7362: Fixed PDO ODBC integration issue

commit a2b2aee52cc65d01f2ecaf9462adc4bd368438ea
Author: Igor Sapego 
Date:   2018-02-28T10:23:12Z

IGNITE-7763: Fixed 32bit tests configurations to prevent OOM

This closes #3557

commit 652f3c4cdbaad40f5de25b06f0c13710aa7f2fd9
Author: Pavel Kuznetsov 
Date:   2018-03-13T12:46:36Z

IGNITE-7531: Data load benchmarks. This closes #3571.

commit 9337a53d9fcd62af87f6760080d350b43e275105
Author: tledkov-gridgain 
Date:   2018-03-16T11:38:38Z

IGNITE-7879: Fixed splitter logic for DISTINCT with subqueries. This closes 
#3634.

commit 7bec8b13cb373002d2a6b1b268d410338259bac2
Author: Igor Sapego 
Date:   2018-03-19T11:17:33Z

IGNITE-7811: Implemented connection failover for ODBC

This closes #3643

commit e512e5e0a2602df0ecfb69b2b5efabce836b04db
Author: Igor Sapego 
Date:   2018-03-20T10:37:02Z

IGNITE-7888: Implemented login timeout for ODBC.

This closes #3657

commit 211fca3a55e84b78ff0c1af04d91e25d6fc846c4
Author: devozerov 
Date:   2018-03-20T11:13:46Z

IGNITE-7984: SQL: improved deadlock handling in DML. This closes #3655.

commit bcd2888d27afe65f1a060e35b99a05ea420979c7
Author: Roman Guseinov 
Date:   2018-02-16T09:57:26Z

IGNITE-7192: Implemented JDBC support FQDN to multiple IPs

This closes #3439

commit d2659d0ec9f6e1a0b905fc7bf23b65fd5522c80a
Author: Alexander Paschenko 
Date:   2018-03-14T09:23:37Z

IGNITE-7253: JDBC thin driver: implemented streaming. This closes #3499. 
This closes #3591.

commit bc9018ef8b116f81b8e06d2ff7651ba2b6c7beae
Author: tledkov-gridgain 
Date:   2018-03-19T08:01:26Z

IGNITE-7029: JDBC thin driver now supports multiple IP addresses with 
transparent failover. This closes #3585.

commit 587022862fd5bdbb076ab6207ae6fd9bc7583c13
Author: Sergey Chugunov 
Date:   2018-03-16T16:24:17Z

IGNITE-7964 rmvId is stored to MetaStorage metapage during operations - 
Fixes #3645.

commit 006ef4d15e4faedb6dfce6ce9637602055b97293
Author: tledkov-gridgain 
Date:   2018-03-22T11:47:06Z

IGNITE-7436: Simple username/password authentication. This closes #3483.

commit 1c7f59c90514670e802d9d07544b00b7562fe6d2
Author: Pavel Tupitsyn 
Date:   2018-01-30T09:48:16Z

.NET: Fix build status icon in README

commit 162df61b305fccfc55e186d07351727f35b55179
Author: Pavel Tupitsyn 
Date:   2018-02-01T11:53:16Z

IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy

This closes #3457

commit 9a0328ebbc9d52f8e96898a384fa45743d2efa5b
Author: Pavel Tupitsyn 
Date:   2018-02-02T12:01:27Z

.NET: Update README regarding C++ interop and thin client

commit b804cfea51c87724b45b40de4fd58d300c049be1
Author: Pavel Tupitsyn 
Date:   2018-01-31T09:39:19Z

.NET: Suppress API parity check for SSL in ClientConnectorConfiguration

commit 6f8014de7250c4c0d87cbc8764afae4a225f654b
Author: apopov 
Date:   2018-02-13T10:13:15Z

IGNITE-3111 .NET can be now configured SSL without Spring

This closes #3498

commit 5131bcd71ce787cf2c61bf98446f5ec0a616ab1c
Author: Pavel Tupitsin 
Date:   2018-02-16T20:36:01Z

IGNITE-3111 .NET: Configure SSL without Spring - cleanup

* Remove unused members from ISslContextFactory
* Fix namespaces
* Remove unused files
* Cleanup tests

commit 4ac4645dcf6e85883ce0de46ba1253ba8135804e
Author: Pavel Tupitsyn 
Date:   2018-02-18T20:22:27Z

.NET: Fix LoadDllTest, IgniteStartStopTest

commit 8709785814a432f981c30274a55e2ef730667421
Author: Pavel Tupitsyn 
Date:   2018-02-18T20:27:29Z

.NET: Fix SslConfigurationTest

commit d2a1136fd725b0e09e1e655152509aeae755c368
Author: Pavel Tupitsyn 
Date:   2018-02-26T22:15:11Z

IGNITE-7329 .NET: Thin client: SSL connection implemented

This closes #3447

commit c2a5c5df71197418bd126e87f62e66681ac25b76
Author: Pavel Tupitsin 
Date:   2018-03-05T16:07:32Z

IGNITE-7878 .NET: Ignore TestStartDefault

commit eec0a6714dd9f59edb637c8c068205843f5b9af6
Author: Alexey Popov 
Date:   2018-03-06T17:47:18Z

IGNITE-7889: .NET: LINQ: Fix GroupBy with Where

This closes #3608

commit 74fbd41f7f4e0b129797297c603c471ea558f2f6
Author: 

Re: Table Names in Spark Catalog

[Nikolay Izhikov]

Hello, Stuart.

Can you do some research and find out how schema is handled in Data Frames for 
a regular RDBMS such as Oracle, MySQL, etc?

В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> Stuart, Nikolay,
> 
> I see that the 'Table' class (returned by listTables method) has a 'database' 
> field. Can we use this one to report schema name?
> 
> In any case, I think we should look into how this is done in data source 
> implementations for other databases. Any relational database has a notion of 
> schema, and I'm sure Spark integrations take this into account somehow.
> 
> -Val
> 
> On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov  wrote:
> > Hello, Stuart.
> > 
> > Personally, I think we should change current tables naming and return table 
> > in form of `schema.table`.
> > 
> > Valentin, could you share your opinion?
> > 
> > 
> > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет:
> > > Igniters,
> > > 
> > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I are
> > > discussing whether to introduce a change which may impact backwards
> > > compatibility; Nikolay suggested we take the discussion to this list.
> > > 
> > > Ignite implements a custom Spark catalog which provides an API by which
> > > Spark users can list the tables which are available in Ignite which can be
> > > queried via Spark SQL. Currently that table name list includes just the
> > > names of the tables, but IGNITE-9228 is introducing a change which allows
> > > optional prefixing of schema names to table names to disambiguate multiple
> > > tables with the same name in different schemas. For the "list tables" API
> > > we therefore have two options:
> > > 
> > > 1. List the tables using both their table names and schema-qualified table
> > > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are the 
> > > same
> > > underlying table. This retains backwards compatibility with users who
> > > expect "myTable" to appear in the catalog.
> > > 2. List the tables using only their schema-qualified names. This 
> > > eliminates
> > > duplication of names in the catalog but will potentially break
> > > compatibility with users who expect the table name in the catalog.
> > > 
> > > With either option we will allow for  Spark SQL SELECT statements to use
> > > either table name or schema-qualified table names, this change would 
> > > purely
> > > impact the API which is used to list available tables.
> > > 
> > > Any opinions would be welcome.
> > > 
> > > Thanks,
> > > Stuart.
> > > 
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9228
> > > [2] https://github.com/apache/ignite/pull/4551

signature.asc
Description: This is a digitally signed message part

[jira] [Created] (IGNITE-9338) ML TF integration: tf cluster can't connect after killing first node with default port 10800

[Stepan Pilschikov (JIRA)]

Stepan Pilschikov created IGNITE-9338:
-

 Summary: ML TF integration: tf cluster can't connect after killing 
first node with default port 10800
 Key: IGNITE-9338
 URL: https://issues.apache.org/jira/browse/IGNITE-9338
 Project: Ignite
  Issue Type: Bug
  Components: ml
Reporter: Stepan Pilschikov


Case: 
- Run cluster with 3 node on 1 host
- Filling caches with data
- Running python script
- Killing lead node with port 10800 with chief + user_script processes
Expect:
- chief and user_script restarted on other node
- script rerun
Actual:
- chief and user_secript restarted on other node but started to crash and run 
again because can't connect to default 10800 port




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Unknown known issue on cache rebalancing delayed

[Maxim Muzafarov]

Roman,

I worked recently on rebalance improvements and haven't found any problems
with delayed cache rebalacne.
Agree with you - let's uncomment this and remove scary comment. Will you
create a ticket for it?

In case of any problems we can easily detec deadlock with newly configured
`FailureHandler`.

On Tue, 21 Aug 2018 at 03:49 Roman Shtykh  wrote:

> Hi Maxim,
>
> I have some issues with a cluster with rebalance delay enabled, but need
> to check more -- if I find it's related I'll share.
> Just wanted to make sure it's not an issue anymore from someone working on
> rebalancing. We should remove that comment then, it looks scary :)
>
> --
> Roman Shtykh
>
>
> On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov <
> maxmu...@gmail.com> wrote:
>
>
> Hello Roman,
>
> Did you faced with real issue of delayed rebalance or it's just only for
> your personal interest?
> If yes, please, share details and we will try to help you.
>
> As for this comment I don't think he is actual. That change was in 2015.
> Much has changed
> within rebalance process since that time. I've uncommented it and
> rechecked with that
> cache configuration and haven't seen any failed tests or issues.
>
> Probably, that problem was about cache in SYNC mode does not start util it
> loads all data
> from other nodes. But currently delayed rebalance works the same way as
> IgniteCache#rebalance(),
> so you can `setRebalanceDelay` to `-1` and call it manually to check.
>
> On Mon, 20 Aug 2018 at 11:19 Roman Shtykh 
> wrote:
>
> Igniters,
> I have found "Known issue, possible deadlock in case of low priority cache
> rebalancing delayed" comment in
> GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please explain
> when using rebalance delay can be an issue and why?
>
> -- Roman
>
> --
> --
> Maxim Muzafarov
>
-- 
--
Maxim Muzafarov

[jira] [Created] (IGNITE-9337) VisorNodeDataCollectorTask may produce HUGE result in case of large cluster with many caches

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-9337:


 Summary: VisorNodeDataCollectorTask may produce HUGE result in 
case of large cluster with many caches
 Key: IGNITE-9337
 URL: https://issues.apache.org/jira/browse/IGNITE-9337
 Project: Ignite
  Issue Type: Task
  Components: wizards
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.7


If cluster consists of 200 nodes and has 8000 caches (real deployment) it will 
produce

200 x 8000 =  1 600 000 instances of VisorCache (aprox 300b) on single 
invocation.

It will produce LARGE amount of data.

We can handle this by adding optional group name that should be collected 
instead of collecting all caches.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9336) [ML] ANN/SVM Trainer tests produce unpredictable results due to random data generation

[Aleksey Zinoviev (JIRA)]

Aleksey Zinoviev created IGNITE-9336:


 Summary: [ML] ANN/SVM Trainer tests produce unpredictable results 
due to random data generation
 Key: IGNITE-9336
 URL: https://issues.apache.org/jira/browse/IGNITE-9336
 Project: Ignite
  Issue Type: Bug
  Components: ml
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev


Remove random data generation and add static dataset into tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.7 release

[Nikolay Izhikov]

Hello, Dmitriy.

I think Transparent Data Encryption will be available in 2.7

В Пн, 20/08/2018 в 13:20 -0700, Dmitriy Setrakyan пишет:
> Hi Nikolay,
> 
> Thanks for being the release manager!
> 
> I am getting a bit lost in all these tickets. Can we specify some
> high-level tickets, that are not plain bug fixes, which will be interesting
> for the community to notice?
> 
> For example, here are some significant tasks that the community is either
> working on or has been working on:
> 
> - Node.JS client
> - Python client
> - Transactional SQL (MVCC)
> - service grid stabilization
> - SQL memory utilization improvements
> - more?
> 
> Can you please solicit status from the community for these tasks?
> 
> D.
> 
> On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov 
> wrote:
> 
> > Hello, Igniters.
> > 
> > I'm release manager of Apache Ignite 2.7.
> > 
> > It's time to start discussion of release. [1]
> > 
> > Current code freeze date is September, 30.
> > If you have any objections - please, responsd to this thread.
> > 
> > [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7

signature.asc
Description: This is a digitally signed message part