Page Locking vs Entry-level Locking

[John Wilson]

Hi,

Ignite documentation talks about entry-level locks and the page structure
has a LOCK_OFFSET that I assume is used to store tag info. I have these
questions.

   1. Does Ignite use a lock-free implementation to lock pages and/or
   entries?
   2. When is a page locked and when is it released?
   3. When an entry is inserted/modified in a page, is the page locked
   (forbidding other threads from inserting entries in the page)? or only the
   entry's offset is locked? (allowing other threads to insert/modify other
   items)

Thanks!

Optimistic Locking and the Prepare Phase

[John Wilson]

Hi,

The design doc below states:

*" In optimistic transactions, locks are acquired on primary nodes during
the "prepare" phase, then promoted to backup nodes and released once the
transaction is committed. Depending on an isolation level, if Ignite
detects that a version of an entry has been changed since the time it was
requested by a transaction, then the transaction will fail at the "prepare"
phase and it will be up to an application to decide whether to restart the
transaction or not."*

Two questions:


   1. If locks are acquired during the prepare phase, why do we state that
   lock acquisition for optimistic locking is delayed (as compared against
   pessimistic locking)?
   2. If "*ignite detects the version has changed since last request by
   transaction, it will fail at prepare phase*". Very confusing. What is
   the last request? I thought the "last request" means the "prepare" phase
   and if so why we say it may fail during prepare phase?

The graphic make sense to me - i.e. locks for optimistic locking are
acquired on the commit phase and not on the prepare phase.

https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+Architecture

Please help clarify.

Thanks.

Re: What happens if Primary Node fails during the Commit Phase

[John Wilson]

I got the answer for #3 here
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pages.
I will post the remaining questions in a separate thread.

On Mon, Feb 12, 2018 at 8:03 PM, John Wilson 
wrote:

> You're always helpful Val. Thanks!
>
>
> I have a question regarding Optimistic Locking
>
>
>1. The documentation here, https://cwiki.apache.
>org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+
>Architecture
>
> ,
>states that locks, for optimistic locking, are acquired during the
>"prepare" phase. But the graphic depicted there and the tutorial here,
>https://www.gridgain.com/resources/blog/apache-ignite-transactions-
>architecture-concurrency-modes-and-isolation-levels
>
> ,
>clearly indicate that locks are acquired during the commit phase; with a
>version information passed along with the key by the coordinator to the
>primary nodes. Can you please explain the discrepancy?
>
> And two questions regarding pages and page locking?
>
>1. Does Ignite use a lock-free algorithm for its B+ tree structure?
>Looking at the source code and the use of a tag field to avoid the ABA
>problem suggests that.
>2. Ignite documentation talks about entry-level locks and page locks.
>When exactly is a page locked and released? Also, when an entry is
>inserted/modified in a page, is the page locked, forbidding other threads
>from inserting other entries in the page, or only the entry's offset is
>locked allowing other threads to insert other entries in the page?
>3. What is the the difference between a directCount and indirectCount
>for a page?
>
> Thanks
>
> On Mon, Feb 12, 2018 at 7:33 PM, vkulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Hi John,
>>
>> 1. True.
>>
>> 2. The blog actually provides the answer:
>>
>> When the Backup Nodes detect the failure, they will notify the Transaction
>> coordinator that they committed the transaction successfully. In this
>> scenario, there is no data loss because the data are backed up and can
>> still
>> be accessed and used by applications.
>>
>> In other words, if primary node fails, backups will not wait for a
>> message,
>> but instead will commit right away and send an ack to the coordinator.
>> Once
>> coordinator gets all required acs, transaction completes.
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Re: IgniteSet implementation: changes required

[Andrey Kuznetsov]

Indeed, all sets, regardless of whether they collocated or not, share
single cache, and also use onheap data structures irresistable to
checkpointing/recovery.

2018-02-13 2:14 GMT+03:00 Dmitriy Setrakyan :

> On Fri, Feb 9, 2018 at 6:26 PM, Andrey Kuznetsov 
> wrote:
>
> > Hi all,
> >
> > Current set implementation has significant flaw: all set data are
> > duplicated in onheap maps on _every_ node in order to make iterator() and
> > size(). For me it looks like simple yet ineffective implementation.
> > Currently, these maps are damaged by checkpointing/recovery, and we could
> > patch them somehow. Another future change to Ignite caches can damage
> them
> > again. This looks fragile when datastructure is not entirely backed by
> > caches. Pavel's proposal seems to be a reliable solution for
> non-collocated
> > sets.
> >
>
> I would agree, but I was under an impression that non-collocated sets are
> already implemented this way. Am I wrong?
>



-- 
Best regards,
  Andrey Kuznetsov.

Re: What happens if Primary Node fails during the Commit Phase

[John Wilson]

You're always helpful Val. Thanks!


I have a question regarding Optimistic Locking


   1. The documentation here,
   
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+Architecture,
   states that locks, for optimistic locking, are acquired during the
   "prepare" phase. But the graphic depicted there and the tutorial here,
   
https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-concurrency-modes-and-isolation-levels,
   clearly indicate that locks are acquired during the commit phase; with a
   version information passed along with the key by the coordinator to the
   primary nodes. Can you please explain the discrepancy?

And two questions regarding pages and page locking?

   1. Does Ignite use a lock-free algorithm for its B+ tree structure?
   Looking at the source code and the use of a tag field to avoid the ABA
   problem suggests that.
   2. Ignite documentation talks about entry-level locks and page locks.
   When exactly is a page locked and released? Also, when an entry is
   inserted/modified in a page, is the page locked, forbidding other threads
   from inserting other entries in the page, or only the entry's offset is
   locked allowing other threads to insert other entries in the page?
   3. What is the the difference between a directCount and indirectCount
   for a page?

Thanks

On Mon, Feb 12, 2018 at 7:33 PM, vkulichenko 
wrote:

> Hi John,
>
> 1. True.
>
> 2. The blog actually provides the answer:
>
> When the Backup Nodes detect the failure, they will notify the Transaction
> coordinator that they committed the transaction successfully. In this
> scenario, there is no data loss because the data are backed up and can
> still
> be accessed and used by applications.
>
> In other words, if primary node fails, backups will not wait for a message,
> but instead will commit right away and send an ack to the coordinator. Once
> coordinator gets all required acs, transaction completes.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Username/password authentication for thin clients

[Denis Magda]

Taras, 

From what I see this license complies with Apache 2.0 and allows to do 
everything we want with the code. Just the license notice has to be preserved. 
So, I would add the source file to Ignite.

—
Denis

> On Feb 12, 2018, at 7:57 AM, Taras Ledkov  wrote:
> 
> Colleagues, Denis,
> 
> It will be great to use bcrypt for password hashing in Ignite.
> Could you suggest the right way to use bcrypt:
> 1. add 'jbcrypt' maven dependency;
> 2. include the single 'BCrypt.java' file to our project [1].
> 
> Does the license allow to include 'BCrypt.java' ?
> 
> [1]. 
> https://github.com/jeremyh/jBCrypt/blob/master/src/main/java/org/mindrot/BCrypt.java
>  
> 
> 
> On 18.01.2018 13:50, Taras Ledkov wrote:
>> Password hashing algorithms of the popular vendors:
>> 
>> mysql: SHA-265, old-native-hash
>> postgres: MD5, DES, Extended DES, Blowfish-based
>> oracle: SHA-1
>> 
>> Some about "comparison" SHA-2 vs bcrypt [1]:
>> 
>> > SHA-512 is a cryptographic hash while bcrypt is a password hash or PBKDF 
>> > (password based key derivation function).
>> 
>> > SHA-512 has been designed to be fast. You don't want any delays when 
>> > validating a signature, for instance. 
>> > There is no reason for generic cryptographic hashes to be slow.
>> 
>> > bcrypt on the other hand is a password hash that performs key 
>> > strengthening on the input. 
>> > Basically it does this by slowing down the calculation so that attackers 
>> > will have to spend 
>> > more resources to find the input by brute forcing or dictionary attacks. 
>> > The idea is that although the legit users - you in this case - will also 
>> > be slowed down, 
>> > they are only slowed down once per password. However the attackers are 
>> > slowed down for each try. 
>> > The legit user is of course much more likely to input the right password 
>> > first.
>> 
>> > Furthermore bcrypt also contains a salt as input, which can be used to 
>> > avert rainbow table attacks.
>> 
>> Conclusion: bcrypt can provide more security but the popular vendors use SHA 
>> and even MD5 by default.
>> 
>> [1]. 
>> https://crypto.stackexchange.com/questions/46550/benchmark-differences-between-sha-512-and-bcrypt
>>  
>> 
>> 
>> On 18.01.2018 9:29, Vladimir Ozerov wrote:
>>> Taras,
>>> 
>>> I think we need a comparison of available options and (possibly) analysis
>>> what other vendors use.
>>> 
>>> On Tue, Jan 16, 2018 at 3:56 PM, Taras Ledkov  
>>>  wrote:
>>> 
 What do you think about usage bcrypt [1], [2] to store encrypted password?
 
 [1] https://stackoverflow.com/questions/1561174/sha512-vs-blowfi 
 
 sh-and-bcrypt
 [2] https://en.wikipedia.org/wiki/Bcrypt 
 
 
 
 
 On 15.01.2018 11:19, Vladimir Ozerov wrote:
 
> 2) Credentials will be stored in a form of [username + hash] [1]
> 
 --
 Taras Ledkov
 Mail-To: tled...@gridgain.com 
 
 
>> 
>> -- 
>> Taras Ledkov
>> Mail-To: tled...@gridgain.com 
> -- 
> Taras Ledkov
> Mail-To: tled...@gridgain.com 

Re: [SparkDataFrame] Query Optimization. Prototype

[Denis Magda]

The testing of the SQL 99 *Core* specification is done and this page will be 
released once 2.4 goes live:
https://apacheignite-sql.readme.io/v2.3/docs/sql-conformance

However, Wiki page is already live:
https://en.wikipedia.org/wiki/SQL_compliance

—
Denis

> On Feb 12, 2018, at 12:20 AM, Vladimir Ozerov  wrote:
> 
> As far as testing of joins, I think it makes sense to start working on it
> when we finish ANSI compliance testing which is already in progress.

What happens if Primary Node fails during the Commit Phase

[John Wilson]

Hi,

Assume the Prepare phase has completed and that the primary node has
received a commit message from the coordinator.

Two questions:

   1. A primary node commits a transaction before it forwards a commit
   message to the backup nodes. True?
   2. What happens if a Primary Node fails while it is committing but
   before the commit message is sent to backup nodes? Do the backup nodes
   commit after some timeout or will they send a fail message to the
   coordinator?

The doc below provides a nice description but doesn't exactly answer my
question.

https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-failover-and-recovery

Thanks,

Re: Ignite diagnostic (SQL system views)

[Denis Magda]

Alex P, sounds like a good plan for me.

Vladimir, do you have any suggestions or corrections?

—
Denis

> On Feb 12, 2018, at 4:57 AM, Alex Plehanov  wrote:
> 
> The views engine and the first view are almost ready to merge (review
> comments are resolved). Which views should we take next? My proposal -
> NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and NODE_ADDRESSES, since
> these views are clear and all topology data available on each node.
> Any objections?
> 
> 2018-01-25 16:27 GMT+03:00 Alex Plehanov :
> 
>> Anton, Vladimir, I've made some fixes. There is only one view left and
>> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'.
>> 
>> High level design of solution:
>> When IgniteH2Indexing is starting, it create and start
>> new GridH2SysViewProcessor, which create and register in H2 (via its own
>> table engine) all implementations of system views. Each system view
>> implementation extends base abstract class GridH2SysView. View
>> implementation describes columns, their types and indexes in constructor
>> and must override method getRows for data retrieval (this method called by
>> H2-compatible table and index implementations for ignite system views).
>> Almost no fixes to existing parsing engine was made, except some places,
>> where GridH2Table instance was expected, but for system views there is
>> another class.
>> 
>> New PR: [1].  Please have a look.
>> 
>> [1] https://github.com/apache/ignite/pull/3433
>> 
>> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov :
>> 
>>> I've created IEP-13 [1] to cover all cases.
>>> Feel free to create issues.
>>> 
>>> [1]
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75962769
>>> 
>>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov 
>>> wrote:
>>> 
 Let's start with a single and the most simple view, e.g.
 LOCAL_TRANSACTIONS. We will review and merge it along with necessary
 infrastructure. Then will handle the rest view in separate tickets and
 separate focused discussions.
 
 On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov  1) It’s not a principal point, I can change schema. The
 INFORMATION_SCHEMA
> was used because it’s already exists and usually used for metadata
>>> tables
> and views. Your proposal is to use schema “IGNITE”, am I understand
>>> you
> right? BTW, for now, we can’t query another (H2) meta tables from the
> INFORMATION_SCHEMA, so, “Ignite system views” is only available views
>>> to
> query from this schema.
> 2) Exactly for this reason the IGNITE_INSTANCE view is useful: to
 determine
> which node we are connected to.
> 3) As the first phase, in my opinion, local views will be enough.
> Performance and caching of distributed views should be discussed at
>>> next
> phases, when distributed views implementation will be planned. In
>>> current
> implementation I tried to use indexing for local views wherever it’s
> possible.
> 4) I don’t think, that JVM info is more critical information than, for
> example, caches or nodes information. When authorization capabilities
> planned to implement?
> 
> About local data: yes, we can rename all currently implemented views
>>> for
> the local node data as LOCAL_..., and create (someday) new whole
>>> cluster
> views (which use distributed requests) without prefix or, for example,
 with
> CLUSTER_ prefix. But some views can show all cluster information using
 only
> local node data, without distributed requests (for example
> IGNITE_NODE_METRICS, IGNITE_PART_ASSIGNMENT, IGNITE_PART_ALLOCATION,
> IGNITE_NODES, etc). Are they local or cluster views in this concept?
 Which
> prefix should be used? And what about caches? Are they local or
>>> cluster?
 On
> local node we can see cluster wide caches (replicated and distributed)
 and
> caches for current node only. Local caches list may differ from node
>>> to
> node. Which prefix should be used for this view? And one more, there
>>> is
 no
> sense for some views to make them cluster wide (for example
> INGNITE_INSTANCE). Should we name it LOCAL_INSTANCE without creating
> INSTANCE view?
> 
> So, next steps: split PR, change schema name (IGNITE?), change view
>>> name
> for caches (CACHES, LOCAL_CACHES?)
> 
> 
> 2018-01-24 13:03 GMT+03:00 Vladimir Ozerov :
> 
>> Hi Alex,
>> 
>> System views could be extremely valuable addition for Ignite.
>>> Ideally,
> user
>> should be able to monitor and manage state of the whole cluster
>>> with a
>> single SQL command line. We have plans to implement it for a very
>>> long
>> time. However, this is very sensitive task which should take a lot
>>> of
>> moving pieces in count, such as usability, 

Re: Username/password authentication for thin clients

[Alexey Kuznetsov]

Taras,

How about to take a look at https://shiro.apache.org/ ?
Apache Shiro™ is a powerful and easy-to-use Java security framework.

It also supports Blowfish.
https://shiro.apache.org/static/1.3.1/apidocs/org/apache/shiro/crypto/BlowfishCipherService.html
And has Apache license.

-- 
Alexey Kuznetsov

Re: IgniteSet implementation: changes required

[Dmitriy Setrakyan]

On Fri, Feb 9, 2018 at 6:26 PM, Andrey Kuznetsov  wrote:

> Hi all,
>
> Current set implementation has significant flaw: all set data are
> duplicated in onheap maps on _every_ node in order to make iterator() and
> size(). For me it looks like simple yet ineffective implementation.
> Currently, these maps are damaged by checkpointing/recovery, and we could
> patch them somehow. Another future change to Ignite caches can damage them
> again. This looks fragile when datastructure is not entirely backed by
> caches. Pavel's proposal seems to be a reliable solution for non-collocated
> sets.
>

I would agree, but I was under an impression that non-collocated sets are
already implemented this way. Am I wrong?

Re: Make Teamcity Green Again

[Dmitriy Setrakyan]

Dmitriy, thanks for pushing this! We have to bring the number of the
failing tests to zero. I hope that the community gets behind this effort.

D.

On Mon, Feb 12, 2018 at 8:26 AM, Dmitry Pavlov 
wrote:

> Hi Folks,
>
> I want to resurrect this process and start to fix tests. Currently approx.
> 60-80 tests are failed each run on master.
>
> As one from the first point I suggest to use TC investigations for brief
> research of test failures.
>
> If you can help with research please create and assign investigation on TC
> (on test fail select dropdown menu, then 'Investigate / Mute'
> then 'Investigated by: me' & 'Resolve: auto/manually'; Runs
> https://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests24Java8_RunAll_IgniteTests24Java8=%3Cdefault%3E
> ). Also help with existing fixes review/merge would be very appreciated.
>
> Issue creation is still desired, but investigation will help us to identify
> tests failures which are under research and failures not researched for
> now.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 7 нояб. 2017 г. в 0:06, Dmitry Pavlov :
>
> > "Ignored" means that the test is disabled from the code, using annotation
> > @Ignore. Not sure it works for our JUnit3 style Java tests. It is used
> for
> > .NET.
> > "Muted" test is disabled in the interface TeamCity, but test code is
> > actually executed (which is why hanging tests needs to be failed in
> > addition to mute).
> >
> > The fact that the run all now became green means all the tests that fall
> > at this point are flaky. If it is about 200 non-muted flaky tests having
> > the probability of passing 98%, then the whole chain will pass only with
> > the probability 0.98 ^ 200 = 1.7%
> >
> > I suggest continue to identify, fix, or mute such flaky tests to get
> > stable green RunAll for correct code.
> >
> > пн, 6 нояб. 2017 г. в 22:57, Denis Magda :
> >
> >> Dmitriy,
> >>
> >> Now it’s time to fix a pretty decent number of muted and ignored tests.
> >> BTW, what’s the difference between “ignored” and “muted”? I’ve seen the
> >> former before.
> >>
> >> —
> >> Denis
> >>
> >> > On Nov 5, 2017, at 9:58 AM, Dmitry Pavlov 
> >> wrote:
> >> >
> >> > Hi Igniters,
> >> >
> >> > I am happy to report that the activity of monitoring and fixing the
> >> tests
> >> > brought the first significant victory!
> >> >
> >> > Today tests launch running all Ignite tests (Ignite 2.0 Tests->RunAll)
> >> is
> >> > completely green:
> >> https://ci.ignite.apache.org/viewLog.html?buildId=930953
> >> >
> >> > This is an reason for being pride, but not a reason to stop this
> >> activity.
> >> > Unfortunately, there are still a number of tests with unpredictable
> >> > failures (flaky, 234 tests), and a number of tests have been muted.
> Full
> >> > project scope is available in the article
> >> >
> >> https://cwiki.apache.org/confluence/display/IGNITE/
> Make+Teamcity+Green+Again
> >> >
> >> > In any case, thank to all of you who helped and continues to help the
> >> > project to correct the tests and bugs.
> >> >
> >> > Sincerely,
> >> > Pavlov Dmitry
> >> >
> >> >
> >> > пт, 21 июл. 2017 г. в 14:29, Pavel Tupitsyn :
> >> >
> >> >>> Green Again
> >> >>> Again
> >> >> As if it ever was green :)
> >> >>
> >> >> Of course +1 on this, let me know if you see any .NET-related
> failures.
> >> >>
> >> >> On Fri, Jul 21, 2017 at 1:06 PM, Николай Ижиков <
> >> nizhikov@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Hello, Igniters.
> >> >>>
> >> >>> Also ready to help to #MakeTeamcityGreenAgain !
> >> >>>
> >> >>> 21 июля 2017 г. 12:56 PM пользователь "Vyacheslav Daradur" <
> >> >>> daradu...@gmail.com> написал:
> >> >>>
> >>  Hi guys.
> >> 
> >>  I vote for #MakeTeamcityGreenAgain. :-)
> >> 
> >>  FYI: it had been described and supported previously[1]
> >> 
> >>  After the completion of my current task I will try to help with
> this
> >>  activity.
> >> 
> >>  [1]
> >>  http://apache-ignite-developers.2346864.n4.nabble.
> >>  com/Test-failures-td14353.html
> >> 
> >>  2017-07-21 12:39 GMT+03:00 Anton Vinogradov :
> >> 
> >> > Nikolay,
> >> >
> >> > That's also a big problem for me, as reviewer, to accept changes
> and
> >>  merge
> >> > them to master.
> >> >
> >> > Dmitriy,
> >> >
> >> > Currently we have some contributions *from community* blocked by
> red
> >> > Teamcity.
> >> >
> >> > On Fri, Jul 21, 2017 at 12:47 AM, Nikolay Izhikov <
> >>  nizhikov@gmail.com>
> >> > wrote:
> >> >
> >> >> +1 to Dmitry Pavlov from me.
> >> >>
> >> >> Green team city would be very helpful.
> >> >>
> >> >>
> >> >>> If you are a newcomer, please share your feeling: are you
> >> >> confused
> >> >>> by
> >> >> test
> >> >>> failures on Teamcity?
> >> >>
> >> >> I'm working on my 

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov 
wrote:

> Hi,
>
> Unfortunately, a quick fix did not give us too much performance boost.
>
> I'm going to implement a complete algorithm change for storing the page
> identifier. But this change is quite significant and will require
> re-testing. I suggest including
> https://issues.apache.org/jira/browse/IGNITE-7638 in the next version, for
> example, to 2.5.
>
> Sincerely,
> Dmitriy Pavlov
>
>
>
Dmitriy, thanks for the update! Are there other tickets that are holding
the release at this point? I remember that there was a performance
degradation issue in FULL_SYNC mode, but I cannot find a ticket.

D.

[jira] [Created] (IGNITE-7681) SQL COPY: measure and fix performance if needed

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7681:
---

 Summary: SQL COPY: measure and fix performance if needed
 Key: IGNITE-7681
 URL: https://issues.apache.org/jira/browse/IGNITE-7681
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.4, 2.5
Reporter: Kirill Shirokov
Assignee: Kirill Shirokov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [SparkDataFrame] Query Optimization. Prototype

[Valentin Kulichenko]

Nikolay,

When you're talking about join optimization, what exactly are you referring
to?

Since other parts of data frames integration are already merged, I think
it's a good time to resurrect this thread? Does it make sense to review it
right now? Or you want to make some more changes?

-Val

On Mon, Feb 12, 2018 at 12:20 AM, Vladimir Ozerov 
wrote:

> Hi Nikolay,
>
> I am not sure if ticket for DECIMAL column metadata exists. If you haven't
> find one under "sql" component, please feel free to create it on your own.
> As far as testing of joins, I think it makes sense to start working on it
> when we finish ANSI compliance testing which is already in progress.
>
> On Wed, Jan 24, 2018 at 12:27 PM, Nikolay Izhikov 
> wrote:
>
>> Hello, Vladimir.
>>
>> Thank you for an answer.
>>
>> > Do you mean whether it is possible to read it from table metadata?
>>
>> Yes, you are right.
>> I want to read scale and precision of DECIMAL column from table metadata.
>>
>> > This will be fixed at some point in future, but I do not have any dates
>> at the moment.
>>
>> Is there ticket for it? I can't find it via jira search
>>
>> > at this moment we do not have complete list of all restrictions on our
>> joins, because a lot of work is delegated to H2.
>> > In some unsupported scenarios we throw an exception.
>> > In other cases we return incorrect results silently (e.g. if you do not
>> co-locate data and forgot to set "distributed joins" flag).
>>
>> Guys, Val, may be we should exclude join optimization from IGNITE-7077
>> while we haven't all limitation on the hand?
>>
>> > We have a plan to perform excessive testing of joins (both co-located
>> and distributed) and list all known limitations.
>>
>> Can I help somehow with this activity?
>>
>>
>> В Ср, 24/01/2018 в 12:08 +0300, Vladimir Ozerov пишет:
>> > Hi Nikolay,
>> >
>> > Could you please clarify your question about scale and precision? Do
>> you mean whether it is possible to read it from table metadata? If yes, it
>> is not possible at the moment unfortunately - we do not store information
>> about lengths, scales and precision, only actual data types are passed to
>> H2 (e.g. String, BigDecimal, etc.). This will be fixed at some point in
>> future, but I do not have any dates at the moment.
>> >
>> > Now about joins - Denis, I think you provided wrong link to our
>> internal GridGain docs where we accumulate information about ANSI
>> compatibility and which will are going to publish on Ignite WIKI when it is
>> ready. In any case, this is not what Nikolay aksed about. The question was
>> about limitation of our joins which has nothing to do with ANSI standard.
>> Unfortunately, at this moment we do not have complete list of all
>> restrictions on our joins, because a lot of work is delegated to H2. In
>> some unsupported scenarios we throw an exception. In other cases we return
>> incorrect results silently (e.g. if you do not co-locate data and forgot to
>> set "distributed joins" flag). We have a plan to perform excessive testing
>> of joins (both co-located and distributed) and list all known limitations.
>> This would require writing a lot of unit tests to cover various scenarios.
>> I think we will have this information in a matter of 1-2 months.
>> >
>> > Vladimir.
>> >
>> > On Tue, Jan 23, 2018 at 11:45 PM, Denis Magda 
>> wrote:
>> > > Agree. The unsupported functions should be mentioned on the page that
>> will cover Ignite ANSI-99 compliance. We have first results available for
>> CORE features of the specification:
>> > > https://ggsystems.atlassian.net/wiki/spaces/GG/pages/4509364
>> 6/ANSI+SQL+99 > et/wiki/spaces/GG/pages/45093646/ANSI+SQL+99>
>> > >
>> > > That’s on my radar. I’ll take care of this.
>> > >
>> > > —
>> > > Denis
>> > >
>> > > > On Jan 23, 2018, at 10:31 AM, Dmitriy Setrakyan <
>> dsetrak...@apache.org> wrote:
>> > > >
>> > > > I think we need a page listing the unsupported functions with
>> explanation
>> > > > why, which is either it does not make sense in Ignite or is planned
>> in
>> > > > future release.
>> > > >
>> > > > Sergey, do you think you will be able to do it?
>> > > >
>> > > > D.
>> > > >
>> > > > On Tue, Jan 23, 2018 at 12:05 AM, Serge Puchnin <
>> sergey.puch...@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> yes, the Cust function is supporting both Ignite and H2.
>> > > >>
>> > > >> I've updated the documentation for next system functions:
>> > > >> CASEWHEN Function, CAST, CONVERT, TABLE
>> > > >>
>> > > >> https://apacheignite-sql.readme.io/docs/system-functions
>> > > >>
>> > > >> And for my mind, next functions aren't applicable for Ignite:
>> > > >> ARRAY_GET, ARRAY_LENGTH, ARRAY_CONTAINS, CSVREAD, CSVWRITE,
>> DATABASE,
>> > > >> DATABASE_PATH, DISK_SPACE_USED, FILE_READ, FILE_WRITE, LINK_SCHEMA,
>> > > >> MEMORY_FREE, MEMORY_USED, LOCK_MODE, LOCK_TIMEOUT, READONLY,
>> CURRVAL,
>> > > >> AUTOCOMMIT, 

[GitHub] ignite pull request #3484: IGNITE-7451: Make Linear SVM for multi-classifica...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3484


---

[GitHub] ignite pull request #3512: IGNITE-7626 Unify code in test which clean up per...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3512


---

[GitHub] ignite pull request #3512: IGNITE-7626 Unify code in test which clean up per...

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/3512

IGNITE-7626 Unify code in test which clean up persistence directories



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7626-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3512


commit 4af52aafd6c3180c7f0ff49a311833f26e69ed2e
Author: EdShangGG 
Date:   2018-02-12T18:32:57Z

IGNITE-7626 Unify code in test which clean up persistence directories




---

[GitHub] ignite pull request #3511: Ignite 7202 1: Direct IO test problem locating

[dspavlov]

GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3511

Ignite 7202 1: Direct IO test problem locating



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7202-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3511


commit dd22e281352c6bdbec90b2b3370e9f0822873c71
Author: dpavlov 
Date:   2018-02-12T17:27:28Z

IGNITE-7202: long running tests fix

commit f850e0fd051ddaf368439d8f0f99148c545ea33b
Author: dpavlov 
Date:   2018-02-12T17:44:48Z

IGNITE-7202: long running tests fix

commit e3d3a8246dfdb34003e01eacce2fa00547b61401
Author: dpavlov 
Date:   2018-02-12T18:19:26Z

IGNITE-7202: long running tests locating problem with direct IO




---

[jira] [Created] (IGNITE-7680) Ignite PDS 1 (Direct IO): failed test IgnitePdsCheckpointSimulationWithRealCpDisabledTest.testDirtyFlag()

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-7680:
--

 Summary: Ignite PDS 1 (Direct IO): failed test 
IgnitePdsCheckpointSimulationWithRealCpDisabledTest.testDirtyFlag()
 Key: IGNITE-7680
 URL: https://issues.apache.org/jira/browse/IGNITE-7680
 Project: Ignite
  Issue Type: Bug
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov


 Ignite PDS 1 (Direct IO) [ tests failed ]   

  org.apache.ignite.testsuites.IgnitePdsNativeIoTestSuite: 
org.apache.ignite.internal.processors.cache.persistence.db.file.IgnitePdsCheckpointSimulationWithRealCpDisabledTest.testDirtyFlag
 (fail rate 100%) 
{noformat}

class org.apache.ignite.IgniteException: Failed to begin checkpoint (it is 
already in progress).

at 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beginCheckpoint(PageMemoryImpl.java:961)
at 
org.apache.ignite.internal.processors.cache.persistence.db.file.IgnitePdsCheckpointSimulationWithRealCpDisabledTest.testDirtyFlag(IgnitePdsCheckpointSimulationWithRealCpDisabledTest.java:564)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
at java.lang.Thread.run(Thread.java:748)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.4 release

[Dmitry Pavlov]

Hi,

Unfortunately, a quick fix did not give us too much performance boost.

I'm going to implement a complete algorithm change for storing the page
identifier. But this change is quite significant and will require
re-testing. I suggest including
https://issues.apache.org/jira/browse/IGNITE-7638 in the next version, for
example, to 2.5.

Sincerely,
Dmitriy Pavlov


ср, 7 февр. 2018 г. в 22:27, Dmitriy Setrakyan :

> Agree on both, the performance fix and the spark data frames. Let's get
> them into the release.
>
> However, Raymond is right. We should know how long the performance fix will
> take. If it adds another month to the development, we should include it
> into the next release. I am hoping that it can be done faster though.
>
>
> Alexey Goncharuk, Dmitriy Pavlov, any ideas?
>
> D.
>
> On Wed, Feb 7, 2018 at 9:07 AM, Nikolay Izhikov 
> wrote:
>
> > Hello, Igniters.
> >
> > Please, consider including IGNITE-7337 - Spark Data Frames: support
> saving
> > a data frame in Ignite [1] in the 2.4 release.
> > It seems we can merge it into the master in a day or few.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7337
> >
> >
> > В Ср, 07/02/2018 в 08:35 -0800, Denis Magda пишет:
> > > I’m voting for the blocker addition into the release. Sergey K. how
> will
> > it affect your testing cycles? Do you need to re-run everything from
> > scratch and how many days you need?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> > > >
> > > > Guys,
> > > >
> > > > Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major
> > > > slowdown when page replacement starts. Even though it's not a
> > regression, I
> > > > suggest we consider it a blocker for 2.4 because this is a huge
> > performance
> > > > issue which can make it virtually impossible to use native
> persistence
> > when
> > > > data size is significantly larger than memory size.
> > > >
> > > > Any objections?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-7638
> > > >
> > > > 2018-01-30 17:10 GMT+03:00 Pavel Tupitsyn :
> > > >
> > > > > Igniters, I will handle 2.4 release if there are no objections.
> > > > > Let's take a bit more time for testing and start vote by the end of
> > this
> > > > > week.
> > > > >
> > > > > Pavel
> > > > >
> > > > > On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda 
> > wrote:
> > > > >
> > > > > > Hi Vyacheslav,
> > > > > >
> > > > > > According to the previous review notes the impact of the changes
> > might be
> > > > > > significant, thus, I would recommend us to move the changes to
> the
> > next
> > > > > > release.
> > > > > >
> > > > > > BTW, don’t hesitate to ping reviewers more frequently if there
> is a
> > > > > > pending/abandon review. We are all the people who are tend to
> > forget or
> > > > > > miss notifications ;)
> > > > > >
> > > > > > > On Jan 26, 2018, at 2:04 AM, Vyacheslav Daradur <
> > daradu...@gmail.com>
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi, Vladimir, it's good news. I'm looking forward to new Ignite
> > > > >
> > > > > release!
> > > > > > >
> > > > > > > Could you please share a release schedule for 'varint'
> > optimizations?
> > > > > > >
> > > > > > > The task [1] is waiting for review for 5 months.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 26, 2018 at 12:51 PM, Vladimir Ozerov <
> > > > >
> > > > > voze...@gridgain.com>
> > > > > > wrote:
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > As far as I can see all required tasks and fixes were
> merged. I
> > > > >
> > > > > propose
> > > > > > to
> > > > > > > > take several days of silence to test what we've done and
> start
> > vote at
> > > > > >
> > > > > > the
> > > > > > > > beginning of the next week.
> > > > > > > >
> > > > > > > > Makes sense?
> > > > > > > >
> > > > > > > > On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda <
> > dma...@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Ok, let’s target Wednesday as a code freeze date.
> > > > > > > > >
> > > > > > > > > Community members who are involved in 2.4 release please
> > merge you
> > > > > >
> > > > > > fixes
> > > > > > > > > and optimizations by that time.
> > > > > > > > >
> > > > > > > > > —
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov <
> > a...@apache.org>
> > > > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Issues
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6711
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6902
> > > > > > > > > >
> > > > > > > > > > are in master and ignite-2.4
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda <
> > dma...@apache.org>
> > > > > >
> > > > > 

[GitHub] ignite pull request #1921: IGNITE-5151: Add some warning when offheap evicti...

[vinx13]

Github user vinx13 closed the pull request at:

https://github.com/apache/ignite/pull/1921


---

[GitHub] ignite pull request #1919: IGNITE-5179: Inconsistent return type in javadoc ...

[vinx13]

Github user vinx13 closed the pull request at:

https://github.com/apache/ignite/pull/1919


---

[GitHub] ignite pull request #1917: IGNITE-5011: Add validation on args and show prop...

[vinx13]

Github user vinx13 closed the pull request at:

https://github.com/apache/ignite/pull/1917


---

Re: Plugins in tests

[Dmitriy Govorukhin]

 Yakov,

Agree, I created the issue for this. I will try to do it in the near future.

IGNITE-7679 


On Tue, Jan 30, 2018 at 8:40 AM, Yakov Zhdanov  wrote:

> Guys,
>
> When running tests from core module I see that Ignite has 2 plugins
> configured by default (because they are available in classpath):
>
> -TestReconnectPlugin 1.0
> -StanByClusterTestProvider 1.0
>
> It seems they were introduced by Dmitry Karachentsev and Dmitry Govorukhin.
> Guys, can you please move the plugins to extdata project similar to
> PlatformTestPlugin and configure them only when needed. This is not correct
> that each test we run for Ignite runs with plugins configured. By default
> Ignite does not have any plugin.
>
> --Yakov
>

[jira] [Created] (IGNITE-7679) Move all test plugins to a separate module

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-7679:
--

 Summary: Move all test plugins to a separate module
 Key: IGNITE-7679
 URL: https://issues.apache.org/jira/browse/IGNITE-7679
 Project: Ignite
  Issue Type: Test
Reporter: Dmitriy Govorukhin
Assignee: Dmitriy Govorukhin
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3477: IGNITE-7626 Unify code in test which clean up per...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3477


---

Re: Make Teamcity Green Again

[Dmitry Pavlov]

Hi Folks,

I want to resurrect this process and start to fix tests. Currently approx.
60-80 tests are failed each run on master.

As one from the first point I suggest to use TC investigations for brief
research of test failures.

If you can help with research please create and assign investigation on TC
(on test fail select dropdown menu, then 'Investigate / Mute'
then 'Investigated by: me' & 'Resolve: auto/manually'; Runs
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll_IgniteTests24Java8=%3Cdefault%3E
). Also help with existing fixes review/merge would be very appreciated.

Issue creation is still desired, but investigation will help us to identify
tests failures which are under research and failures not researched for now.

Sincerely,
Dmitriy Pavlov

вт, 7 нояб. 2017 г. в 0:06, Dmitry Pavlov :

> "Ignored" means that the test is disabled from the code, using annotation
> @Ignore. Not sure it works for our JUnit3 style Java tests. It is used for
> .NET.
> "Muted" test is disabled in the interface TeamCity, but test code is
> actually executed (which is why hanging tests needs to be failed in
> addition to mute).
>
> The fact that the run all now became green means all the tests that fall
> at this point are flaky. If it is about 200 non-muted flaky tests having
> the probability of passing 98%, then the whole chain will pass only with
> the probability 0.98 ^ 200 = 1.7%
>
> I suggest continue to identify, fix, or mute such flaky tests to get
> stable green RunAll for correct code.
>
> пн, 6 нояб. 2017 г. в 22:57, Denis Magda :
>
>> Dmitriy,
>>
>> Now it’s time to fix a pretty decent number of muted and ignored tests.
>> BTW, what’s the difference between “ignored” and “muted”? I’ve seen the
>> former before.
>>
>> —
>> Denis
>>
>> > On Nov 5, 2017, at 9:58 AM, Dmitry Pavlov 
>> wrote:
>> >
>> > Hi Igniters,
>> >
>> > I am happy to report that the activity of monitoring and fixing the
>> tests
>> > brought the first significant victory!
>> >
>> > Today tests launch running all Ignite tests (Ignite 2.0 Tests->RunAll)
>> is
>> > completely green:
>> https://ci.ignite.apache.org/viewLog.html?buildId=930953
>> >
>> > This is an reason for being pride, but not a reason to stop this
>> activity.
>> > Unfortunately, there are still a number of tests with unpredictable
>> > failures (flaky, 234 tests), and a number of tests have been muted. Full
>> > project scope is available in the article
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/Make+Teamcity+Green+Again
>> >
>> > In any case, thank to all of you who helped and continues to help the
>> > project to correct the tests and bugs.
>> >
>> > Sincerely,
>> > Pavlov Dmitry
>> >
>> >
>> > пт, 21 июл. 2017 г. в 14:29, Pavel Tupitsyn :
>> >
>> >>> Green Again
>> >>> Again
>> >> As if it ever was green :)
>> >>
>> >> Of course +1 on this, let me know if you see any .NET-related failures.
>> >>
>> >> On Fri, Jul 21, 2017 at 1:06 PM, Николай Ижиков <
>> nizhikov@gmail.com>
>> >> wrote:
>> >>
>> >>> Hello, Igniters.
>> >>>
>> >>> Also ready to help to #MakeTeamcityGreenAgain !
>> >>>
>> >>> 21 июля 2017 г. 12:56 PM пользователь "Vyacheslav Daradur" <
>> >>> daradu...@gmail.com> написал:
>> >>>
>>  Hi guys.
>> 
>>  I vote for #MakeTeamcityGreenAgain. :-)
>> 
>>  FYI: it had been described and supported previously[1]
>> 
>>  After the completion of my current task I will try to help with this
>>  activity.
>> 
>>  [1]
>>  http://apache-ignite-developers.2346864.n4.nabble.
>>  com/Test-failures-td14353.html
>> 
>>  2017-07-21 12:39 GMT+03:00 Anton Vinogradov :
>> 
>> > Nikolay,
>> >
>> > That's also a big problem for me, as reviewer, to accept changes and
>>  merge
>> > them to master.
>> >
>> > Dmitriy,
>> >
>> > Currently we have some contributions *from community* blocked by red
>> > Teamcity.
>> >
>> > On Fri, Jul 21, 2017 at 12:47 AM, Nikolay Izhikov <
>>  nizhikov@gmail.com>
>> > wrote:
>> >
>> >> +1 to Dmitry Pavlov from me.
>> >>
>> >> Green team city would be very helpful.
>> >>
>> >>
>> >>> If you are a newcomer, please share your feeling: are you
>> >> confused
>> >>> by
>> >> test
>> >>> failures on Teamcity?
>> >>
>> >> I'm working on my very first ignite issue
>> >> and it's not easy to understand TC build results.
>> >>
>> >>
>> >> 21.07.2017 00:30, Dmitriy Setrakyan пишет:
>> >>
>> >> I think the right question to ask is why do we have RED tests to
>> >>> begin
>> >>> with. In my view, TC should be green. If there are tests that do
>> >> not
>> > pass,
>> >>> we should disable them and file blocker tickets for the next
>> >>> releases.
>> >>>
>> >>> D.
>> >>>
>> >>> On Thu, Jul 20, 2017 at 

[GitHub] ignite pull request #3510: IGNITE-7676 Add affinity version to snapshot plug...

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/3510

IGNITE-7676 Add affinity version to snapshot plugin stub



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7676

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3510


commit 41cc1a2376e8a5ec9b48e89bbd877b2865d502eb
Author: EdShangGG 
Date:   2018-02-12T15:58:32Z

IGNITE-7676 Add affinity version to snapshot plugin stub




---

[jira] [Created] (IGNITE-7678) Persistence: newly created cache creates partitions in MOVING state

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-7678:


 Summary: Persistence: newly created cache creates partitions in 
MOVING state
 Key: IGNITE-7678
 URL: https://issues.apache.org/jira/browse/IGNITE-7678
 Project: Ignite
  Issue Type: Test
  Components: persistence
Affects Versions: 2.4
Reporter: Alexey Goncharuk
Assignee: Alexey Goncharuk
 Fix For: 2.5


initPartitions0 will generate partitions in MOVING state if there are no 
partition files in the FS. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3509: GG-13266 Node which not in baseline participating...

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/3509

GG-13266 Node which not in baseline participating in snapshot

GG-13344 CANCEL of RESTORE could hang

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7676

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3509.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3509


commit 7cb1e344a3070eeacc21b3680a39b34a12f0f275
Author: EdShangGG 
Date:   2018-02-12T15:05:10Z

GG-13266 Node which not in baseline participating in snapshot
GG-13344 CANCEL of RESTORE could hang




---

Re: Username/password authentication for thin clients

[Taras Ledkov]

Colleagues, Denis,

It will be great to use bcrypt for password hashing in Ignite.
Could you suggest the right way to use bcrypt:
1. add 'jbcrypt' maven dependency;
2. include the single 'BCrypt.java' file to our project [1].

Does the license allow to include 'BCrypt.java' ?

[1]. 
https://github.com/jeremyh/jBCrypt/blob/master/src/main/java/org/mindrot/BCrypt.java



On 18.01.2018 13:50, Taras Ledkov wrote:


Password hashing algorithms of the popular vendors:

mysql: SHA-265, old-native-hash
postgres: MD5, DES, Extended DES, Blowfish-based
oracle: SHA-1

Some about "comparison" SHA-2 vs bcrypt [1]:

> SHA-512 is a cryptographic hash while bcrypt is a password hash or 
PBKDF (password based key derivation function).


> SHA-512 has been designed to be fast. You don't want any delays when 
validating a signature, for instance.

> There is no reason for generic cryptographic hashes to be slow.

> bcrypt on the other hand is a password hash that performs key 
strengthening on the input.
> Basically it does this by slowing down the calculation so that 
attackers will have to spend
> more resources to find the input by brute forcing or dictionary 
attacks.
> The idea is that although the legit users - you in this case - will 
also be slowed down,
> they are only slowed down once per password. However the attackers 
are slowed down for each try.
> The legit user is of course much more likely to input the right 
password first.


> Furthermore bcrypt also contains a salt as input, which can be used 
to avert rainbow table attacks.


Conclusion: bcrypt can provide more security but the popular vendors 
use SHA and even MD5 by default.


[1]. 
https://crypto.stackexchange.com/questions/46550/benchmark-differences-between-sha-512-and-bcrypt


On 18.01.2018 9:29, Vladimir Ozerov wrote:

Taras,

I think we need a comparison of available options and (possibly) analysis
what other vendors use.

On Tue, Jan 16, 2018 at 3:56 PM, Taras Ledkov  wrote:


What do you think about usage bcrypt [1], [2] to store encrypted password?

[1]https://stackoverflow.com/questions/1561174/sha512-vs-blowfi
sh-and-bcrypt
[2]https://en.wikipedia.org/wiki/Bcrypt



On 15.01.2018 11:19, Vladimir Ozerov wrote:


2) Credentials will be stored in a form of [username + hash] [1]


--
Taras Ledkov
Mail-To:tled...@gridgain.com




--
Taras Ledkov
Mail-To:tled...@gridgain.com


--
Taras Ledkov
Mail-To: tled...@gridgain.com

[GitHub] ignite pull request #3509: GG-13266 Node which not in baseline participating...

[EdShangGG]

Github user EdShangGG closed the pull request at:

https://github.com/apache/ignite/pull/3509


---

[jira] [Created] (IGNITE-7677) CacheBaselineTopologyTest flaky-fails on TC

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-7677:


 Summary: CacheBaselineTopologyTest flaky-fails on TC
 Key: IGNITE-7677
 URL: https://issues.apache.org/jira/browse/IGNITE-7677
 Project: Ignite
  Issue Type: Test
Reporter: Alexey Goncharuk
Assignee: Alexey Goncharuk






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3508: ignite-7606: unit test fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3508


---

[GitHub] ignite pull request #3508: ignite-7606: unit test fix

[dspavlov]

GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3508

ignite-7606: unit test fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7606-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3508


commit b028beba4b1b87d3d8fb61e8bc481fa04f5fc938
Author: dpavlov 
Date:   2018-02-12T15:49:47Z

ignite-7606: unit test fix




---

[jira] [Created] (IGNITE-7676) Add affinity version to snapshot plugin stub

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-7676:
-

 Summary: Add affinity version to snapshot plugin stub
 Key: IGNITE-7676
 URL: https://issues.apache.org/jira/browse/IGNITE-7676
 Project: Ignite
  Issue Type: Improvement
Reporter: Eduard Shangareev
Assignee: Eduard Shangareev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3507: IGNITE-7652: ContinuousQueryWithTransformer examp...

[nizhikov]

GitHub user nizhikov opened a pull request:

https://github.com/apache/ignite/pull/3507

IGNITE-7652: ContinuousQueryWithTransformer example

SelfTest for ContinuousQuery examples added.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nizhikov/ignite IGNITE-7652

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3507


commit 7cd4aa77e4a1833ab24bbdfe8c98a4381c1b4851
Author: Nikolay Izhikov 
Date:   2018-02-12T15:22:52Z

IGNITE-7652: ContinuousQueryWithTransformer example. SelfTest for 
ContinuousQuery examples added




---

[jira] [Created] (IGNITE-7675) Ignite Hadoop: test failures: "Stream has been closed: IgfsOutputStreamImpl"

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-7675:
--

 Summary: Ignite Hadoop: test failures: "Stream has been closed: 
IgfsOutputStreamImpl"
 Key: IGNITE-7675
 URL: https://issues.apache.org/jira/browse/IGNITE-7675
 Project: Ignite
  Issue Type: Bug
  Components: hadoop
Reporter: Dmitriy Pavlov


https://ci.ignite.apache.org/viewLog.html?buildId=1089582=buildResultsDiv=IgniteTests24Java8_IgniteHadoop


{noformat}
java.io.IOException: Stream has been closed: IgfsOutputStreamImpl 
[mode=PRIMARY, writeFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, 
res=true, hash=1714500898], fileInfo=IgfsFileInfo [len=0, blockSize=524288, 
lockId=bf6d22a8161-768e5785-d20e-4bd0-8070-42e43d53, affKey=null, 
fileMap=IgfsFileMap [ranges=null], evictExclude=true], streamRange=null]
--- Stdout: ---
[2018-02-12 13:08:13,048][INFO ][main][root] >>> Starting test: 
HadoopIgfs20FileSystemLoopbackPrimarySelfTest#testRenameIfSrcPathIsAlreadyBeingOpenedToRead
 <<<
[2018-02-12 13:08:13,133][INFO ][test-runner-#2732%grid%][root] Caught expected 
exception: File not found: igfs://igfs@/srcFile
[2018-02-12 13:08:13,139][INFO ][main][root] >>> Stopping test: 
HadoopIgfs20FileSystemLoopbackPrimarySelfTest#testRenameIfSrcPathIsAlreadyBeingOpenedToRead
 in 70 ms <<<
--- Stderr: ---
[2018-02-12 13:08:13,137][ERROR][main][root] Test failed.
java.io.IOException: Stream has been closed: IgfsOutputStreamImpl 
[mode=PRIMARY, writeFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, 
res=true, hash=1714500898], fileInfo=IgfsFileInfo [len=0, blockSize=524288, 
lockId=bf6d22a8161-768e5785-d20e-4bd0-8070-42e43d53, affKey=null, 
fileMap=IgfsFileMap [ranges=null], evictExclude=true], streamRange=null]
at 
org.apache.ignite.internal.processors.igfs.IgfsAbstractOutputStream.checkClosed(IgfsAbstractOutputStream.java:190)
at 
org.apache.ignite.internal.processors.igfs.IgfsOutputStreamImpl.flush(IgfsOutputStreamImpl.java:140)
at 
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsInProc.flush(HadoopIgfsInProc.java:614)
at 
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfsOutputStream.flush(HadoopIgfsOutputStream.java:131)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
at 
org.apache.ignite.internal.processors.hadoop.impl.igfs.HadoopIgfs20FileSystemAbstractSelfTest.testRenameIfSrcPathIsAlreadyBeingOpenedToRead(HadoopIgfs20FileSystemAbstractSelfTest.java:1137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: IGNITE-6005 is ready for review

[Ilya Lantukh]

Hi Nikolay,

Thanks for contribution!

I will try to review your pull request by tomorrow.

On Mon, Feb 12, 2018 at 11:45 AM, Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> I've fix some relatively small issue(100 loc) IGNITE-6005 [1], PR [2]
> The fix in DataStructuresProcessor.java
>
> As far as I can see from git log Ilya Lantukh maintains this piece of code.
>
> Ilya, can you please take a look at the fix?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-6005
>
> [2] https://github.com/apache/ignite/pull/2773




-- 
Best regards,
Ilya

[GitHub] ignite pull request #3505: Ignite 2.3.3

[alamar]

GitHub user alamar opened a pull request:

https://github.com/apache/ignite/pull/3505

Ignite 2.3.3



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.3.3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3505


commit be91bbf8bcb7e9c71fe6d3bea0f79763f9606558
Author: Krzysztof Chmielewski 
Date:   2017-10-10T14:50:59Z

Fixed "IGNITE-6234 Initialize schemaIds to empty set if schemas field is 
null during the deserialization".

Signed-off-by: nikolay_tikhonov 

commit 08389601728512dc4e7fa5b953f5afe34ae4506f
Author: AMRepo 
Date:   2017-10-10T08:57:20Z

IGNITE-6545: Failure during Ignite Service.cancel() can break normal 
shutdown process. This closes #2807.

(cherry picked from commit 8ffa109)

commit 57547b5afae059a0a6dfa13c08b2e0b6c0e96ebd
Author: devozerov 
Date:   2017-10-13T09:34:35Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit 08798f8e47bdfdd68a557385ed2ce98b4bb1609a
Author: devozerov 
Date:   2017-10-13T11:12:44Z

IGNITE-6605: SQL: common backup filter. This closes #2836.

commit 2b59a241de3935a338842b8fc3221aedc8e11e1d
Author: devozerov 
Date:   2017-10-16T07:33:36Z

IGNITE-6631: Minor improvements to GridH2KeyValueRowOnheap. This closes 
#2855.

commit 98438c954c5f9a08634cf3132361268456397864
Author: devozerov 
Date:   2017-10-16T09:38:54Z

IGNITE-6632: SQL: simplified GridH2Row inheritance tree. This closes #2856.

commit 95b7ab518dd3c3db6fcc5142c2ee85da2516c2b6
Author: devozerov 
Date:   2017-10-16T10:37:11Z

IGNITE-6634: Removed IgniteDistributedJoinTestSuite. It's tests are 
distributed between "Query" and "Query 2" suites. This closes #2857.

commit 9c91deff877ebc0eed84559d4abca71408e3cb0a
Author: devozerov 
Date:   2017-10-16T13:46:13Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit 911ab7ab7a8a6968d219b053adb2338477738506
Author: Alexey Popov 
Date:   2017-10-17T11:45:42Z

IGNITE-6627 .NET: Fix serialization of enums within generic collections

* Fix EnumEqualityComparer serialization
* Fix enum arrays serialization
* Fix empty objects missing metadata

This closes #2864

commit 3ba374c319ac7048a05871692060e2f143d6acdf
Author: Alexey Kuznetsov 
Date:   2017-10-06T17:11:37Z

IGNITE-6463 Web Console: Fixed output of big numbers in SQL query results.
(cherry picked from commit 35589a7)

commit b67feb0f175bfbd6ffbefe82a8d693c8ab7d4213
Author: Vasiliy Sisko 
Date:   2017-10-09T10:55:23Z

IGNITE-5767 Web console: Use byte array type instead of java.lang.Object 
for binary JDBC types.
(cherry picked from commit 3184437)

commit 8e1560322b87d79b3d3250832a3969ac4032d6fc
Author: Alexey Kuznetsov 
Date:   2017-10-06T18:10:08Z

IGNITE-6574 Remove pending requests in case STATUS_AUTH_FAILURE && 
credentials == null.
(cherry picked from commit 85261a3)

commit 7a0300ae35894c389b126e95615f720a99a3d360
Author: devozerov 
Date:   2017-10-18T11:18:08Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit ad01f9b099d0bf92537378859ad6d5a52de57748
Author: Alexey Kuznetsov 
Date:   2017-10-19T02:43:20Z

IGNITE-6647 Web Console: Implemented support of schema migration scripts.
(cherry picked from commit c65399c)

commit 0c66344bc752dac98b256dd140fcab95d1662862
Author: Pavel Tupitsyn 
Date:   2017-10-19T09:36:39Z

IGNITE-6627 .NET: Fix repeated known metadata updates

This closes #2876

commit 1b8abd214ed2afcd3fd1f6a4c71a19d6fe1a4b01
Author: Alexey Kuznetsov 
Date:   2017-10-20T04:23:23Z

IGNITE-6647 Added missing Mongo injector.
(cherry picked from commit 173ecef)

commit a221066b3d029afc392be704a810c0e830fc0c49
Author: Alexey Kuznetsov 
Date:   2017-10-20T14:15:02Z

IGNITE-6647 Web Console: Added folder for modules migrations.
(cherry picked from commit 3700717)

commit da8a9d5a968ba071697a28adb01bc59f80d1893c
Author: Pavel Tupitsyn 
Date:   2017-10-23T08:55:33Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

# Conflicts:
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Apache.Ignite.Core.Tests.csproj

commit 69fdac3acf768ecb9df80d4412c4de5ffd5bc4f5
Author: Dmitriy Shabalin 
Date:   2017-10-23T09:09:47Z

IGNITE-5909 Added list editable component.
(cherry picked from commit 01daee6)

commit 4a2c38333c112d4956d6394667672c1470503435
Author: apopov 
Date:   2017-10-24T08:56:33Z

[jira] [Created] (IGNITE-7674) It is possible to create BinaryObject with wrong field type and that can lead to broken Transaction (TransactionHeuristicException)

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-7674:
--

 Summary: It is possible to create BinaryObject with wrong field 
type and that can lead to broken Transaction (TransactionHeuristicException)
 Key: IGNITE-7674
 URL: https://issues.apache.org/jira/browse/IGNITE-7674
 Project: Ignite
  Issue Type: Bug
  Components: binary, sql
Affects Versions: 2.3
Reporter: Sergey Kosarev


Usuaully if we have any data in a cache, if we try to create BynaryObject with 
the same field name and another type we BinaryObjectException is thrown

Wrong value has been set [typeName=org.apache.ignite.internal.binary.Foo, 
fieldName=intField, fieldType=int, assignedValueType=String]

, but there are cases we can create an inconsistent BinaryObject

suppose we have 

{code:java}
class Foo {
private String strField;

private int intField;

public Foo(String strField, int intField) {
this.intField = intField;
this.strField = strField;
}
}{code}

case 1
{code:java}
binary.builder(Foo.class.getName())
.removeField("intField")
.build()
.toBuilder()
.setField("intField", "String")
.build();{code}

case 2 (if we remove all fields schema flag is cleared )
{code:java}
fooCache.withKeepBinary().get(1)
.toBuilder()
.removeField("intField")
.removeField("strField")
.build()
.toBuilder()
.setField("intField", "String")
.build(){code}

It is especially bad when we have an index on this field and cache is 
transactional.
if we put wrong BinaryObject into the cache, we got 
TransactionHeuristicException on commit and broken transaction (data can be 
comitted or not in some cases)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: IGNITE-4908 is ready for review (Ignite.reentrantLock looks much slower than IgniteCache.lock)

[Anton Vinogradov]

Nikolay,

Could you please review changes?

On Mon, Feb 12, 2018 at 3:23 PM, Александр Меньшиков 
wrote:

> Hi, guys! I'm waiting (more than 2 months) for CR ignite-4908 (
> https://issues.apache.org/jira/browse/IGNITE-4908 all links inside). It's
> about Ignite.reentrantlock.
>
> 2017-11-29 12:35 GMT+03:00 Александр Меньшиков :
>
>> Igniters,
>>
>> I’ve implemented new fast reentrant lock within an issue[1].
>>
>> Could someone review prepared PR [2, 3, 4]?
>>
>> Some details are described below.
>>
>> The main idea:
>>
>> Current locks implementation is based on shared states in a cache, while
>> the new lock uses IgniteCache#invoke* methods to update shared state
>> atomically. New lock implementation doesn’t use a continuous query, so the
>> cache became atomic now.
>>
>> New lock implementation has two types: fair and unfair, split into
>> different classes for performance reasons.
>>
>> Some benchmarks results (hardware: Core i5 (2 gen) + 6GB RAM):
>>
>> Speed up: single thread + fair:   21.9x (1 node), 3.4x (2 nodes), 9.9x (5
>> nodes), 17.9x (10 nodes)
>>
>> Speed up: single thread + unfair: 22.4x (1 node), 3.2x (2 nodes), 8.0x (5
>> nodes), 19.0x (10 nodes)
>>
>> Speed up: multi-threads + fair:   3.9x (1 n,2 t), 3.5x (1 n,10t), 13.5x
>> (5 n,2t), 15.0x (5n, 10t)
>>
>> Speed up: multi-threads + unfair: 33.5x (1 n,2t), 210x (1 n,10t), 318x
>>  (5 n,2t), 389x  (5n, 10t)
>>
>> Benchmarks’ summary:
>>
>> 1) The unfair lock has a local reentrant lock which is used for local
>> synchronization as a guard before the shared state. This allows reaching
>> performance close to a local reentrant lock.
>>
>> 2) One server node can be a primary one for the shared state, this gives
>> us a performance boost on one node only.
>>
>> 3) Speedup grows with a number of nodes.
>>
>>
>> [1] JIRA: https://issues.apache.org/jira/browse/IGNITE-4908
>>
>> [2] PR: https://github.com/apache/ignite/pull/2360
>>
>> [3] Upsource review: https://reviews.ignite.apache.
>> org/ignite/review/IGNT-CR-248
>>
>> [4] Team City: https://ci.ignite.apache.org/p
>> roject.html?projectId=Ignite20Tests_Ignite20Tests=
>> pull%2F2360%2Fhead
>>
>
>

[GitHub] ignite pull request #3504: Ignite 7673 2.1.7

[dkarachentsev]

GitHub user dkarachentsev opened a pull request:

https://github.com/apache/ignite/pull/3504

Ignite 7673 2.1.7



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7673-2.1.7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3504


commit 8dbdd03143362bb39d96242b23975efb22412709
Author: Ivan Rakov 
Date:   2017-08-22T14:03:42Z

IGNITE-6154 also fixed check for WAL record

commit afad8e0fc58160f7876925dc6c3051be7a168155
Author: Ilya Lantukh 
Date:   2017-08-23T09:18:44Z

Muted hanging test.

commit ad38f7b4b5e6845b2d7eb888f805484504f5
Author: Ilya Lantukh 
Date:   2017-08-23T11:12:42Z

gg-12662 : Fixed JDBC backward compatibility.

commit 28c906e3e0c51e6f1a4a95b2027d248f9b5035c2
Author: Sergey Chugunov 
Date:   2017-08-02T15:14:46Z

IGNITE-5542 CacheGroup configuration from cluster is merged with local 
settings

(cherry picked from commit 88818ec)

commit caeb11936fa3534b9468d443c11744362044cae5
Author: sboikov 
Date:   2017-08-23T12:19:52Z

ignite-6124 Guard logging with isInfoEnabled

(cherry picked from commit bebe4d8)

commit 6f407ebabb9dc27459fdbee6423640132b995b1d
Author: tledkov-gridgain 
Date:   2017-08-23T12:46:23Z

IGNITE-6169: Fixed thin JDBC driver compatibility problem.

commit 9dac636c4eef494fe612389c19218eec92057fc0
Author: Ilya Kasnacheev 
Date:   2017-08-23T13:26:58Z

IGNITE-4643: Fixed NPE in  JdbcDatabaseMetadata.getIndexInfo(). This closes 
#2481.

commit 77241cdc45c90ee9bab4a7a0f3d5a1a7664e3426
Author: sboikov 
Date:   2017-08-23T13:45:34Z

Merge remote-tracking branch 'community/ignite-2.1.4' into ignite-2.1.4

commit a5e376f63886696331e5be0c457dc0624c49e3d4
Author: sboikov 
Date:   2017-08-23T13:44:04Z

ignite-6124 Added missed initialization of merged join exchanges in 
GridDhtPartitionsExchangeFuture.onBecomeCoordinator

(cherry picked from commit 0c5dca9)

commit be5589db9e0600b295b745ddab5e7aae390ac7ae
Author: Ilya Lantukh 
Date:   2017-08-23T14:25:33Z

ignite-5986 : Fixed failing .NET test.

commit 43e4ff2c0ecd1ef30d18cf1fbc9052f5ba703d05
Author: sboikov 
Date:   2017-07-18T14:52:51Z

Fixed test IgniteClusterActivateDeactivateTestWithPersistence.

(cherry picked from commit 54585ab)

commit d596b7806db3f002f83da5a02bc882d03dae3dfd
Author: Ilya Lantukh 
Date:   2017-08-23T15:23:06Z

Updated classnames.properties.

commit 3e08cd401d598a34832e72afc5e6c94a3a9ab081
Author: sboikov 
Date:   2017-08-23T15:29:52Z

ignite-6174 Temporary changed test until issue not fixed

(cherry picked from commit 4fe8f76)

commit 44e0b4cd62142dce8cf39f826449b9a04e22e1cf
Author: Alexey Kuznetsov 
Date:   2017-08-24T07:57:36Z

IGNITE-6136 Fixed version for demo.
(cherry picked from commit e1bf8d7)

commit 8d1838b03d6c1e5f86dfbb7f41c59895775e20c1
Author: Dmitry Pavlov 
Date:   2017-07-27T11:51:25Z

Adjusted memory policy to prevent OOM.

commit a3ec54b16bce1a569fbefba17188ccb4702b82a4
Author: sboikov 
Date:   2017-08-24T11:09:12Z

ignite-6124 DataStreamerImpl: do not wait for exchange future inside cache 
gateway.

(cherry picked from commit 3ab523c)

commit 30e6d019a21f4a045a50d7d95a04507e3b646e69
Author: sboikov 
Date:   2017-08-24T11:10:34Z

Merge remote-tracking branch 'community/ignite-2.1.4' into ignite-2.1.4

commit 41f574a7372ffc04b69809298798f24fb34c161f
Author: Dmitriy Govorukhin 
Date:   2017-08-24T12:58:27Z

Fixed test.

commit 943736b36d67381157fc2807cd7af4b03d44fef3
Author: nikolay_tikhonov 
Date:   2017-08-24T15:58:16Z

Revert "IGNITE-5947 Fixed "ClassCastException when two-dimensional array is 
fetched from cache".
* Due to this changes break compatibility with .NET;
* This fix doesn't cover all cases.

Signed-off-by: nikolay_tikhonov 

commit c2e836b5b9b183404f4507c64c13ab5c05653d24
Author: EdShangGG 
Date:   2017-08-24T16:15:24Z

ignite-6175 JVM Crash in Ignite Binary Objects Simple Mapper Basic suite

Signed-off-by: Andrey Gura 

commit b2b596b4f59bcf7a1b7397a6fd681a0ae47092db
Author: Andrey Novikov 
Date:   2017-08-25T03:48:15Z

IGNITE-5200 Web Console: Don't cache generated chunks in production.
(cherry picked from commit e1eb1b9)

commit 9399610d2dd4b67b1da6475ce2141787fb8dbb0e
Author: Ilya Lantukh 
Date:   2017-08-25T10:12:32Z

[jira] [Created] (IGNITE-7673) Remove using of Collections.UnmodifiableCollection from GridTaskSessionImpl

[Dmitry Karachentsev (JIRA)]

Dmitry Karachentsev created IGNITE-7673:
---

 Summary: Remove using of Collections.UnmodifiableCollection from 
GridTaskSessionImpl
 Key: IGNITE-7673
 URL: https://issues.apache.org/jira/browse/IGNITE-7673
 Project: Ignite
  Issue Type: Improvement
  Components: compute
Affects Versions: 2.3
Reporter: Dmitry Karachentsev
Assignee: Dmitry Karachentsev
 Fix For: 2.5


GridTaskWorker marshals GridTaskSessionImpl.siblings which is 
UnmodifiableCollection which forces use of OptimizedMarshaller that may have 
performance impact. It should be replaced with original collection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Ignite diagnostic (SQL system views)

[Alex Plehanov]

The views engine and the first view are almost ready to merge (review
comments are resolved). Which views should we take next? My proposal -
NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and NODE_ADDRESSES, since
these views are clear and all topology data available on each node.
Any objections?

2018-01-25 16:27 GMT+03:00 Alex Plehanov :

> Anton, Vladimir, I've made some fixes. There is only one view left and
> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'.
>
> High level design of solution:
> When IgniteH2Indexing is starting, it create and start
> new GridH2SysViewProcessor, which create and register in H2 (via its own
> table engine) all implementations of system views. Each system view
> implementation extends base abstract class GridH2SysView. View
> implementation describes columns, their types and indexes in constructor
> and must override method getRows for data retrieval (this method called by
> H2-compatible table and index implementations for ignite system views).
> Almost no fixes to existing parsing engine was made, except some places,
> where GridH2Table instance was expected, but for system views there is
> another class.
>
> New PR: [1].  Please have a look.
>
> [1] https://github.com/apache/ignite/pull/3433
>
> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov :
>
>> I've created IEP-13 [1] to cover all cases.
>> Feel free to create issues.
>>
>> [1]
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75962769
>>
>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov 
>> wrote:
>>
>> > Let's start with a single and the most simple view, e.g.
>> > LOCAL_TRANSACTIONS. We will review and merge it along with necessary
>> > infrastructure. Then will handle the rest view in separate tickets and
>> > separate focused discussions.
>> >
>> > On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov > >
>> > wrote:
>> >
>> > > 1) It’s not a principal point, I can change schema. The
>> > INFORMATION_SCHEMA
>> > > was used because it’s already exists and usually used for metadata
>> tables
>> > > and views. Your proposal is to use schema “IGNITE”, am I understand
>> you
>> > > right? BTW, for now, we can’t query another (H2) meta tables from the
>> > > INFORMATION_SCHEMA, so, “Ignite system views” is only available views
>> to
>> > > query from this schema.
>> > > 2) Exactly for this reason the IGNITE_INSTANCE view is useful: to
>> > determine
>> > > which node we are connected to.
>> > > 3) As the first phase, in my opinion, local views will be enough.
>> > > Performance and caching of distributed views should be discussed at
>> next
>> > > phases, when distributed views implementation will be planned. In
>> current
>> > > implementation I tried to use indexing for local views wherever it’s
>> > > possible.
>> > > 4) I don’t think, that JVM info is more critical information than, for
>> > > example, caches or nodes information. When authorization capabilities
>> > > planned to implement?
>> > >
>> > > About local data: yes, we can rename all currently implemented views
>> for
>> > > the local node data as LOCAL_..., and create (someday) new whole
>> cluster
>> > > views (which use distributed requests) without prefix or, for example,
>> > with
>> > > CLUSTER_ prefix. But some views can show all cluster information using
>> > only
>> > > local node data, without distributed requests (for example
>> > > IGNITE_NODE_METRICS, IGNITE_PART_ASSIGNMENT, IGNITE_PART_ALLOCATION,
>> > > IGNITE_NODES, etc). Are they local or cluster views in this concept?
>> > Which
>> > > prefix should be used? And what about caches? Are they local or
>> cluster?
>> > On
>> > > local node we can see cluster wide caches (replicated and distributed)
>> > and
>> > > caches for current node only. Local caches list may differ from node
>> to
>> > > node. Which prefix should be used for this view? And one more, there
>> is
>> > no
>> > > sense for some views to make them cluster wide (for example
>> > > INGNITE_INSTANCE). Should we name it LOCAL_INSTANCE without creating
>> > > INSTANCE view?
>> > >
>> > > So, next steps: split PR, change schema name (IGNITE?), change view
>> name
>> > > for caches (CACHES, LOCAL_CACHES?)
>> > >
>> > >
>> > > 2018-01-24 13:03 GMT+03:00 Vladimir Ozerov :
>> > >
>> > > > Hi Alex,
>> > > >
>> > > > System views could be extremely valuable addition for Ignite.
>> Ideally,
>> > > user
>> > > > should be able to monitor and manage state of the whole cluster
>> with a
>> > > > single SQL command line. We have plans to implement it for a very
>> long
>> > > > time. However, this is very sensitive task which should take a lot
>> of
>> > > > moving pieces in count, such as usability, consistency, performance,
>> > > > security, etc..
>> > > >
>> > > > Let me point several major concerns I see at the moment:
>> > > >
>> > > > 1) Usability: INFORMATION_SCHEMA
>> > > > This schema is part of 

Re: IGNITE-4908 is ready for review (Ignite.reentrantLock looks much slower than IgniteCache.lock)

[Александр Меньшиков]

Hi, guys! I'm waiting (more than 2 months) for CR ignite-4908 (
https://issues.apache.org/jira/browse/IGNITE-4908 all links inside). It's
about Ignite.reentrantlock.

2017-11-29 12:35 GMT+03:00 Александр Меньшиков :

> Igniters,
>
> I’ve implemented new fast reentrant lock within an issue[1].
>
> Could someone review prepared PR [2, 3, 4]?
>
> Some details are described below.
>
> The main idea:
>
> Current locks implementation is based on shared states in a cache, while
> the new lock uses IgniteCache#invoke* methods to update shared state
> atomically. New lock implementation doesn’t use a continuous query, so the
> cache became atomic now.
>
> New lock implementation has two types: fair and unfair, split into
> different classes for performance reasons.
>
> Some benchmarks results (hardware: Core i5 (2 gen) + 6GB RAM):
>
> Speed up: single thread + fair:   21.9x (1 node), 3.4x (2 nodes), 9.9x (5
> nodes), 17.9x (10 nodes)
>
> Speed up: single thread + unfair: 22.4x (1 node), 3.2x (2 nodes), 8.0x (5
> nodes), 19.0x (10 nodes)
>
> Speed up: multi-threads + fair:   3.9x (1 n,2 t), 3.5x (1 n,10t), 13.5x (5
> n,2t), 15.0x (5n, 10t)
>
> Speed up: multi-threads + unfair: 33.5x (1 n,2t), 210x (1 n,10t), 318x  (5
> n,2t), 389x  (5n, 10t)
>
> Benchmarks’ summary:
>
> 1) The unfair lock has a local reentrant lock which is used for local
> synchronization as a guard before the shared state. This allows reaching
> performance close to a local reentrant lock.
>
> 2) One server node can be a primary one for the shared state, this gives
> us a performance boost on one node only.
>
> 3) Speedup grows with a number of nodes.
>
>
> [1] JIRA: https://issues.apache.org/jira/browse/IGNITE-4908
>
> [2] PR: https://github.com/apache/ignite/pull/2360
>
> [3] Upsource review: https://reviews.ignite.apache.
> org/ignite/review/IGNT-CR-248
>
> [4] Team City: https://ci.ignite.apache.org/project.html?projectId=
> Ignite20Tests_Ignite20Tests=pull%2F2360%2Fhead
>

[GitHub] ignite pull request #3503: Log4j configuration with DEBUG logging for Object...

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/3503

Log4j configuration with DEBUG logging for ObjectBinaryProcessor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
spring_suite_test_hang_investigation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3503


commit 8a5dd687ec3f25470e79a60ffc1790d6f0139e24
Author: Sergey Chugunov 
Date:   2018-02-12T11:46:17Z

Log4j configuration with DEBUG logging for ObjectBinaryProcessor




---

[jira] [Created] (IGNITE-7672) Fix Javadoc in Java 8

[Peter Ivanov (JIRA)]

Peter Ivanov created IGNITE-7672:


 Summary: Fix Javadoc in Java 8
 Key: IGNITE-7672
 URL: https://issues.apache.org/jira/browse/IGNITE-7672
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.4
Reporter: Peter Ivanov
Assignee: Peter Ivanov
 Fix For: 2.5


Currently, Javadoc builds on TC with {{-Xdoclint:none}} option enabled, that 
turns off Java 8's new Javadoc check algorithm. Natively we have lots of 
warnings and errors in master branch.
# Create per-module tasks (with inclusion to this one) for receiving scope of 
debt.
# Create corresponding build on TC for testing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7671) Fix .gitignore files are tracked error

[Peter Ivanov (JIRA)]

Peter Ivanov created IGNITE-7671:


 Summary: Fix .gitignore files are tracked error
 Key: IGNITE-7671
 URL: https://issues.apache.org/jira/browse/IGNITE-7671
 Project: Ignite
  Issue Type: Task
Reporter: Peter Ivanov
Assignee: Peter Ivanov


Current {{.gitignore}} has definitions of files to ignore, which are already 
under the version control system (some *.sh scripts for example). It needs to 
be fixed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7670) Errors Installing Ignite OSGi Karaf features on JBoss fuse

[Stefano Violi (JIRA)]

Stefano Violi created IGNITE-7670:
-

 Summary: Errors Installing Ignite OSGi Karaf features on JBoss fuse
 Key: IGNITE-7670
 URL: https://issues.apache.org/jira/browse/IGNITE-7670
 Project: Ignite
  Issue Type: Bug
  Components: osgi
Affects Versions: 2.3
 Environment: jboss-fuse-6.2.1.redhat-159
Reporter: Stefano Violi


Installing Ignite's features repository on JBoss Fuse generates the following 
error:

*Karaf shell command:*

JBossFuse:karaf@root18007> features:addurl 
mvn:org.apache.ignite/ignite-osgi-karaf/2.3.0/xml/features

*Error:*
Error executing command: Unable to add repositories:
    Unrecognized root element: 
\{http://karaf.apache.org/xmlns/features/v1.3.0}features

 

I also generated a new feature repository compliant with Fuse, adding the 
ignite-core and ignite-indexing features: in this way i installed them on JBoss 
Fuse.

By the way, when i use these features from my component and I start Ignite with 
the following  Java code

Ignition.start();

I get the following error:

Caused by: java.lang.NoClassDefFoundError: org/h2/server/Service
    at java.lang.Class.forName0(Native Method)[:1.8.0_112]
    at java.lang.Class.forName(Class.java:264)[:1.8.0_112]
    at 
org.apache.ignite.internal.IgniteComponentType.inClassPath(IgniteComponentType.java:153)

 

I think there is some OSGi package import/export problem on Ignite features and 
boundles.

Below the feature:

 

feature name="ignite-core" version="2.3.0" description="Apache Ignite :: Core">
    
    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.javax-cache-api/1.0.0_1
    mvn:org.apache.ignite/ignite-osgi/2.5.0-SNAPSHOT
    mvn:org.apache.ignite/ignite-core/2.5.0-SNAPSHOT
    
    
    
    
    
    
    mvn:org.osgi/org.osgi.enterprise/5.0.0
    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.lucene/5.5.2_1
    mvn:com.h2database/h2/1.4.195
    mvn:org.apache.ignite/ignite-indexing/2.3.0
    

 

Thanks

Kind regards

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7669) java.lang.ClassNotFoundException: org.springframework.asm.commons.EmptyVisitor in session clustering

[Sanjeet Jha (JIRA)]

Sanjeet Jha created IGNITE-7669:
---

 Summary: java.lang.ClassNotFoundException: 
org.springframework.asm.commons.EmptyVisitor in session clustering
 Key: IGNITE-7669
 URL: https://issues.apache.org/jira/browse/IGNITE-7669
 Project: Ignite
  Issue Type: Bug
  Components: binary, websession
Affects Versions: 2.3
Reporter: Sanjeet Jha


I am trying to add web session clustering in my web app. but it shows above 
error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3502: IGNITE-7632 Fixed NPE in update igfs metrics

[pvinokurov]

GitHub user pvinokurov opened a pull request:

https://github.com/apache/ignite/pull/3502

IGNITE-7632 Fixed NPE in update igfs metrics



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7632

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3502.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3502


commit a03b2eaaef8ac5c54b286c434f9ac5836428933a
Author: pvinokurov 
Date:   2018-02-12T10:24:30Z

IGNITE-7632 Fixed NPE in update igfs metrics




---

[GitHub] ignite pull request #3501: ignite-7648

[ascherbakoff]

GitHub user ascherbakoff opened a pull request:

https://github.com/apache/ignite/pull/3501

ignite-7648

Revert IGNITE_ENABLE_FORCIBLE_NODE_KILL system property.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7648

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3501.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3501


commit 7aa6138d65bfaee5d295537035ddef9df43ed8cb
Author: Aleksei Scherbakov 
Date:   2018-02-09T15:57:03Z

IGNITE-7648 wip.

commit 7f8d9d5d816ce56022012840ddfb1df1a1f36aab
Author: Aleksei Scherbakov 
Date:   2018-02-12T09:52:08Z

IGNITE-7648 wip.




---

[GitHub] ignite pull request #3500: ignite-7649 properly setting transformer instance...

[zstan]

Github user zstan closed the pull request at:

https://github.com/apache/ignite/pull/3500


---

IGNITE-6005 is ready for review

[Nikolay Izhikov]

Hello, Igniters.

I've fix some relatively small issue(100 loc) IGNITE-6005 [1], PR [2]
The fix in DataStructuresProcessor.java

As far as I can see from git log Ilya Lantukh maintains this piece of code.

Ilya, can you please take a look at the fix?

[1] https://issues.apache.org/jira/browse/IGNITE-6005

[2] https://github.com/apache/ignite/pull/2773

signature.asc
Description: This is a digitally signed message part

[GitHub] ignite pull request #2858: IGNITE-6521 Update JVM options (default and recom...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2858


---

Re: [SparkDataFrame] Query Optimization. Prototype

[Vladimir Ozerov]

Hi Nikolay,

I am not sure if ticket for DECIMAL column metadata exists. If you haven't
find one under "sql" component, please feel free to create it on your own.
As far as testing of joins, I think it makes sense to start working on it
when we finish ANSI compliance testing which is already in progress.

On Wed, Jan 24, 2018 at 12:27 PM, Nikolay Izhikov 
wrote:

> Hello, Vladimir.
>
> Thank you for an answer.
>
> > Do you mean whether it is possible to read it from table metadata?
>
> Yes, you are right.
> I want to read scale and precision of DECIMAL column from table metadata.
>
> > This will be fixed at some point in future, but I do not have any dates
> at the moment.
>
> Is there ticket for it? I can't find it via jira search
>
> > at this moment we do not have complete list of all restrictions on our
> joins, because a lot of work is delegated to H2.
> > In some unsupported scenarios we throw an exception.
> > In other cases we return incorrect results silently (e.g. if you do not
> co-locate data and forgot to set "distributed joins" flag).
>
> Guys, Val, may be we should exclude join optimization from IGNITE-7077
> while we haven't all limitation on the hand?
>
> > We have a plan to perform excessive testing of joins (both co-located
> and distributed) and list all known limitations.
>
> Can I help somehow with this activity?
>
>
> В Ср, 24/01/2018 в 12:08 +0300, Vladimir Ozerov пишет:
> > Hi Nikolay,
> >
> > Could you please clarify your question about scale and precision? Do you
> mean whether it is possible to read it from table metadata? If yes, it is
> not possible at the moment unfortunately - we do not store information
> about lengths, scales and precision, only actual data types are passed to
> H2 (e.g. String, BigDecimal, etc.). This will be fixed at some point in
> future, but I do not have any dates at the moment.
> >
> > Now about joins - Denis, I think you provided wrong link to our internal
> GridGain docs where we accumulate information about ANSI compatibility and
> which will are going to publish on Ignite WIKI when it is ready. In any
> case, this is not what Nikolay aksed about. The question was about
> limitation of our joins which has nothing to do with ANSI standard.
> Unfortunately, at this moment we do not have complete list of all
> restrictions on our joins, because a lot of work is delegated to H2. In
> some unsupported scenarios we throw an exception. In other cases we return
> incorrect results silently (e.g. if you do not co-locate data and forgot to
> set "distributed joins" flag). We have a plan to perform excessive testing
> of joins (both co-located and distributed) and list all known limitations.
> This would require writing a lot of unit tests to cover various scenarios.
> I think we will have this information in a matter of 1-2 months.
> >
> > Vladimir.
> >
> > On Tue, Jan 23, 2018 at 11:45 PM, Denis Magda  wrote:
> > > Agree. The unsupported functions should be mentioned on the page that
> will cover Ignite ANSI-99 compliance. We have first results available for
> CORE features of the specification:
> > > https://ggsystems.atlassian.net/wiki/spaces/GG/pages/
> 45093646/ANSI+SQL+99  net/wiki/spaces/GG/pages/45093646/ANSI+SQL+99>
> > >
> > > That’s on my radar. I’ll take care of this.
> > >
> > > —
> > > Denis
> > >
> > > > On Jan 23, 2018, at 10:31 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org> wrote:
> > > >
> > > > I think we need a page listing the unsupported functions with
> explanation
> > > > why, which is either it does not make sense in Ignite or is planned
> in
> > > > future release.
> > > >
> > > > Sergey, do you think you will be able to do it?
> > > >
> > > > D.
> > > >
> > > > On Tue, Jan 23, 2018 at 12:05 AM, Serge Puchnin <
> sergey.puch...@gmail.com>
> > > > wrote:
> > > >
> > > >> yes, the Cust function is supporting both Ignite and H2.
> > > >>
> > > >> I've updated the documentation for next system functions:
> > > >> CASEWHEN Function, CAST, CONVERT, TABLE
> > > >>
> > > >> https://apacheignite-sql.readme.io/docs/system-functions
> > > >>
> > > >> And for my mind, next functions aren't applicable for Ignite:
> > > >> ARRAY_GET, ARRAY_LENGTH, ARRAY_CONTAINS, CSVREAD, CSVWRITE,
> DATABASE,
> > > >> DATABASE_PATH, DISK_SPACE_USED, FILE_READ, FILE_WRITE, LINK_SCHEMA,
> > > >> MEMORY_FREE, MEMORY_USED, LOCK_MODE, LOCK_TIMEOUT, READONLY,
> CURRVAL,
> > > >> AUTOCOMMIT, CANCEL_SESSION, IDENTITY, NEXTVAL, ROWNUM, SCHEMA,
> > > >> SCOPE_IDENTITY, SESSION_ID, SET, TRANSACTION_ID, TRUNCATE_VALUE,
> USER,
> > > >> H2VERSION
> > > >>
> > > >> Also an issue was created for review current documentation:
> > > >> https://issues.apache.org/jira/browse/IGNITE-7496
> > > >>
> > > >> --
> > > >> BR,
> > > >> Serge
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > >>
> > >
> >
> >
>

Re: default SQL timeout

[Vladimir Ozerov]

Makes sense. Going forward we should have fine-grained resource control:
max duration (timeouts), max results, etc..

On Fri, Dec 22, 2017 at 1:41 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> To my knowledge, it's only possible to set timeout on query level which
> actually can be an issue. Developers and admins do not always have full
> control on what is executed on the cluster and may want to set timeout on
> server side rather then rely on clients.
>
> Here is the ticket for this:
> https://issues.apache.org/jira/browse/IGNITE-7285
>
> -Val
>
> On Tue, Dec 19, 2017 at 7:02 PM, Dmitriy Setrakyan 
> wrote:
>
> > Igniters,
> >
> > Is there any way to set a default SQL timeout for queries? If not, should
> > we have it?
> >
> > D.
> >
>