Re: Hello Ignite Community

[Monil Shah]

Hey Raghu,

Welcome..:) I have also joined the community couple of days back. Is there
any specific documentation you are following to understand the architecture
Ignite?



On Sat, May 5, 2018 at 6:05 AM, Raghu Veer  wrote:

> Hello All,
>
> I happen to be a software engineer working in storage and virtualization.
> I got introduced to Ignite by one of my colleagues and I got immediately
> interested in the idea of what Ignite is doing.
> I am very much excited to join the community and work alongside great
> developers and add whatever little value I can from my side.
>
> Thanks,
> Raghu Veer S
>



-- 
Thanks and Regards
Monil Shah
Software Engineer
+16018131329

Hello Ignite Community

[Raghu Veer]

Hello All,

I happen to be a software engineer working in storage and virtualization. I got 
introduced to Ignite by one of my colleagues and I got immediately interested 
in the idea of what Ignite is doing.
I am very much excited to join the community and work alongside great 
developers and add whatever little value I can from my side.

Thanks,
Raghu Veer S

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Nikolay Izhikov]

Hello, Guys.

Here are answers to the TDE design questions.
I will create FAQ in IEP-18 with this answers, also.

> 1. MEK, CEK rotation. Should we provide the way to change(regenerate) MEK, 
> CEK from time to time?

Yes. PCI DSS are require it. See 3.6.4, 3.6.5 sections. 

> 2. Does CEK(table keys) relate to user access permission?

No. It is prohibited by PCI DSS. 3.4.1: "Decryption keys must not be associated 
with user accounts" 

> 3. WAL encryption. How will it be implemented?

LogicalRecord – key+value for encrypted cache entries are encrypted. 
PhysicalRecord.FullPageSnaphot - regular page encryption algorithm used. 
PhysicalRecord.DeltaRecord - delta is encrypted. 

> 4. We should keep CEKs in MetaStore.

Yes, we should. 
We can't store CEKs in KeyStore because of PCI DSS 3.5.3.c: "Key-encrypting 
keys are stored separately from data-encrypting keys." 

> 5. How should we handle following case

I propose to not include the node to cluster with the appropriate exception 
message. 

> 6. Public API to deal with CEK should be provided.

The first draft of public API for TDE - 
https://github.com/nizhikov/ignite/pull/12 
 
> 7. Can each node use own CEK? What are pros and cons for that? 

I propose to use one CEK per cache. 
CEK would be the same on every cluster node. 
It simplifies Ignite cluster management, backup procedure(like saving key 
copies on some external device), etc. 

> 8. How we can ensure that decryption succeed? In case CEK is broken. It can 
> be broken because of memory corruption, network errors, etc.

Currently, page integrity is checked by CRC. 
I propose to reuse this procedure to ensure decryption integrity. 

> 9. Specific encryption algorithm has to be chosen prior an implementation. 

AES CBC – 256, 192, 128 bits. 

> 10. Page integrity are checked by CRC. How this process would be changed for 
> encrypted pages?

The process doesn't change: 
1. Decrypt page. 
2. Check CRC. 

> 11. Page header has well-known content. This can be used for known-plain-text 
> attacks. We should measure the treatment and find the way to deal with it.

This question requires additional research.
We have to understand: Is there any issue here?
But at first glance, We can encrypt blocks in reverse order – from last to 
first. 
Last blocks don't have well know so there is no issue here.

В Пт, 27/04/2018 в 19:41 +0300, Nikolay Izhikov пишет:
> Hello, Igniters.
> 
> We've discussed TDE design privately with some respected community members 
> including Vladimir Ozerov and Alexey Goncharyuk.
> Here the list of questions we have to address before starting TDE 
> implementation:
> 
> 1. MEK, CEK rotation. 
> Should we provide the way to change(regenerate) MEK, CEK from time to time? 
> Is it required by PCI DSS standard? 
> 
> 2. Does CEK(table keys) relate to user access permission? 
> Need to study other vendors implementation.  
> 
> 3. WAL encryption. How will it be implemented? What issues we have to solve? 
> 
> 4. We should keep CEKs in MetaStore. 
> Not a question, just to write down decision.
> 
> 5. How should we handle following case:  
> a. Node X left cluster. 
> b. Node X joins cluster.
> c. Between steps a and b encryption keys has been changed 
> 
> 6. Public API to deal with CEK should be provided. 
> Looks like we need to support following methods:
> a. Generate new CEK when encrypted cache are created 
> b. Decrypt CEK whenever needed. 
> 
> 7. Can each node use own CEK? What are pros and cons for that? 
> 
> 8. How we can ensure that decryption succeed? 
> In case CEK is broken. It can be broken because of memory corruption, network 
> errors, etc. 
> 
> 9. Specific encryption algorithm has to be chosen prior an implementation. 
> We have to support usage of other algorithms. 
> 
> 10. Page integrity are checked by CRC. How this process would be changed for 
> encrypted pages?
> 
> 11. Page header has well-known content. This can be used for known-plain-text 
> attacks. 
> We should measure the treatment and find the way to deal with it.

signature.asc
Description: This is a digitally signed message part

[GitHub] ignite pull request #3945: IGNITE-8436 Added shutdown for services thread po...

[rprabhat]

GitHub user rprabhat opened a pull request:

https://github.com/apache/ignite/pull/3945

IGNITE-8436 Added shutdown for services thread pool



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rprabhat/ignite ignite-8436

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3945.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3945


commit 57667acad2ee002c42122243dfbb25a02ce7d7e3
Author: Prabhat Ranjan 
Date:   2018-05-05T10:40:55Z

Merge pull request #1 from apache/master

merge from master

commit 9f8f8f75a8277e0e8299a39e37990461ad375a59
Author: Prabhat Ranjan 
Date:   2018-05-05T14:04:51Z

IGNITE-8436 Added shutdown for services thread pool.




---

Re: memory-only mode for Ignite indexes

[Vladimir Ozerov]

In general I do not support this initiative. There are two serious reasons
for that:
1) Our indexes are slow on updates due to architectural flaws. First, every
index entry must be of fixed size. For this reason we cannot inline full
values in general case and suffer from data page lookups [1]. Second, final
comparisons always compare primary keys, so another lookup is needed [2].
Third, our indexes are fat because we are lacking prefix compression [3].
2) Some vendors do have memory-only indexes - SQL Server, Couchbase,
MemSQL, to name a few. But they are memory optimized - no pages, no BTrees.
Lock-free skiplist is used instead. This is correct design which really
fast. But we are very far from it at the moment.

Taking this in count I would not consider memory-only BTree indexes in the
nearest future. Instead, we should focus on performance. When mentioned
things are fixed/implemented, our indexes will be both memory-efficient and
very fast to update.

[1]
https://issues.apache.org/jira/browse/IGNITE-8385
[2]
https://issues.apache.org/jira/browse/IGNITE-8384
[3]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite#IEP-20:DataCompressioninIgnite-IndexPrefixCompression

сб, 5 мая 2018 г. в 3:46, Dmitriy Setrakyan :

> Igniters,
>
> One of the main complaints I hear from users is that whenever the
> persistence is turned on, index creation can really slow down the
> performance, because of massive amounts of writes to disk. The reason
> Ignite is writing indexes to disk is to support fast restarts - nothing
> needs to be rebuilt on startup, and Ignite can become operational right
> away.
>
> However, as far as I can tell, most users care about faster operations
> after the system is started and much less about the startup speed. What if
> we added a mode where we do not persist indexes at all? This way data
> ingestion and overall throughput will significantly increase (of course, at
> the cost of startup type getting longer because we have to rebuild the
> indexes).
>
> There are 2 ways to achieve this in Ignite. The simplest way is not mark
> index pages dirty in memory, so they will never participate in
> check-pointing process. We also have to make sure that index pages never
> get evicted form memory. This can be done fairly quickly. The disadvantage
> of this approach is that if indexes fill up most of the memory, then it
> will be very difficult to find a page to evict, which may hurt the
> performance.
>
> The other way is to have a separate in-memory off-heap region for indexes.
> This region should never be persisted. It maybe somewhat bigger
> refactoring, as we currently do not separate between index and data pages.
> However, the advantage of this approach is that this region can be flushed
> to disk practically as is during a graceful shutdown of the node, and hence
> shorten the restart time.
>
> I think we should start from the 1st approach and then think about the 2nd
> one. What do you think?
>
> D.
>